Image reconstruction and endoscopic tracking
The described system enhances endoscopic mapping and tracking by using an aiming beam footprint and landmark identification to generate a target map, correcting for geometric distortions, thus improving surgical precision and safety.
Patent Information
- Application Number
- JP2024158622
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-07-24
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2041-07-16
AI Technical Summary
Existing endoscopic systems lack robustness in mapping and tracking endoscopic positions, particularly in the presence of image distortion or deformation, leading to inefficiencies and potential inaccuracies in surgical procedures.
An imaging system captures endoscopic images with an aiming beam footprint, identifies landmarks, and generates a target map by integrating multiple images, using a video processor to track endoscope position, and corrects for geometric distortions through transformations like scaling and rotation.
Improves endoscopic mapping and tracking accuracy, reducing procedure time and enhancing operator intervention capabilities, thereby improving surgical precision and safety.
Smart Images

Figure 2024169475000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 055,936, filed July 24, 2020, the contents of which are incorporated by reference in their entirety herein.
[0002] The present disclosure relates generally to endoscopy, and more particularly to systems and methods for endoscopic mapping of targets and tracking of endoscope position during a procedure. [Background technology]
[0003] Endoscopes are typically used to provide access to internal locations of a subject for visual access by a physician. Endoscopes are usually inserted into the patient's body to send light to a target (e.g., a target anatomical structure or object) under examination and collect light reflected from the object. The reflected light carries information about the object of examination and can be used to create an image of the object. Some endoscopes include a working channel through which the operator can perform suction or pass instruments such as brushes, biopsy needles, or forceps, or perform minimally invasive surgery to remove unwanted tissue or foreign objects from the patient's body.
[0004] Some endoscopes may include or be used with laser or plasma systems for delivering surgical laser energy to a target anatomical structure or object, such as soft or hard tissue. Examples of laser treatments include ablation, coagulation, vaporization, fragmentation, etc. In lithotripsy applications, lasers have been used to break up stone structures or ablate larger stones into smaller fragments in the kidney, gallbladder, ureter, among other areas of stone formation.
[0005] Video systems have been used to assist physicians or technicians in visualizing the treatment site and navigating the endoscope during endoscopic procedures. Image-guided endoscopy generally requires locating the endoscope position and tracking the endoscope's movement in the coordinate system of the target area. Accurate mapping of the target area as well as efficient endoscope localization and tracking may improve the accuracy of endoscopic manipulation during endoscopic procedures, enhance the physician's or technician's ability to intervene, and improve the efficacy of treatment (e.g., laser treatment). Summary of the Invention [Means for solving the problem]
[0006] The present specification describes systems, devices, and methods for endoscopic mapping of a target and tracking of an endoscope position within a subject's body during a procedure. An exemplary system includes an imaging system configured to capture an endoscopic image of a target including a footprint of an aiming beam directed at the target, and a video processor configured to identify one or more landmarks from the captured endoscopic images and determine their respective positions relative to the aiming beam footprint, and generate a target map by integrating the multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images. The target map can be used to track the endoscope position during an endoscopic procedure.
[0007] Example 1 is a system for endoscopic mapping of a target, the system including an imaging system configured to capture endoscopic images of the target, the endoscopic images including a footprint of an aiming beam directed at the target, and a video processor configured to identify one or more landmarks from the captured endoscopic images and determine their respective positions relative to the aiming beam footprint, and generate a target map by integrating the multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images.
[0008] In Example 2, the subject matter of Example 1 optionally includes a video processor that can be configured to identify a tissue type at the location of the aiming beam and mark the aiming beam footprint with a visual identifier indicative of the identified tissue type.
[0009] In Example 3, the subject matter of any one or more of Examples 1-2 optionally includes a spectrometer communicatively coupled to the video processor, the spectrometer configured to measure one or more spectroscopic characteristics of the illumination light signal reflected from the target, and the video processor configured to identify a tissue type at the position of the aiming beam based on the one or more spectroscopic characteristics, and mark the aiming beam footprint with a visual identifier indicative of the identified tissue type.
[0010] In example 4, the subject matter of any one or more of examples 2-3 optionally includes a video processor that can be configured to identify tissue types as normal tissue or abnormal tissue.
[0011] In Example 5, the subject matter of any one or more of Examples 2-4 optionally includes a video processor that can be configured to mark the aiming beam footprint in different colors to indicate different tissue types.
[0012] In Example 6, the subject matter of any one or more of Examples 1-5 optionally includes a video processor that can be configured to identify one or more landmarks from the endoscopic image based on changes in brightness of pixels in the endoscopic image.
[0013] In Example 7, the subject matter of Example 6 optionally includes one or more landmarks represented in the endoscopic image as line segments or intersecting line segments.
[0014] In Example 8, the subject matter of any one or more of Examples 1-7 optionally includes a video processor that can be configured to select a subset of landmarks from the landmarks identified from one or more of the plurality of endoscopic images based on whether laser energy is activated at each target site where the identified landmarks are located, and generate a target map by integrating the plurality of endoscopic images based on the selected subset of landmarks.
[0015] In Example 9, the subject matter of any one or more of Examples 1-8 optionally includes a plurality of endoscopic images that may include images of various portions of the target, including a first endoscopic image of a first target portion captured from a first endoscopic position and a second endoscopic image of a second target portion captured from a second endoscopic position, and the video processor is configured to identify matching landmarks including two or more landmarks in the first endoscopic image that match corresponding two or more landmarks in the second endoscopic image, align the first and second endoscopic images with respect to the matching landmarks in a coordinate system of the first image, and generate a target map using at least the aligned first and second images.
[0016] In Example 10, the subject matter of Example 9 optionally includes a video processor that can be configured to transform the second image including one or more of scaling, translating, or rotating the second image, and align the transformed second image and the first image with respect to the matching landmarks.
[0017] In Example 11, the subject matter of Example 10 optionally includes transforming the second image, which may include matrix multiplication with a transformation matrix.
[0018] In Example 12, the subject matter of any one or more of Examples 10-11 optionally includes a video processor that can be configured to scale the second image using a scaling factor based on a ratio of the distance between two of the matching landmarks in the first image to the distance between two corresponding landmarks in the second image.
[0019] In Example 13, the subject matter of any one or more of Examples 10-12 optionally includes a video processor that can be configured to scale the second image by a scaling factor based on a ratio of a geometric characteristic of the aiming beam footprint in the first image to a geometric characteristic of the aiming beam footprint in the second image.
[0020] In Example 14, the subject matter of any one or more of Examples 10-13 optionally includes a video processor that can be configured to convert the second image to correct for a change in an orientation of the endoscope between the first image and the second image, the orientation of the endoscope indicating the inclination of the tip of the endoscope relative to the target site.
[0021] In Example 15, the subject matter of Example 14 optionally includes a video processor that can be configured to detect a change in an orientation of the endoscope using a first gradient between two of the matching landmarks in the first image and a second gradient between two corresponding landmarks in the second image.
[0022] In Example 16, the subject matter of any one or more of Examples 14-15 optionally includes a video processor that can be configured to detect a change in the orientation of the endoscope using a first geometric characteristic of the aiming beam footprint in the first image and a second geometric characteristic of the aiming beam footprint in the second image.
[0023] In Example 17, the subject matter of Example 16 optionally includes at least one of the first or second aiming beam footprints having an elliptical shape having a major axis and a minor axis, and at least one of the first or second geometric features having a ratio of a length of the major axis to a length of the minor axis.
[0024] In Example 18, the subject matter of any one or more of Examples 1 to 17 optionally includes an endoscopic tracking system configured to identify matching landmarks from a real-time image of a target treatment site captured by an imaging system from an unknown endoscopic position during an endoscopic procedure, the matching landmarks including two or more landmarks in a target map that match corresponding two or more landmarks in the real-time image, register the real-time image to the target map using the matching landmarks, and track a position of the tip of the endoscope based on the registration of the real-time image.
[0025] In Example 19, the subject matter of Example 18 optionally includes an endoscopic tracking system that can be configured to identify matching landmarks based on one or more ratios of distances between landmarks in the real-time image and one or more ratios of distances between landmarks in the target map.
[0026] In Example 20, the subject matter of any one or more of Examples 18-19 optionally includes an endoscopic tracking system that can be configured to generate an indication of changes in tissue type at the target site.
[0027] Example 21 is a method for endoscopic mapping of a target, the method comprising the steps of directing an aiming beam at a target, capturing an endoscopic image of the target via an imaging system, the endoscopic image including a footprint of the aiming beam, identifying one or more landmarks from the captured endoscopic image via a video processor and determining respective positions of the one or more landmarks relative to the aiming beam footprint, and generating, via the video processor, a target map by integrating the multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images.
[0028] In Example 22, the subject matter of Example 21 optionally includes the steps of identifying a tissue type at the position of the aiming beam using an illumination light signal reflected from the target, and marking the aiming beam footprint with a visual identifier indicative of the identified tissue type.
[0029] In Example 23, the subject matter of any one or more of Examples 21-22 optionally includes a step of identifying one or more landmarks from the endoscopic image based on changes in brightness of pixels in the endoscopic image.
[0030] In Example 24, the subject matter of any one or more of Examples 21 to 23 optionally includes the multiple endoscopic images including images of various regions of the target including a first endoscopic image of a first target region captured at a first endoscopic position and a second endoscopic image of a second target region captured at a second endoscopic position, and the method includes a step of identifying matching landmarks including two or more landmarks in the first endoscopic image that match corresponding two or more landmarks in the second endoscopic image, a step of registering the first and second endoscopic images with respect to the matching landmarks in a coordinate system of the first image, and a step of generating a target map using at least the registered first and second images.
[0031] In Example 25, the subject matter of Example 24 optionally includes a step of registering the first and second endoscopic images, the step including transforming the second image including one or more of scaling, translating, or rotating the second image, and registering the transformed second image and the first image relative to matching landmarks.
[0032] In Example 26, the subject matter of Example 25 optionally includes transforming the second image, which includes scaling the second image by a scaling factor based on a ratio of a distance between two of the matching landmarks in the first image to a distance between two corresponding corresponding landmarks in the second image.
[0033] In Example 27, the subject matter of any one or more of Examples 25-26 optionally includes a step of transforming the second image, which includes a step of scaling the second image by a magnification factor based on a ratio of a geometric characteristic of the aiming beam footprint in the first image to a geometric characteristic of the aiming beam footprint in the second image.
[0034] In Example 28, the subject matter of any one or more of Examples 25-27 optionally includes a step of transforming the second image to correct for a change in endoscope orientation between the first image and the second image, where the orientation of the endoscope indicates the inclination of the tip of the endoscope relative to the target site.
[0035] In Example 29, the subject matter of Example 28 optionally includes detecting a change in orientation of the endoscope using a first gradient between two of the matching landmarks in the first image and a second gradient between two corresponding landmarks in the second image.
[0036] In Example 30, the subject matter of any one or more of Examples 28-29 optionally includes a step of detecting a change in the orientation of the endoscope using a first geometric characteristic of the aiming beam footprint in the first image and a second geometric characteristic of the aiming beam footprint in the second image.
[0037] In Example 31, the subject matter of any one or more of Examples 21 to 30 optionally includes the steps of capturing real-time images of a target treatment site using an imaging system from an unknown endoscopic position during an endoscopic procedure, identifying matching landmarks including two or more landmarks in a target map that match corresponding two or more landmarks in the real-time image, registering the real-time image to the target map using the matching landmarks, and tracking the position of the tip of the endoscope based on the registration of the real-time image.
[0038] In Example 32, the subject matter of Example 31 optionally includes a step of identifying matching landmarks based on one or more ratios of distances between landmarks in the real-time image and one or more ratios of distances between landmarks in the target map.
[0039] Example 33 is at least one non-transitory machine-readable storage medium including instructions that, when executed by one or more processors of the machine, cause the machine to perform operations comprising: aiming an aiming beam at a target; capturing an endoscopic image of the target, where the endoscopic image includes a footprint of the aiming beam; identifying one or more landmarks from the captured endoscopic image and determining respective positions of the one or more landmarks relative to the aiming beam footprint; and generating a target map by integrating the multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images.
[0040] In Example 34, the subject matter of Example 33 optionally includes the instructions causing the machine to perform operations further comprising identifying a tissue type at the location of the aiming beam and marking the aiming beam footprint with a visual identifier indicative of the identified tissue type.
[0041] In Example 35, the subject matter of any one or more of Examples 33-34 optionally includes, wherein the instructions cause the machine to perform an operation further comprising identifying one or more landmarks from the endoscopic image based on a change in brightness of pixels of the endoscopic image.
[0042] In Example 36, the subject matter of any one or more of Examples 33-35 optionally includes, wherein the multiple endoscopic images include images of various regions of the target including a first endoscopic image of a first target region captured at a first endoscopic position and a second endoscopic image of a second target region captured at a second endoscopic position, and the instructions cause the machine to perform operations further including identifying matching landmarks including two or more landmarks in the first endoscopic image that match corresponding two or more landmarks in the second endoscopic image, aligning the first and second endoscopic images with respect to the matching landmarks in a coordinate system of the first image, and generating a target map using at least the aligned first and second images.
[0043] In Example 37, the subject matter of Example 36 optionally includes where the act of aligning the first and second endoscopic images includes transforming the second image including one or more of scaling, translating, or rotating the second image, and aligning the transformed second image and the first image relative to the matching landmarks.
[0044] In Example 38, the subject matter of Example 37 optionally includes the act of transforming the second image including scaling the second image by a scaling factor based on a ratio of a distance between two of the matching landmarks in the first image to a distance between two corresponding corresponding landmarks in the second image.
[0045] In Example 39, the subject matter of any one or more of Examples 37-38 optionally includes a step of scaling the second image by a scaling factor based on a ratio of a geometric characteristic of the aiming beam footprint in the first image to a geometric characteristic of the aiming beam footprint in the second image.
[0046] In Example 40, the subject matter of any one or more of Examples 37 to 39 optionally includes the act of transforming the second image correcting for a change in an orientation of the endoscope between the first image and the second image, the orientation of the endoscope indicating an inclination of the tip of the endoscope relative to the target site.
[0047] In Example 41, the subject matter of Example 40 optionally includes the instructions causing the machine to perform an operation further comprising detecting a change in orientation of the endoscope using a first gradient between two of the matching landmarks in the first image and a second gradient between two corresponding landmarks in the second image.
[0048] In Example 42, the subject matter of any one or more of Examples 40 to 41 optionally includes the instructions causing the machine to perform an operation further comprising detecting a change in orientation of the endoscope using a first geometric characteristic of the aiming beam footprint in the first image and a second geometric characteristic of the aiming beam footprint in the second image.
[0049] In Example 43, the subject matter of any one or more of Examples 33-42 optionally includes the instructions causing the machine to perform operations further comprising capturing real-time images of a target treatment site from an unknown endoscope position during an endoscopic procedure, identifying matching landmarks including two or more landmarks in the target map that match corresponding two or more landmarks in the real-time images, registering the real-time images to the target map using the matching landmarks, and tracking a position of the tip of the endoscope based on the registration of the real-time images.
[0050] In Example 44, the subject matter of any one or more of Examples 33-43 optionally includes, wherein the act of identifying matching landmarks is based on one or more ratios of distances between landmarks in the real-time image and one or more ratios of distances between landmarks in the target map.
[0051] This Summary is an overview of some of the teachings of the present application and is not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details regarding the present subject matter are set forth in the detailed description and appended claims. Other aspects of the present disclosure will become apparent to those skilled in the art upon reading and understanding the following detailed description and viewing the drawings that form a part thereof, each of which is not to be construed in a limiting sense. The scope of the present disclosure is defined by the appended claims and their legal equivalents.
[0052] Various embodiments are illustrated by way of example in the accompanying drawing figures. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present subject matter. [Brief description of the drawings]
[0053] [Figure 1] FIG. 1 illustrates an example of a medical system for use in an endoscopic procedure. [Diagram 2] FIG. 2 is a schematic diagram of a portion of the system shown in FIG. [Diagram 3] FIG. 2 is a block diagram illustrating an example of an endoscope controller for controlling various portions of the system shown in FIG. [Figure 4A] 1A-1C show examples of sequences of endoscopic images or video frames captured at different target sites and landmarks, and aiming beam footprints detected therefrom. [Figure 4B] 1A-1C show examples of sequences of endoscopic images or video frames captured at different target sites and landmarks, and aiming beam footprints detected therefrom. [Figure 4C] 1A-1C show examples of sequences of endoscopic images or video frames captured at different target sites and landmarks, and aiming beam footprints detected therefrom. [Figure 4D]1A-1C show examples of sequences of endoscopic images or video frames captured at different target sites and landmarks, and aiming beam footprints detected therefrom. [Figure 4E] 1A-1C show examples of sequences of endoscopic images or video frames captured at different target sites and landmarks, and aiming beam footprints detected therefrom. [Figure 4F] 1A-1C show examples of sequences of endoscopic images or video frames captured at different target sites and landmarks, and aiming beam footprints detected therefrom. [Diagram 5] FIG. 13 shows an example of a target map reconstructed from multiple endoscopic images or video frames captured at different target sites. [Figure 6A] 1A-1C are diagrams illustrating, by way of example, the effect of endoscope orientation on endoscopic image features and correction for changes in endoscope orientation between endoscopic images. [Figure 6B] 1A-1C are diagrams illustrating, by way of example, the effect of endoscope orientation on endoscopic image features and correction for changes in endoscope orientation between endoscopic images. [Figure 6C] 1A-1C are diagrams illustrating, by way of example, the effect of endoscope orientation on endoscopic image features and correction for changes in endoscope orientation between endoscopic images. [Figure 6D] 1A-1C are diagrams illustrating, by way of example, the effect of endoscope orientation on endoscopic image features and correction for changes in endoscope orientation between endoscopic images. [Figure 6E] 1A-1C are diagrams illustrating, by way of example, the effect of endoscope orientation on endoscopic image features and correction for changes in endoscope orientation between endoscopic images. [Figure 6F] 1A-1C are diagrams illustrating, by way of example, the effect of endoscope orientation on endoscopic image features and correction for changes in endoscope orientation between endoscopic images. [Figure 7]FIG. 13 illustrates an example of recognizing matching landmarks between a real-time image and a reconstructed target map, and registering the real-time image to the target map with respect to the matching landmarks. [Figure 8] 1 is a flow chart illustrating a method for endoscopic mapping of targets within a subject during treatment. [Figure 9] 1 is a flow diagram illustrating an example of a method for endoscope tracking using a reconstructed target map. [Figure 10] FIG. 1 is a block diagram illustrating an example machine upon which any one or more of the techniques (e.g., methodologies) discussed herein may be performed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0054] Minimally invasive endoscopic surgery is a surgical procedure in which a rigid or flexible endoscope is introduced into a target area of a subject's body through a natural orifice or small incision in the skin. The endoscope can be used to introduce additional surgical instruments, such as laser fibers, into the subject's body through similar ports to provide visual feedback to the surgeon of the surgical site and surgical instruments.
[0055] Endoscopic surgery may include pre-operative and intra-operative phases. The pre-operative phase includes acquiring images or video frames of a target anatomical structure or object using an imaging system (e.g., a video camera) and reconstructing a map using the images or video frames. The map may be used for diagnostic evaluation or endoscopic surgery planning. In the intra-operative phase, an endoscope may be introduced to a target treatment site. The surgeon may move and pivot the distal end of the endoscope and acquire real-time images of the treatment site, such as via an imaging system located at the distal end of the endoscope. The position and orientation of a surgical tool (e.g., a laser fiber) at the distal end of the endoscope may be monitored and tracked throughout the procedure.
[0056] One conventional intraoperative tracking involves a freehand technique in which the surgeon views the surgical field on a monitor that displays real-time images or videos of the surgical field without an automatic tracking or navigation system. This approach fails to establish a relationship between the images that facilitates tracking of the position and orientation of the endoscopic surgical tool relative to the target. Another approach involves a navigation-based tracking system, such as an optical or electromagnetic tracking system, that tracks the position and orientation of the endoscopic surgical tool. An image registration procedure can be performed to register the real-time image to the target map. Fiducial markers displayed on the real-time image are used as references to guide the surgeon with real-time feedback of the position and orientation of the endoscopic surgical tool. The fiducial markers can be external objects attached to the patient, or internal anatomical fiducials. External fiducials can lack positional consistency and increase the complexity of the system. The use of internal fiducials generally limits the physical movement of the endoscope, such as requiring the scope to touch the anatomical fiducial during the procedure, which can increase the procedure time. Conventional navigation-based endoscopic tracking systems can also suffer from poor tracking performance when there are distortions or deformations of the image, such as changes in the position, gaze direction, and orientation (e.g., tilt or skew) of the camera relative to the target surface. For at least the above reasons, the inventors have recognized an unmet need for an improved endoscopic mapping and tracking system that is more robust to image distortion or deformation when used in endoscopic procedures.
[0057] Described herein are systems, devices, and methods for endoscopic mapping of a target and tracking an endoscope position within a subject's body during a procedure. An exemplary system includes an imaging system configured to capture an endoscopic image of a target including a footprint of an aiming beam directed at the target, and a video processor configured to identify one or more landmarks from the captured endoscopic images and determine their respective positions relative to the aiming beam footprint, and generate a target map by integrating the multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images. The target map can be used to track the position of the endoscope during a procedure.
[0058] Systems, devices, and methods according to various embodiments discussed herein can provide improved endoscopic mapping of targets and tracking of endoscopic position during endoscopic procedures. According to various examples of the present disclosure, various image features can be generated from endoscopic images, including, for example, landmarks and their positions relative to the aiming beam footprint, spatial relationships between landmarks, shape and geometric properties of the aiming beam footprint, among others. Image registration, target map reconstruction, and endoscopic tracking based on those image features described herein are more resilient to image rotation, zoom, shrink, changes in camera position, line of sight, or changes in endoscope orientation during endoscopic procedures. Improved navigation and endoscopic tracking can improve the operator's ability to intervene and the accuracy of endoscopic movements, reduce procedure times, and improve overall procedure effectiveness, patient safety, and system reliability.
[0059] The subject matter described herein can be applied to various endoscopic applications, including, but not limited to, arthroscopy, bronchoscopy, colonoscopy, laparoscopy, brain endoscopy, and endoscopic cardiac surgery. Examples of endoscopic cardiac surgery include, but are not limited to, endoscopic coronary artery bypass, endoscopic mitral and aortic valve repair and replacement. As used herein, "endoscopic" is broadly defined as a characterization of an image acquired by any type of endoscope capable of imaging from within the body. Examples of endoscopes for the purposes of the present invention include, but are not limited to, any type of flexible or rigid scope (e.g., endoscope, arthroscope, bronchoscope, choledochoscope, colonoscope, cystoscope, duodenoscope, gastroscope, hysteroscope, laparoscope, laryngoscope, neuroscope, otoscope, push enteroscope, nasopharyngoscope, sigmoidoscope, sinusoscope, thoracoscope, etc.), and any device similar to a scope with an imaging system (e.g., a nested cannula with imaging). Imaging is local, and surface images may be acquired optically using fiber optics, lenses, or miniaturized (e.g., CCD-based) imaging systems. Examples of fluoroscopes for purposes of the present invention include, but are not limited to, X-ray imaging systems.
[0060] FIG. 1 is a diagram showing an example of a medical system 100 for use in an endoscopic procedure. The system 100 includes an endoscope 102, an endoscope controller 103, a light source 104, a laser device 106, and a display 108. A schematic diagram of a portion of the system 100 is shown in FIG. 2. The endoscope 102 includes an insertion section 110 at a distal portion and an operating unit 107 at a proximal portion of the endoscope 102. The insertion section 110 can be configured to be inserted into a target site of a subject, capture an image of the target 101, and optionally perform a procedure therein. The insertion section 110 can be formed using an illumination fiber (illumination guide), an electric cable, an optical fiber, or the like. In the example shown in FIG. 1, the insertion section 110 includes a distal end 110a incorporating an imaging unit, a bendable bending section 110b including a plurality of bending pieces, and a flexible tube section 110c having flexibility provided on the proximal end side of the bending section 110b.
[0061] 2, the distal end 110a may include an illumination guide 120 coupled to the light source 104 and configured to project illumination light 230 onto the target 101 via an illumination lens 122. The distal end 110a may include an observation unit, such as an imaging system 115 configured to image the target 101. The imaging system 115 may include an image sensor 116 and an associated lens system 118. An example of the image sensor 116 may include a CCD or CMOS camera sensitive to ultraviolet (UV), visible (VIS), or infrared (IR) wavelengths. The endoscope 102 may include an insertion port 107b located within the endoscope 102 and coupled to a treatment tool channel 102a extending along the insertion portion 110. An optical pathway 112 disposed within the channel 102a through the insertion port 107b has a proximal end operably connected to the laser device 106 and extends distally from a distal end opening 102b of the channel 102a. Laser energy, such as treatment beam 240a or aiming beam 240b, may be transmitted through optical pathway 112, emitted from a distal end 112a of optical pathway 112, and directed toward target 101. Endoscope 102 may optionally include an air / water supply nozzle (not shown) at distal end 110a.
[0062] The operating unit 107 can be configured to be held by an operator. The operating unit 107 may be disposed at a proximal end of the endoscope 102 and is configured to communicate with the endoscope controller 103 and the light source 104 via a flexible universal cord 114 extending from the operating unit 107. As shown in FIG. 1, the operating unit 107 includes a bending knob 107a for bending the bending portion 110b in the up-down and left-right directions, a treatment tool insertion port 107b for inserting a treatment tool such as medical forceps or an optical path 112 into a body cavity of a subject, and a plurality of switches 107c for operating peripheral devices such as the endoscope controller 103, the light source 104, an air supply device, a water supply device, or a gas supply device. A treatment tool such as the optical path 112 is inserted through the channel 102a from the treatment tool insertion port 107b so that its distal end is exposed from the opening 102b of the channel 102a at the distal end of the insertion portion 110 (see FIG. 2).
[0063] The endoscope controller 103 can control the operation of one or more elements of the system 100, such as the light source 104, the laser device 106, or the display 108, which displays an image of the target 101 based on the image or video signal sensed by the image sensor 116. The distal end of the endoscope 102 can be positioned and oriented such that the aiming beam 240b is directed to a target location within the field of view (FOV) of the imaging system 115, and the endoscope image includes a footprint of the aiming beam 240b. Although the aiming beam 240b is shown as a laser beam emitted from a laser energy source, other light sources can be used to generate an aiming beam that travels along an optical fiber. The endoscope controller 103 can apply image processing to the endoscope image and reconstruct a map of the target by integrating multiple endoscope images. In some examples, the endoscope controller 103 can use the reconstructed target map to localize and track the tip of the endoscope during an endoscopic procedure. Examples of the endoscope controller 103, including endoscopic mapping of targets and tracking of endoscope position, are discussed below with reference to, for example, FIG. 3.
[0064] The universal cord 114 includes an illumination fiber, a cable, etc. The universal cord 114 may be branched at its proximal end. One end of the branched end is a connector 114a, and the other proximal end of the branched end is a connector 114b. The connector 114a is detachable from a connector of the endoscope controller 103. The connector 114b is detachable from the light source 104. The universal cord 114 propagates illumination light from the light source 104 to the distal end 110a via the connector 114b and the illumination guide 120. Furthermore, the universal cord 114 can transmit an image or video signal captured by the imaging system 115 to the endoscope controller 103 via a signal line 124 (see FIG. 2) in the cord and via the connector 114a. The endoscope controller 103 performs image processing of the image or video signal output from the connector 114a and controls at least a part of the components constituting the system 100.
[0065] The light source 104 can generate illumination light while the endoscope 102 is being used in a procedure. The light source 104 can include, for example, a xenon lamp, a light emitting diode (LED), a laser diode (LD), or any combination thereof. In one example, the light source 104 can include two or more light sources that emit light with different illumination characteristics, referred to as illumination modes. The light source 104 emits light under the control of the endoscope controller 103 and provides it to the endoscope 102 connected via the connector 114b and the illumination guide of the universal cord 114 as illumination light within the subject's body. The illumination mode can be a white light illumination mode or a special light illumination mode, such as a narrow band imaging mode, an autofluorescence imaging mode, or an infrared imaging mode. The special light illumination can concentrate and intensify light of a particular wavelength, for example, to better visualize the superficial microvasculature and mucosal surface structure, to highlight subtle contrasts of mucosal irregularities.
[0066] The display 108 may include, for example, a liquid crystal display, an organic electroluminescence display, or the like. The display 108 may display information including endoscopic images of a target subject via a video cable 108a for image processing by the endoscope controller 103. In some examples, one or more of the endoscopic images may each include a footprint of the aiming beam 240b. By operating the endoscope 102 while viewing the images displayed on the display 108, the operator may observe and track the behavior of the endoscope within the subject's body.
[0067] The laser device 106 is for use with an optical pathway 112, such as a laser fiber. With reference to FIG. 2, the laser device 106 can include one or more energy sources, such as a first energy source 202 and a second energy source 204, to generate laser energy coupled to a proximal end of the optical pathway 112. In one example, a user can select the energy source, such as via a button 106a (see FIG. 1) or foot switch (not shown) on the laser device 106, through software or a user interface on the display 108, or other input, manual or automatic, as known in the art.
[0068] The first energy source 202 may be optically coupled to the optical path 112 and configured to deliver a treatment beam 240a to the target 101 through the optical path 112. By way of example and not limitation, the first energy source 202 may include a thulium laser used to generate laser light for delivery to the target tissue through the optical path 112 to operate in different treatment modes, such as a cutting (ablation) mode and a coagulation (hemostasis) mode. Other energy sources known in the art for such treatment of tissue, or any other treatment mode, such as Ho:YAG, Nd:YAG and CO2, and others known in the art, may also be used for the first energy source 202.
[0069] The second energy source 204 may be optically coupled to the optical path 112 and configured to direct the aiming beam 240b through the optical path 112 to the target 101. Although the aiming beam 240b is shown as a laser beam emitted from a laser source, other light sources may be used to generate the aiming beam 240b traveling along the optical fiber. The aiming beam 240b may be emitted when the target is illuminated by the illumination light 230. In some examples, the second energy source 204 may emit at least two different aiming beams, where a first aiming beam has at least one characteristic that is different from the second aiming beam. Such different characteristics may include wavelength, power level, and / or emission pattern. For example, the first aiming beam may have a wavelength in a range of 500 nm to 550 nm, and the second aiming beam may have a wavelength in a range of 635 nm to 690 nm. The characteristics of the different aiming beams may be processed by the endoscope controller 103 and selected based on the visibility of the aiming beam in the image displayed on the display 108 under a particular illumination mode provided by the light source 104.
[0070] The laser device 106 may include a controller 206 with hardware, such as a microprocessor, that controls the operation of the first energy source 202 and the second energy source 204. In the example shown in FIG. 2, in response to the illumination light 230, light reflected from the target 101 may enter the optical path 112 from the distal end 112a. The optical path 112 configured to transmit the laser beam may also be used as a path for returning the reflected light to the laser device 106. The splitter 205 may collect the reflected light and split it from the laser beam delivered to the target 101 via the same optical path 112. The laser device 106 may include a spectrometer 208 operably coupled to the splitter 205 and configured to detect the light reflected from the splitter. Alternatively, the reflected light may be guided through an optical path (e.g., an optical fiber) separate from the optical path 112. The spectrometer 208 may be operably coupled to a dedicated optical path and detect the light reflected therefrom.
[0071] The spectrometer 208 can measure one or more spectroscopic characteristics from the sensed reflected signal. Examples of the spectrometer 208 can include a Fourier transform infrared (FTIR) spectrometer, a Raman spectrometer, a UV-VIS spectrometer, a UV-VIS-IR spectrometer, or a fluorescence spectrometer, among others. The spectroscopic characteristics can include characteristics such as reflectance, reflectance spectrum, absorption index, etc. The spectroscopic characteristics can indicate a structure category (e.g., anatomical tissue or stone) or a particular structure type that is indicative of the chemical composition of the target.
[0072] 3 is a block diagram illustrating an example of an endoscope controller 103 for use in system 100. Endoscope controller 103 includes hardware, such as a microprocessor, for performing operations in accordance with various examples described herein. Endoscope controller 103 may include a device controller 310, a video processor 320, an endoscope tracking device 330, and a memory 340. Device controller 310 may control the operation of one or more components of system 100, such as endoscope 102, display 108, light source 104, or laser device 106.
[0073] The video processor 320 may receive image or video signals from the imaging system 115 via signal line 124 and process the image or video signals to generate images or video frames that may be displayed on the display 108. In some examples, multiple endoscopic images (e.g., video frames) may be generated and displayed on the display 108. Multiple endoscopic images may be taken at the same site of the target 101 when the tip of the endoscope and the imaging system 115 remain stationary, while the distal end 112a of the optical path 112 (e.g., laser fiber) moves and may direct the laser beam to different locations on the target. In some examples, multiple endoscopic images of the same target site may be taken at different times. The later taken images may be registered with previously taken images of the same target site, such as through a process of image translation and / or image registration. The registered images may be used to determine changes in tissue conditions at the target site. Additionally or alternatively, multiple endoscopic images may be taken at different sites of the target 101, such as when the distal end of the endoscope pans across the target 101. During endoscope panning, the distal end of the endoscope is panned to various endoscope positions {L1, L2, ..., L N} (i.e., at the distal end of the endoscope). The imaging system 115, under the control of the endoscope controller 103, can be moved to and positioned at each of the target sites {S1, S2, ..., S N}, a sequence of images (or video frames) {G1, G2, ..., G N For example, an endoscopic image G of a target region Si within the FOV of the imaging system 115 can be captured. i The lens system 118 is positioned at the endoscope position L i The distal end of the endoscope may be photographed when positioned and oriented at different endoscope positions L j When the camera is moved to another endoscopic image G j , different target sites S that fall within the FOV of the imaging system 115 jThe video processor 320 may then process the resulting images {G1, G2, ..., G3, ...} to generate a map of the target 101, such as according to various examples further described below with reference to Figures 4A-4F and 5. N} can be integrated.
[0074] As described above, the tip of the endoscope may be positioned and oriented such that the aiming beam 240b falls within the FOV of the imaging system 115, and the aiming beam footprint may be captured in the endoscopic image (e.g., Gi). In one example, the video processor 320 may color the aiming beam footprint a different color than the background of the endoscopic image. The video processor 320 may identify where the aiming beam 240b is currently illuminating by matching the color of the aiming beam 240b to the color of a pixel in the endoscopic image.
[0075] The video processor 320 may include a landmark detector 321 configured to detect one or more landmarks from the endoscopic image. The landmarks may be created manually by the operator or may be automatically identified using image processing algorithms. In one example, the landmark detector 321 may detect landmarks based on changes in the brightness of pixels in the endoscopic image. In one example, the landmark detector 321 may detect landmarks using edge detection constrained by a minimum contrast threshold and a number of pixels between similar positive and negative contrast gradients. The detected landmarks may indicate blood vessels. Edge detection may include detecting a light-to-dark transition in pixel brightness indicating the start of a vessel segment, and a subsequent dark-to-light transition in pixel brightness indicating the end of the vessel segment. Additional criteria may be applied to confirm the detection of a vessel. For example, an edge defined by a transition between a light pixel and a dark pixel may be used as a landmark if the subsequent transition occurs at least in a user-defined number of dark pixels and is bounded on both sides by a threshold number of light pixels, so long as those points along one or both edges continue for a length at least longer than another threshold. In another example, a detected edge is identified as a vessel if a linear regression of the pixels of the edge produces a line with an R-squared or other goodness of fit greater than a target threshold, such as 0.8 in the example.
[0076] Landmarks may have different forms in the endoscopic map. In one example, a landmark may be represented in the endoscopic image as a line segment. In another example, a landmark may be represented in the endoscopic image as two or more line segments that intersect at a point, called a crossing landmark or a point landmark. In some examples, for two non-intersecting, non-parallel line segments that are close to each other (such as within a certain distance range), the landmark detector 321 may algorithmically extend one line segment until it intersects with the other line segment to create a point landmark.
[0077] The landmark detector 321 may locate the landmarks relative to the aiming beam footprint in the coordinate system of the endoscopic image. For example, the location of the point landmark in the endoscopic map may be represented by a vector between the point landmark and the aiming beam footprint, or a distance along the X and Y axes in the coordinate system. In some examples, the landmark detector 321 may determine spatial relationships between the landmarks, such as distances and gradients between the landmarks, in the coordinate system of the endoscopic image. According to various examples, information regarding the landmarks and their locations, the aiming beam footprint, and the spatial relationships between the landmarks are stored in the memory 340 and used for endoscopic image registration, target map reconstruction, or endoscope tracking during an endoscopic procedure, according to various examples described herein.
[0078] The landmark detector 321 may select a subset of the detected landmarks for storage in memory 340 or for applications such as image registration, target map reconstruction, or endoscope tracking. In one example, the subset of landmarks may be selected based on the location of the landmarks, such as the spatial distribution of the landmarks in the endoscope image. For example, landmarks that are spread throughout the endoscope image may be selected more favorably than a cluster of closely packed landmarks in the endoscope image. In another example, the subset of landmarks may be selected based on whether laser energy is activated at the target site where the landmarks are located. Since laser energy may affect the accuracy and consistency of landmark detection, in one example, a landmark that is not activated by laser energy may be selected more favorably than another landmark that is activated by laser energy.
[0079] In some examples, endoscope controller 103 may control light source 104 or illumination lens 122 to generate special lighting conditions of target 101 to improve landmark detection and localization. For example, light source 104 may provide blue or green illumination to increase contrast in an endoscopic image of target 101 and better define vasculature that is less likely to move or change over time. This allows for more consistent landmark detection and localization under slightly different lighting conditions. In one example, endoscope controller 103 may temporarily change the lighting, such as turning on a green or blue light source to best identify landmarks, and then return to normal lighting mode after the landmarks have been identified.
[0080] The video processor 320 may include a target identifier 322 configured to identify a target type at the aiming beam location of the target 101. In one example, the identification of the target type may be based on one or more spectral characteristics of the illumination light reflected from the target 101. The spectroscopic characteristics may be measured using the spectrometer 208. The identified target type may include an anatomical tissue type or a stone type. Examples of stone types may include stones or stone fragments in various stone formation areas such as the urinary system, gallbladder, nasal passages, gastrointestinal tract, stomach, or tonsils. Examples of anatomical tissue types may include soft tissues (e.g., muscle, tendons, ligaments, blood vessels, fascia, skin, fat, and fibrous tissue), hard tissues such as bone, connective tissues such as cartilage, among others. In one example, the target identifier 322 may identify the tissue type at the aiming beam location of the target 101 as normal and abnormal tissue, or mucosa or muscle tissue based on the characteristics of the reflected illumination signal.
[0081] The video processor 320 may mark the aiming beam footprint in the endoscopic image as displayed on the display 108 with a visual identifier indicative of the identified tissue type. In one example, the visual identifier may include a color code such that the aiming beam footprint may be colored with different colors to indicate different tissue types. For example, the aiming beam footprint may be colored green if the target site is recognized as normal tissue or red if the target site is recognized as abnormal tissue (e.g., cancer). In one example, the video processor 320 may mark the aiming beam footprint with a visual identifier indicative of a change in tissue type over time at the target site (e.g., from normal to abnormal or vice versa), such as by using a color different from the color representing normal or abnormal tissue. In another example, the video processor 320 may mark the aiming beam footprint with a visual identifier indicative of a treatment status at the target site. For example, if the target site is being treated (e.g., with laser treatment), the aiming beam footprint may be represented by a dot of a color different from the color representing normal or abnormal tissue.
[0082] 4A-4F illustrate, by way of example and not limitation, a sequence of endoscopic images (e.g., video frames) {G1, G2, ..., G3, G4, ..., G5, ...} captured at a target 101, such as the interior of a kidney, bladder, urethra, or ureter, among other anatomical structures of interest. N}. The endoscopic images may be displayed on the display 108. As mentioned above, the endoscopic images may be taken at different times at the same target site, or may be taken at different target sites {S1, S2, ... S} as the tip of the endoscope pans across the target 101. N}. The target identifier 322 may be a set of target regions {S1, S2, ... S N}, the video processor 320 may mark the aiming beam footprint in the corresponding endoscopic image with a respective visual identifier (e.g., color) that identifies the corresponding target type, detect and locate landmarks from the endoscopic image, and integrate the endoscopic image into a target map of the target 101 based on the landmarks identified from the endoscopic image.
[0083] As shown in Figures 4A-4F, endoscopic images 410-460 are generated as the tip of the endoscope pans across the target 101 while the distal end of the endoscope is moved manually or automatically to different endoscopic positions. Figure 4A shows an endoscopic image 410 that includes a graphical representation of an illuminated target site S1 that falls within the FOV of the imaging system 115 and a circular aiming beam footprint 412. The aiming beam footprint 412 is colored green to indicate that the tissue at the aiming beam location is normal tissue. In this example, an image 413 of the distal end 112a of the optical path 112 (e.g., laser fiber) and an image 414 of the distal portion of the endoscope 102 are also shown in the image 410.
[0084] The image 410 also includes landmarks 415A-415C, such as those detected by the target identifier 322. In this example, the landmarks 415B and 415C are each represented by two line segments that intersect to form a point landmark, and the landmark 415A is represented by two line segments that algorithmically intersect (e.g., by projecting one line segment towards the other) to form the point landmark. As described above with reference to FIG. 3, the locations of the landmarks 415A-415C may be determined by the target identifier 322. The image 410, including information regarding the aiming beam footprint 412 and the landmarks 415A-415C, may be stored in the memory 340.
[0085] When the distal end of the endoscope is moved manually or automatically to a new endoscope position, another endoscope image 420 may be generated, as shown in FIG. 4B. The new endoscope image 420 includes a graphical representation of the new illuminated target site S2 corresponding to the new endoscope position, and a new aiming beam footprint 422. The aiming beam footprint 422 is colored green because the tissue in the current aiming beam is recognized as normal tissue. New landmarks may be included in the image if detected from the current endoscope image. In this example, no new landmarks are detected from the endoscope image 420. The previous aiming beam footprint 412 and previously generated landmarks 415A-415C may be retained in the current image 420 if they are located within the FOV of the imaging system at the current endoscope position.
[0086] If the movement of the distal tip is performed with a small step size, the illuminated target sites S1 and S2 may overlap and the two endoscopic images 410 and 420 may both cover a common area of the target 101, as shown in Figures 4A and 4B. One or more matching landmarks may be identified from the endoscopic images 410 and 420. Such matching landmarks may be used to align the images 410 and 420 to reconstruct a map of the target, according to various examples described below.
[0087] The endoscope panning process may continue and additional endoscope images may be generated. FIG. 4C shows image 430 including a graphical representation of the new illuminated target site S3, with the new aiming beam footprint 432 colored red indicating that abnormal tissue has been recognized at the current aiming beam position. New landmarks 435A-435B may be detected from the current endoscope image. The previous aiming beam footprint and previously generated landmarks (e.g., 415A-415C) are still within the FOV of the imaging system at the current endoscope position and may be retained in image 430.
[0088] 4D shows an image 440 including a graphical representation of the new illuminated target site S4, with the new aiming beam footprint 442 colored green indicating that normal tissue is recognized at the current aiming beam position. New landmarks 445A-445C may be detected from the current endoscope image. The new aiming beam footprint (including its position and color representing tissue type) and the new landmarks (including their positions relative to the aiming beam footprint), as well as previous aiming beam footprints and previously generated landmarks, may be stored in memory 340. Previous aiming beam footprints and previously generated landmarks (e.g., 435B) that are within the FOV of the imaging system at the current endoscope position may be retained in the image 440.
[0089] The distal end of the endoscope may be moved manually or automatically along a specific path or according to a specific pattern, such that the endoscopic images generated during the panning process may together provide panoramic coverage of a substantial surface area of the target 101. As a non-limiting example, FIGS. 4A-4F show a rectangular path indicated by the aiming beam footprint in the corresponding endoscopic images. After a leftward horizontal movement (as shown in FIGS. 4A-4D), the distal end of the endoscope moves vertically upwards while endoscopic images may be taken at each target site. FIG. 4E shows an image 450 including a graphical representation of a new illuminated target site S5, with the new aiming beam footprint 452 colored green indicating that normal tissue has been recognized at the current aiming beam position. New landmarks 455A-455B may be detected from the current endoscopic image. Previous aiming beam footprints and previously generated landmarks within the FOV of the imaging system at the current endoscopic position are retained in the image 450.
[0090] After the upward vertical movement, the distal end of the endoscope makes a rightward horizontal movement during which endoscopic images may be taken at each target site. FIG. 4F shows image 460 including a graphical representation of the illuminated target site S6, which includes a portion of the previously visited illuminated site captured in image 410, with a new aiming beam footprint 462 colored green indicating that normal tissue is recognized at the current aiming beam position. New landmarks 465A-465B may be detected from the current endoscopic image. Previous aiming beam footprints and previously generated landmarks that are within the FOV of the imaging system at the current endoscope position are retained in image 460. This includes previously generated landmark 435A, once outside of endoscopic images 440 and 450.
[0091] Returning to FIG. 3, the video processor 320 receives a plurality of endoscopic images (or video frames) {G1, G2, . . . , G3, G4, G5, G6, G7, G8, G9, G10, G11, G12, G13, G14, G15, G16, G17, G18, G19, G20, G21, G22, G23, G24, G25, G26, G27, G28, G29, G30, G31, G32, G33, G34, G35, G36, G37, G38, G39, G40, G41, G42, G43, G44, G45, G46, G47, G48, G49, G50, G51, G52, G53, G54, G55, G56, G57 N 4A to 4F, the stored endoscopic image G i may include a graphical representation of the illuminated target site, the aiming beam footprint (including its location and a color representative of tissue type), and one or more landmarks (including their locations relative to the aiming beam footprint and the spatial relationship between the landmarks). The target map generator 323 maps the stored endoscopic images {G1, G2, ..., G N} with their relative positions. Image registration may be performed to register a second endoscopic image (e.g., a second target site S j Image G taken at j ) that matches two or more landmarks identified from the first endoscopic image (e.g., a first target site S i Image G taken at i), and aligning the second image to the first image with respect to the identified matching landmarks. For example, image 430 of FIG. 4C may be aligned with image 420 of FIG. 4B using matching landmarks 415A-415C present in both images 420 and 430. The aligned images can then be stitched together with respect to the matching landmarks to reconstruct the target map. In some examples, the landmark detector 321 may adjust the landmark detection algorithm (e.g., lower the edge detection threshold) to allow more landmarks to be identified from the endoscopic image. Multiple landmarks can increase the likelihood of identifying matching landmarks between images and improve the accuracy of the image alignment.
[0092] 5 shows an example of a target map 500 of a target 101, such as a substantial region of the bladder. The reconstructed map in addition to the stitched image may further include one or more of a set of landmarks identified from multiple endoscopic images, an aiming beam footprint, a target type identifier (e.g., a color code of the aiming beam footprint), a location of the landmark relative to the aiming beam footprint, or a spatial relationship between the landmarks. The target map 500 may be used to aid in medical diagnosis or treatment planning, such as locating and tracking an endoscope during an endoscopic procedure.
[0093] Geometric distortions or deformations may be introduced into the endoscopic images used to reconstruct the target map (e.g., endoscopic images 410-460 in FIGS. 4A-4F used to reconstruct target map 500 in FIG. 5), resulting in mismatches in common areas between the images. For example, moving the endoscope tip closer to or farther from the target, or bodily movements (e.g., breathing), may cause the images to expand or contract. Changes in the viewing direction (the direction from the imaging system 115 at the distal end of the endoscope toward the target) may cause the image to rotate. In some cases, changes in the orientation of the endoscope may cause geometric distortions or deformations. Herein, the orientation of the endoscope refers to the tilt or distortion of the tip of the endoscope relative to the target plane. Changes in the orientation of the endoscope from one image to another may cause distortions in length, shape, and other geometric properties. To correct for such distortions or deformations, in some examples, the target map generator 323 may transform an image before aligning it with another image. Examples of image transformations may include one or more of scaling, translating, rotating, or shearing the image in a coordinate system, among other rigid, similarity-based, or affine transformations. In one example, the image may be scaled by a scaling factor based on landmark-to-landmark distances measured from the two images, respectively, as described below with respect to FIG. 7, or based on geometric features measured from the aiming beam footprints in the two images, respectively. In one example, changes in endoscope orientation may be corrected as described below, for example, based on landmark-to-landmark gradients measured from the two images, respectively, or based on geometric features measured from the aiming beam footprints in the two images, respectively, as described below with respect to FIG. 6A-6F.
[0094] The transformation may be implemented as a transformation matrix multiplied by the image data (e.g., a data array). The transformed image may be aligned with another image with respect to landmarks that match between the two images. In some examples, the alignment may be based on the gradient of multiple landmarks relative to each other. Such alignment may be insensitive to the distance between the landmarks (with respect to differences in magnification or distance of the endoscope from the target). Image transformations according to various examples described herein may improve the robustness of the target map reconstruction to differences in rotation, magnification, or shrinkage of the endoscopic image.
[0095] The landmarks in the transformed endoscopic images may often be stored in the memory 340 in a transformed state as if they were on the two-dimensional projection surface of the target. Thus, the landmarks in the transformed endoscopic images are invariant to surface unevenness, warping, rotation, scale, among other distortions or deformations. The stored transformed endoscopic images may be integrated to form an integrated target map. The stored landmarks may serve as a basis for comparison of a new image or as a basis for transforming a new image to the two-dimensional projection surface and registering the new image to the stored target map, for example, as described below with reference to FIG. 7.
[0096] 6A-6F are diagrams illustrating the effect of endoscope orientation on endoscopic image characteristics and methods for correcting for different endoscope orientations between two endoscopic images. The endoscope orientation correction methods discussed herein may be applied to image registration applications, such as registering real-time intraoperative endoscopic images to a target map (such as target map 500), as described below with reference to FIG. 7. Endoscope orientation refers to the tilt or skew angle θ of lens system 118 relative to the target plane. Two endoscopic images G taken with different endoscopic orientations i and G jFor G, image properties such as landmark locations (e.g., distance to the aiming beam footprint) and spatial relationships between landmarks (e.g., distance between landmarks) are measured in the coordinate systems of each of the two images. i and G j are endoscopic images of the same target region, the difference in the orientation of the endoscope can be corrected to obtain the endoscopic image G i and G j The image characteristics of the two images are measured in the same coordinate system. The assessment of anatomical differences or similarities based on image characteristics (e.g., distance between landmarks) between two images is more robust to different image conditions. i and G j are endoscopic images of different target sites (such as two of the images in Figures 4A to 4F during panning of the endoscope), the endoscopic images G can be corrected by correcting for such differences in the orientation of the endoscope. i and G j (as part of the target map 500) i and G j Integration with may provide a more reliable representation of the extended surface area of the target 101.
[0097] FIG. 6A illustrates a first endoscopic orientation θ1 where the tip of the endoscope 102 is perpendicular to the surface of the target site 611 (i.e., θ1=90 degrees) and the lens system 118 is parallel to the target site 611. An endoscopic image 615 taken at the endoscopic orientation θ1 is shown in FIG. 6C. FIG. 6B illustrates a second endoscopic orientation θ2 where the tip of the endoscope 102 is tilted relative to the target site 621 (i.e., θ2 is an acute angle) and the lens system 118 is not parallel to the target site 621. An image 625 taken at the endoscopic orientation θ2 is shown in FIG. 6D. Matching landmarks {M1, M2, M3}, such as the shape of intersecting line segments, can be identified by the landmark detector 321 from the endoscopic images 615 and 625.
[0098] The target map generator 323 may use the generated features from images 615 and 625, respectively, to detect a change in the endoscope orientation from θ1 to θ2, transform the endoscope image 625 to correct for the change in the endoscope orientation, and align the transformed image 625 with the image 615 with respect to the matching landmarks {M1, M2, M3}. As shown in FIG. 6C and FIG. 6D, due to the difference in the endoscope orientation, the inter-landmark spatial relationship (e.g., distance and relative position) between the landmarks {M1, M2, M3} in the image 615 may appear different than the relative position between the landmarks {M1, M2, M3} in the image 625. In one example, the inter-landmark spatial relationship is the gradient k between the landmarks M1 and M3. 13 etc., can be represented by the gradient between two landmarks in the coordinate system, which is the distance y 13 and the distance x between M1 and M3 on the x-axis 13 The ratio of 13 =y 13 / x 13 To determine the change in the orientation of the endoscope, the target map generator 323 calculates a first gradient (e.g., k 13 =y 13 / x 13 ) and a second gradient between the same two landmarks in image 625 (e.g., k 13 '=y 13 ' / x 13 In the illustrated example, k 13 and k 13 A relative gradient, such as the ratio between ' and ', can indicate a change in the orientation of the endoscope.
[0099] Additionally or alternatively, in some examples, the target map generator 323 may detect changes in endoscope orientation using geometric features generated from the aiming beam footprint in images 615 and 625, respectively. FIGS. 6A and 6B show the distal end 112a of a laser fiber (one example of a light path 112) directing an aiming beam to a respective target site. The resulting aiming beam footprints 612 and 622 have different geometric characteristics due to differences in endoscope orientation, as shown in the respective endoscope images 615 and 625. Corresponding to an endoscope orientation θ1=90°, FIG. 6E shows a circular aiming beam footprint 612 with a diameter d. Corresponding to an endoscope orientation θ1<90°, FIG. 6F shows an elliptical shaped aiming beam footprint 622 with a major axis 623 of length “a” and a minor axis 624 of length “b”. In one example, the target map generator 323 may use an ellipse axis length ratio R e = a / b may be used to determine the orientation of the endoscope. For an elliptical footprint 622, the ratio of the lengths of the elliptical axes is R e >1. A larger ratio of the lengths of the ellipse axes indicates a more tilted orientation of the endoscope. For a circular footprint 612 of diameter d, the major and minor axes are a=b=d, and the ratio of the lengths of the ellipse axes is R e = 1. The target map generator 323 may determine a change in endoscope orientation based on a comparison between the ellipse axis length ratios calculated from aiming beam footprints 612 and 622, respectively, transform the endoscope image 625 to correct for the change in endoscope orientation, and align the transformed image 625 with image 615 with respect to the identified matching landmarks.
[0100] Returning to FIG. 3, the endoscope tracking device 330 may locate and track the tip of the endoscope during an endoscopic procedure using a pre-generated target map (e.g., the target map 500 shown in FIG. 5). Endoscope tracking may begin with capturing real-time images or video signals from a treatment site of the target 101 using the imaging system 115 and generating real-time images or video frames using the video processor 320, similar to what has been described above with respect to generating an endoscopic image (e.g., one of those shown in FIGS. 4A-4F) for reconstructing the target map 500. The imaging system 115 may be positioned at an unknown endoscopic location. The landmark detector 321 may identify one or more landmarks from the real-time image. The endoscope tracking device 330 may register the real-time image to a pre-generated target map (e.g., the target map 500) of the target 101 and locate the site captured in the real-time image from the target map.
[0101] The endoscope tracking device 330 may determine changes in tissue conditions at the target site (e.g., from normal tissue to abnormal tissue or vice versa). The endoscope tracking device 330 may locate and track the tip of the endoscope during the procedure based on landmarks identified from the real-time image and stored landmarks associated with the target map, etc. In one example, the endoscope tracking device 330 may recognize two or more matching landmarks between landmarks of the target map and landmarks of the real-time image, register the real-time image to the target map using the recognized matching landmarks, and locate and track the tip of the endoscope based on the registration of the real-time image.
[0102] 7 illustrates an example of recognizing matching landmarks between a real-time image 710 and a reconstructed target map 500 and registering the real-time image 710 to the target map 500 with respect to the matching landmarks. In one example, the matching landmarks may be recognized based on the distance ratio (r) between the landmarks. As discussed above with reference to FIG. 5, the target map 500 is reconstructed based on the pairwise inter-landmark distances {d1, d2, d3, ..., d K}, where K represents the number of landmark pairs identified from the target map 500. The distance ratios {r1, r2, ..., rM} are the sum of the K inter-landmark distances {d1, d2, d3, ..., d K}, where M represents the number of distance ratios.
[0103] According to an example, to recognize a matching landmark, the endoscope tracking device 330 may identify an intersection landmark 711 that is closest to the current aiming beam footprint 701 in the real-time image from among a set of intersection landmarks (i.e., intersection line segments) having respective intersection locations. Distances from the landmark 711 to other landmarks in the image 710 may be measured, where D1 is the distance to Pc, D2 is the distance to Pd, D3 is the distance to Pb, D4 is the distance to Pa, etc. The endoscope tracking device 330 may then calculate a distance ratio (R) between distances originating from the same landmark, such as the landmark 711 in this example, R1=D1 / D2, R2=D1 / D3, R3=D1 / D4, etc. The distance ratios {R1, R2, R3} may be compared to distance ratios {r1, r2, ..., rM} associated with the target map 500. The distance ratios {R1, R2, R3} (corresponding to the original landmarks 711 in the real-time image 710) are compared with the distance ratios {rx, ry, rz} (corresponding to the original landmarks P in the target map 500). k, R1=rx, R2=ry, and R3=rz, then the distances {D1, D2, D3, D4} are likely to match the distances {d1, d2, d3, d4}, and the landmarks {Pa, Pb, Pc, Pd} in the real-time image 710 are likely to match the landmarks {p1, p2, p3, p4} in the target map 500. The longer the possible matching distances, the higher the probability that the landmarks match between the real-time image 710 and the map 500.
[0104] The endoscope tracking device 330 may then determine a correspondence between the distances {D1, D2, D3, D4} and the distances {d1, d2, d3, d4} based on the ratio of the distances between the landmarks. For example, if D1 / D2=d1 / d2, it may determine that D1=d1 and D2=d2. By checking various combinations until all are matched, the endoscope tracking device 330 may identify which distances in {D1, D2, D3, D4} correspond to which distances in {d1, d2, d3, d4}. Because all matches were made to D1 by identifying D1 as d1, the remaining distances D2, D3, and D4 may match d2, d3, and d4. With the correspondences established between D1 and d1, between D2 and d2, between D3 and d3, and between D4 and d4, the correspondences between {Pa, Pb, Pc, Pd} and {P1, P2, P3, P4} can also be determined.
[0105] The endoscope tracking device 330 may register the real-time image 710 to the target map 500 using the identified matching landmarks {Pa, Pb, Pc, Pd} in the real-time image 710 that match the landmarks {P1, P2, P3, P4} in the target map 500. To correct for geometric distortion or deformation of the image, such as due to image zoom, zoom, rotation, or changes in endoscope orientation, the endoscope tracking device 330 may transform the real-time image 710 or the target map 500 in a manner similar to that described above with respect to the transformation of the first endoscopic image, align it with the second endoscopic image, and reconstruct the panoramic target map using at least the transformed first and second images, as described above with reference to Figures 4A-4F. The transformation may include one or more of scaling, translation, or rotation, among other operations. The transformation may be implemented as a transformation matrix multiplied by the image data (e.g., data array) of the target map 500 in the coordinate system of the real-time image 710. Alternatively, the transformation may be applied to the real-time image 710.
[0106] As shown in FIG. 7, the transformation may include scaling the map 500 by a scale factor λ to compensate for different image magnification or shrinkage between the real-time image 710 and the map 500. The scaled map 720 includes the landmarks with their positions (e.g., relative distance to the aiming beam footprint) and the inter-landmark distances also scaled by the scale factor λ. In one example, the scale factor λ may be determined using a ratio of the distance between two matching landmarks (e.g., P1 and P2) in the real-time image 710 to the distance between two corresponding landmarks (e.g., Pa and Pb) in the map 500. In one example, the largest inter-landmark distance among the matching landmarks in the real-time image 710 may be selected to calculate the distance ratio λ. For example, if D4 is the largest distance among {D1, D2, D3, D4}, then the scale factor λ=D4 / d4.
[0107] The magnification λ may alternatively or additionally be determined using a comparison between the shape of the aiming beam footprint in real-time image 710 and the shape of the aiming beam footprint in map 500. In one example, the magnification λ may be determined using a ratio of a geometric feature of the aiming beam footprint in real-time image 710 to a corresponding geometric feature of the aiming beam footprint in map 500. For example, if real-time image 710 is a square with a diameter d R 6E), and the map 500 has a circular footprint of diameter d M If the circular footprint of R / d M In one example, the real-time image 710 is a R and the length of the minor axis is b R 6F, and the map 500 has a major axis length of a M and the length of the minor axis is b M If the elliptical footprint of the R / a M , or λ=b R / b M It is.
[0108] The magnification λ calculated above assumes that the surface onto which the aiming beam is projected is flat. In some cases, the aiming beam projection surface is not perfectly flat, but instead has a three-dimensional shape. This may result in a change in the calculated magnification λ. The system may accommodate changes from the ideal magnification λ. In one example, multiple aiming beam footprints may be captured when the aiming beam is directed at slightly different locations of the target site at each projection surface. Overlapping the multiple aiming beam footprints may reveal shape changes in the aiming beam footprint. Geometric features (e.g., diameter of a circular footprint or length of major or minor axis of an elliptical footprint) may be measured from the multiple aiming beam footprints, respectively, and corresponding multiple magnifications may be calculated. An average or weighted average of the multiple magnifications may be performed to obtain an expected value of the magnification λ.
[0109] The determination of the magnification factor λ described above is based on the assumption that the endoscope orientation does not change substantially between the real-time image 710 and the map 500 (e.g., the skew angle of the endoscope tip relative to the target plane is substantially the same). If there are substantially different endoscope orientations, metrics such as the location of landmarks, the distance between landmarks, the shape of the aiming beam footprint, and their geometric properties (e.g., the length of the major and minor axes) may be affected by the endoscope orientation. The real-time image 710 or the map 500 may be transformed to correct for changes in the endoscope orientation, such as in accordance with the description with respect to Figures 6A-6F. The magnification factor λ may then be determined from the transformed images.
[0110] As shown in FIG. 730, the scaled map 720 (including landmarks therein) may be aligned to the target map 500 with respect to the identified matching landmarks such as P1 and Pa. The real-time image 710 may be translated toward the scaled map 720 such that Pa is at the same coordinate as P1 on the scaled map 720, denoted by P1(Pa). The scaled map 720 may then be rotated clockwise by an angle α, i.e., ∠Pb-P1(Pa)-P2 (730). After the rotation, the matching landmark Pb is at the same coordinate as P2 on the scaled map 720, denoted by P2(Pb), as shown in the registered map 740. Because the scaling and rotation operations preserve the relative positions (e.g., angles) between the landmarks, the other matching landmarks P3 and P4 also overlap with the landmarks Pc and Pd on the scaled map 720, denoted by P3(Pc) and P4(Pd) in the registered map 740. Thus, the real-time image 710 is registered to the target map 500 with respect to the matching landmarks P1-P4 (corresponding to Pa-Pd in the image 710).
[0111] As described above, registration of real-time images taken during an endoscopic procedure to a target map may be used in various applications to improve the accuracy and efficiency of an endoscopic procedure. In one example, image registration may assist the operator in identifying a treatment site from a pre-generated target map in real-time with improved accuracy. Because the target map stores information of the endoscope tip position relative to a number of stored landmarks, the image registration discussed herein may aid in locating and tracking the endoscope tip in real-time throughout the procedure. In the case of a target map that stores information about target type (e.g., normal or abnormal tissue) at various aiming beam positions of the target, the endoscope tracker 330 may detect and track changes in tissue type over time at various target sites or provide an assessment of the effectiveness of a treatment delivered to the target.
[0112] 8 is a flow diagram illustrating a method 800 for endoscopic mapping of targets within a subject's body during a procedure. The method 800 may be implemented and performed by a medical system for use in an endoscopic procedure, such as the system 100 or variations thereof. Although the processes of the method 800 are depicted in a flowchart, they need not be performed in a particular order. In various examples, some of the processes may be performed in a different order than the order depicted herein.
[0113] At 810, an aiming beam may be emitted from the tip of the endoscope and directed at a portion of the target, such as a portion of the target 101. The aiming beam may be generated by a laser source, such as the second laser energy source 204. Alternatively, the aiming beam may be generated by another light source and transmitted, such as via an optical fiber.
[0114] At 820, an image of the target site may be captured by an imaging system, such as imaging system 115. The image may be taken when lens system 118 of imaging system 115 is placed in the endoscopic position. The image may be taken when the target is illuminated with electromagnetic radiation in the optical range from UV to IR (also referred to as illumination light). The illumination light may be generated by a light source, such as light source 104, and transmitted to the target site via illumination guide 120. In one example, light source 104 may include two or more light sources that emit light with different illumination characteristics.
[0115] The aiming beam directed at the target site falls within the FOV of the imaging system, and thus an image captured at the target site may include not only a graphical representation of the illuminated target (e.g., the surface of the target anatomical structure), but also the footprint of the aiming beam. The image may be displayed to a user, such as on a display 108, as shown in any of Figures 4A-4F. The aiming beam footprint may be colored using a color different from the background of the endoscopic image. In one example, the location of the aiming beam footprint may be identified from the endoscopic image by matching the color of the aiming beam to the color of a pixel in the endoscopic image.
[0116] A signal reflected from the target in response to the illumination light may be analyzed, such as using a spectrometer 208, to identify the type of target at the aiming beam location of the target. In one example, based on the spectroscopic characteristics of the reflected signal, tissue at the target site may be identified as normal and abnormal tissue, or mucosal or muscle tissue, among other anatomical structure types. In some examples, the target site may be identified as one of multiple stone types having respective compositions.
[0117] The aiming beam footprint in the endoscopic image may be marked with a visual identifier indicative of the tissue type at the target aiming beam location. In one example, the aiming beam footprint may be colored with different colors to indicate different tissue types. For example, the aiming beam footprint may be colored green if the target site is recognized as normal tissue or red if the target site is recognized as abnormal tissue (e.g., cancer). In some examples, the aiming beam footprint may be marked with an identifier to indicate a change in tissue type over time or to indicate a treatment status.
[0118] At 830, one or more landmarks may be identified from the captured image of the target site and the location of the landmarks relative to the aiming beam footprint may be determined, such as using the video processor 320. In one example, the landmarks represent anatomical structures (e.g., blood vessels). The landmarks may be detected based on changes in brightness of pixels of the endoscopic image. In one example, the landmarks may be detected using edge detection constrained by a contrast threshold and the number of pixels between similar positive and negative contrast gradients.
[0119] The location of one or more landmarks, such as x and y distances in a coordinate system of the endoscopic image, may be determined relative to the aiming beam footprint in the same endoscopic image. In another example, landmark localization includes determining inter-landmark distances in the coordinate system of the endoscopic image. In some examples, a subset of detected landmarks may be selected based on the spatial distribution of landmarks in the endoscopic image. For example, the selected subset may include landmarks that are distributed throughout the endoscopic image (as opposed to a cluster of closely spaced landmarks in one region of the image). In another example, landmarks may be selected based on whether laser energy is activated on the landmarks, since laser energy may distort such landmarks. For example, a landmark that is not activated by laser energy may be selected favorably over another landmark that is activated by laser energy.
[0120] In some examples, targets can be illuminated with special lighting conditions to improve landmark detection and localization. For example, targets may be illuminated with blue or green lighting to increase the contrast of endoscopic images of the target and better define vasculature that is less likely to move or change over time. This allows for more consistent landmark detection and localization under slightly different lighting conditions.
[0121] At 840, a target map may be reconstructed, such as by integrating multiple images of different portions of the target, based on the respective landmarks identified from the multiple images. For example, as the tip of the endoscope pans across the target, either manually by the operator or automatically by an endoscope actuator, the distal end of the endoscope moves and various endoscope positions {L1, L2, ..., L N}, and the imaging system is arranged to image a plurality of target sites {S1, S2, ..., S N}, endoscopic images {G1, G2, ..., GN}. Examples of sequences of images (or video frames) captured at different locations on a target are shown in Figures 4A-4F.
[0122] Multiple endoscopic images {G1, G2, ..., G N}, a target map can be reconstructed. N} may be aligned with respect to landmarks identified from the images. In one example, two endoscopic images G i and G j Between, Image G j Image G that matches two or more landmarks identified from i Matching landmarks may then be identified, including two or more landmarks identified from image G. i and G j can be aligned with respect to the identified matching landmarks.
[0123] In some instances, endoscopic images (e.g., G i ) is a different endoscopic image (e.g., G j ) before being aligned with the target. The transformation may correct for geometric image distortions or deformations, such as image magnification or shrinkage caused by moving the distal end of the endoscope closer to or farther from the target, body movement (e.g., breathing), image rotation caused by changes in line of sight from the imaging system towards the target, or distortions of length, shape, and other image properties caused by changes in the orientation of the endoscope. Examples of image transformations may include one or more of scaling, translating, rotating, or shearing transformations of the image in a coordinate system, among other rigid, similarity-based, or affine transformations.
[0124] In one example, the first endoscopic image G i and the second endoscopic image G j Before aligning and, j is Image G iThe distance between two of the matching landmarks in image G j In another example, the scale factor λ can be calculated based on the ratio of the distance between the corresponding landmarks in image G to the distance between the corresponding landmarks in image G. i Geometric features generated from the aiming beam footprint in image G j 7, examples of geometric features may include the diameter of a circular footprint, or the length of the major (or minor) axis of an elliptical footprint.
[0125] In one example, the first endoscopic image G i The second endoscopic image G j Before aligning with the image G i and G j Changes in the orientation of the endoscope between images G and G can be detected and corrected. i The first gradient between two of the matching landmarks in image G j In another example, a change in the orientation of the endoscope may be determined based on a comparison of the gradient between the two matching landmarks in image G with a second gradient between the two matching landmarks in image G. i The first geometric feature of the aiming beam footprint in image G j In one example, at least one of the first or second aiming beam footprints may have an elliptical shape having a major axis and a minor axis, and at least one of the first or second geometric characteristics may include a ratio of a length of a major axis to a length of a minor axis of the elliptical shaped aiming beam footprint, as described above with reference to FIGS.
[0126] The transformed images may be aligned and integrated into a reconstructed target map, such as that shown in FIG. 5. The reconstructed map may include one or more of a set of landmarks identified from the multiple endoscopic images, the aiming beam footprint generated during the tissue painting process, a target type identifier (such as a color-coded aiming beam footprint), the location of the landmark relative to the aiming beam footprint, or the relative location between the landmarks. The target map, including information regarding the landmarks and the aiming beam footprint, may be stored in memory 340. At 850, the target map may be used to locate and track the tip of the endoscope during an endoscopic procedure, as described below with reference to FIG. 9. Additionally or alternatively, the target map may be used to determine changes in tissue condition at the target site (e.g., from normal to abnormal or vice versa).
[0127] The image reconstruction and endoscopic tracking systems and methods according to various examples discussed herein can accommodate tolerances for their measurements. The comparisons or formulas described or inferred from the description herein are not limited to perfect equality. In one example, the systems and methods described herein can first compare values for perfect equality, but then gradually expand the tolerance around each calculation to identify overlaps that are then treated as equal. For example, the algorithm can gradually expand the tolerance range from ideal to marginal when comparing anatomical maps of the same patient from two different time points that may vary by weeks, months, or years, or create larger regions around each landmark, after which the overlaps are checked using standard statistical methods such as, for example, a median t-test or Mann-Whitney comparison with a target confidence level of 5-25%. In one example, the tolerance range or confidence interval can range from 0% to X% of the distance between each pair of landmarks being compared, where X% can be 20% in one example and 25% in another example. Alternatively, the tolerance or confidence interval can be gradually increased until adequate agreement is obtained between the majority of landmarks, such as when approximately 70-100% of the landmarks are found to match.
[0128] FIG. 9 is a flow diagram illustrating an example of a method 900 for endoscope tracking using a reconstructed target map, such as one generated using method 800. At 910, during an endoscopic procedure, a real-time image of a treatment site at a target site may be captured, such as via an imaging system disposed at an unknown endoscope position. At 920, matching landmarks may be identified, including two or more landmarks in the target map that respectively match two or more landmarks in the real-time image. The matching landmarks may be identified based on a distance ratio between the landmarks, as described above with reference to FIG. 7. At 930, the real-time image may be registered to the target map using the identified matching landmarks. Registration may include image transformation (e.g., translation, scaling, rotation, among others) and image registration, as described above with reference to FIG. 7. At 940, the tip of the endoscope may be located and tracked based on the registration of the real-time image. Because the target map stores information of the position of the tip of the endoscope with respect to multiple landmarks, the endoscope tracking device 330 may locate and track the tip of the endoscope in real time throughout the procedure based on the landmarks on the registered images. For a target map that stores information about tissue type (e.g., normal or abnormal tissue at different target sites as indicated by the footprint of the aiming beam), the endoscope tracking device 330 can detect and track changes in tissue type over time at various sites of the target, or the effectiveness of a treatment delivered thereto.
[0129] 10 generally illustrates a block diagram of an example machine 1000 that may perform any one or more of the techniques (e.g., methodologies) discussed herein. Portions of this description may apply to the computing framework of various portions of system 100, such as endoscope controller 103.
[0130] In alternative embodiments, the machine 1000 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a network deployment, the machine 1000 may operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In one example, the machine 1000 may function as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 1000 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be performed by the machine. Furthermore, although only one machine is shown, the term "machine" shall be construed to include any collection of machines that individually or collectively execute a set (or sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations, etc.
[0131] As described herein, examples may include or be operated by logic or many components or mechanisms. A circuit set is a collection of circuits implemented in a tangible entity that includes hardware (e.g., simple circuits, gates, logic, etc.). The membership of a set of circuit sets may be flexible over time and in response to variations in the underlying hardware. A circuit set includes members that, alone or in combination, may perform a specified operation when operated. In one example, the hardware of a circuit set may be inappropriately designed (e.g., hardwired) to perform a particular operation. In one example, the hardware of a circuit set may include various connected physical components (e.g., execution units, transistors, simple circuits, etc.) including computer-readable media that are physically modified to encode particular operational instructions (e.g., magnetically, electrically, movable arrangements of immutable mass particles, etc.). In connecting the physical components, the underlying electrical properties of the hardware components are changed from insulators to conductors or vice versa. These instructions enable the embedded hardware (e.g., an execution unit or loading mechanism) to create members of the circuits configured in the hardware through variable connections to perform some of the specific operations during operation. Thus, the computer-readable medium is communicatively coupled to other components of the circuit set members when the device is operating. In one example, any of the physical components may be used in multiple members of multiple circuit sets. For example, during operation, the execution unit may be used in a first circuit of a first circuit set at one time and reused by a second circuit in the first circuit set or by a third circuit in the second circuit set at a different time.
[0132] The machine (e.g., a computer system) 1000 may include a hardware processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1004, and a static memory 1006, some or all of which may communicate with each other via an interlink (e.g., a bus) 1008. The machine 1000 may further include a display unit 1010 (e.g., a raster display, a vector display, a holographic display, etc.), an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In one example, the display unit 1010, the input device 1012, and the UI navigation device 1014 may be touch screen displays. The machine 1000 may additionally include a storage device (e.g., a drive unit) 1016, a signal generating device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1021, such as a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensor. The machine 1000 may include an output controller 1028, such as a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection, to communicate with or control one or more peripheral devices (e.g., a printer, a card reader, etc.).
[0133] The storage device 1016 may include a machine-readable medium 1022 on which is stored one or more sets of data structures or instructions 1024 (e.g., software) that embody or are utilized by any one or more of the techniques or functions described herein. The instructions 1024 may also reside, completely or at least partially, within the main memory 1004, within the static memory 1006, or within the hardware processor 1002 during its execution by the machine 1000. In one example, one or any combination of the hardware processor 1002, the main memory 1004, the static memory 1006, or the storage device 1016 may constitute a machine-readable medium.
[0134] Although the machine-readable medium 1022 is illustrated as a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 1024.
[0135] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 1000, causing the machine 1000 to perform any one or more of the techniques of this disclosure, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory, as well as optical and magnetic media. In one example, a massive machine-readable medium comprises a machine-readable medium having a plurality of particles having an unchanging (e.g., stationary) mass. Thus, a massive machine-readable medium is not a transitory propagating signal. Specific examples of massive machine-readable media may include non-volatile memory such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPSOM)) and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.
[0136] The instructions 1024 may be further transmitted or received over a communications network 1026 using a transmission medium via the network interface device 1020 utilizing any one of a number of transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Exemplary communications networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile telephone networks (e.g., cellular networks), plain old telephone (POTS) networks, and wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as WiFi, the IEEE 802.16 family of standards known as WiMax), the IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In one example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas for connecting to the communications network 1026. In one example, the network interface device 1020 may include multiple antennas for wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" is intended to include any intangible medium capable of storing, encoding, or carrying instructions for execution by the machine 1000, including digital or analog communications signals, or other intangible media for facilitating communication of such software.
[0137] Additional Notes The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as "examples." Such examples may include elements in addition to those shown or described. However, the inventors also contemplate examples in which only the shown or described elements are provided. Furthermore, the inventors also contemplate examples that use any combination or permutation of the shown or described elements (or one or more aspects thereof) with respect to a particular example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) shown or described herein.
[0138] The terms "a" or "an" are used herein to include one or more, as is common in patent documents, regardless of any other instances or usages of "at least one" or "one or more." The term "or" is used herein to refer to a non-exclusive "or," such that "A or B" includes "A but not B," "B but not A," and "A and B," unless otherwise indicated. The terms "including" and "in which" are used herein as the plain English equivalents of the respective terms "comprising" and "wherein." Also, in the following claims, the terms "including" and "comprising" are intended to be open-ended, i.e., systems, devices, articles, compositions, formulations, or processes that include elements in addition to the elements listed after such terms in a claim, will still be considered to be within the scope of that claim. Moreover, in the following claims, the terms "first," "second," "third," etc. are used merely as labels and are not intended to impose numerical requirements on their objects.
[0139] The above description is intended to be illustrative, not limiting. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used by those of ordinary skill in the art who review the above description. The Abstract is provided to enable the reader to quickly ascertain the nature of the technical disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be construed as intending that unclaimed disclosed features are essential to the claims. Rather, inventive subject matter may lie in fewer than all features of a particular disclosed embodiment. Thus, it is contemplated that the following claims are incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. [Explanation of symbols]
[0140] 100 Medical Systems 101 Target 102 Endoscope 102a Treatment Tools Channel 102b Distal end opening 103 Endoscope Controller 104 Light source 106 Laser Device 107 Operating Unit 107a Curved knob 107b Insertion port, treatment tool insertion port 107c Switch 108 Display 108a Video Cable 110 Insertion part 110a Distal end 110b Curved portion 110c Flexible tube section 112 Light Path 112a Distal end 114 Universal Code 114a Connector 114b Connector 115 Imaging System 116 Image Sensor 118 Lens System 120 Lighting Guide 122 Lighting Lens 124 Signal Line 202 The first energy source 204 Second energy source, second laser energy source 205 Splitter 206 Controller 208 Spectrometer 230 Illumination 240a Therapeutic Beam 240b Aiming Beam 310 Device Controller 320 Video Processor 321 Landmark Detector 322 target identifier 323 Target Map Generator 330 Endoscope Tracking Device 340 Memory 410~460 Endoscopic images 410 images 412 Aiming Beam Footprint 413 images 414 images 415A Landmark 415B Landmark 415C Landmark 420 Endoscopic Images 422 Aiming Beam Footprint 430 images 432 Aiming Beam Footprint 435A~435B New Landmark 440 images 442 Aiming Beam Footprint 445A~445C New Landmark 450 images 452 Aiming Beam Footprint 455A~455B New Landmark 460 images 462 Aiming Beam Footprint 465A~465B New Landmark 500 Target Map 611 Target Area 612 Aiming Beam Footprint 615 Endoscopic Images 621 Target Area 622 Aiming Beam Footprint, Elliptical Footprint 625 Endoscopic Images 701 Aiming Beam Footprint 710 Real-time Images 711 Crossing Landmark 720 Map 730 Figures 740 Maps 800 ways 1000 machines 1002 Hardware Processor 1004 Main memory 1006 Static Memory 1008 Interlink 1010 Display Unit 1012 Alphanumeric Input Device 1014 User interface (UI) navigation devices 1016 Storage Devices 1018 Signal generating device 1020 Network Interface Device 1021 Sensor 1022 Machine-readable medium 1024 instructions 1026 Communication Network 1028 Output Controller
Claims
1. 1. A system for endoscopic mapping of a target, comprising: an imaging system configured to capture an endoscopic image of the target, the endoscopic image including a footprint of an aiming beam directed at the target; and identifying one or more landmarks from the captured endoscopic image and determining their respective positions relative to the aiming beam footprint; generating a target map by integrating the plurality of endoscopic images based on the landmarks identified from one or more of the plurality of endoscopic images; a video processor configured to A system comprising:
2. 10. The system of claim 1, wherein the processor is configured to identify a tissue type at a location of the aiming beam and mark the aiming beam footprint with a visual identifier indicative of the identified tissue type.
3. a spectrometer communicatively coupled to the video processor, the spectrometer configured to measure one or more spectral characteristics of an illumination light signal reflected from the target; 3. The system of claim 1 or 2, wherein the video processor is configured to identify a tissue type at a location of the aiming beam based on the one or more spectroscopic characteristics and to mark the aiming beam footprint with a visual identifier indicative of the identified tissue type.
4. The system of claim 2 or 3, wherein the video processor is configured to identify the tissue type as normal tissue or abnormal tissue.
5. 4. The system of claim 2 or 3, wherein the video processor is configured to mark the aiming beam footprint with different colors to indicate different tissue types.
6. 6. The system of claim 1, wherein the video processor is configured to identify the one or more landmarks from the endoscopic image based on changes in brightness of pixels of the endoscopic image.
7. The system of claim 6 , wherein the one or more landmarks are represented in the endoscopic image as line segments or intersecting line segments.
8. The video processor, selecting a subset of the landmarks from the landmarks identified from one or more of the plurality of endoscopic images based on whether laser energy is activated at each target site at which the identified landmarks are located; generating the target map by integrating the plurality of endoscopic images based on the selected subset of landmarks; The system of claim 1 , configured to:
9. the plurality of endoscopic images includes images of various portions of the target including a first endoscopic image of a first target portion captured from a first endoscopic position and a second endoscopic image of a second target portion captured from a second endoscopic position, and the video processor: identifying matching landmarks including two or more landmarks in the first endoscopic image that match corresponding two or more landmarks in the second endoscopic image; registering the first and second endoscopic images with respect to the matching landmarks in a coordinate system of the first image; generating the target map using at least the registered first and second images; The system of claim 1 , configured to:
10. The video processor, transforming the second image, including one or more of scaling, translating, or rotating the second image; aligning the transformed second image and the first image with respect to the matching landmarks; The system of claim 9 , configured to:
11. The system of claim 10 , wherein the transformation of the second image comprises matrix multiplication with a transformation matrix.
12. 12. The system of claim 10 or 11, wherein the video processor is configured to scale the second image using a scaling factor based on a ratio of a distance between two of the matching landmarks in the first image to a distance between the two corresponding landmarks in the second image.
13. 12. The system of claim 10 or 11, wherein the video processor is configured to scale the second image by a scaling factor based on a ratio of a geometric characteristic of the aiming beam footprint in the first image to a geometric characteristic of the aiming beam footprint in the second image.
14. 14. The system of claim 10, wherein the video processor is configured to transform the second image to correct for a change in endoscope orientation between the first and second images, the endoscope orientation indicating a tilt of the tip of the endoscope relative to a target site.
15. 15. The system of claim 14, wherein the video processor is configured to detect the change in endoscope orientation using a first gradient between two of the matching landmarks in the first image and a second gradient between the corresponding two landmarks in the second image.
16. 15. The system of claim 14, wherein the video processor is configured to detect the change in endoscope orientation using a first geometric characteristic of an aiming beam footprint in the first image and a second geometric characteristic of an aiming beam footprint in the second image.
17. at least one of the first or second aiming beam footprints has an elliptical shape having a major axis and a minor axis; 17. The system of claim 16, wherein at least one of the first or second geometric characteristics comprises a ratio of a length of the major axis to a length of the minor axis.
18. identifying matching landmarks from real-time images of the target treatment site captured by the imaging system from an unknown endoscopic position during an endoscopic procedure, the matching landmarks including two or more landmarks in the target map that match corresponding two or more landmarks in the real-time images; registering the real-time image to the target map using the matching landmarks; tracking a position of a tip of an endoscope based on said registration of said real-time images; 18. The system of claim 1 , comprising an endoscope tracking system configured to:
19. 20. The system of claim 18, wherein the endoscope tracking system is configured to identify the matching landmarks based on one or more ratios of distances between landmarks in the real-time image and one or more ratios of distances between landmarks in the target map.
20. 20. The system of claim 18 or 19, wherein the endoscopic tracking system is configured to generate an indication of changes in tissue type at the target site.
21. 1. A method for endoscopic mapping of a target, comprising: directing an aiming beam at a target; capturing an endoscopic image of the target via an imaging system, the endoscopic image including a footprint of the aiming beam; identifying, via a video processor, one or more landmarks from the captured endoscopic image and determining a position of each of the one or more landmarks relative to the aiming beam footprint; generating, via the video processor, a target map by integrating a plurality of endoscopic images based on landmarks identified from one or more of the plurality of endoscopic images; A method comprising:
22. identifying a tissue type at the location of the aiming beam using an illumination light signal reflected from the target; marking the aiming beam footprint with a visual identifier indicative of the identified tissue type; The method of claim 21 , comprising:
23. 23. The method of claim 21 or 22, wherein identifying the one or more landmarks from the endoscopic image is based on changes in brightness of pixels in the endoscopic image.
24. the plurality of endoscopic images includes images of various portions of the target including a first endoscopic image of a first target portion captured at a first endoscopic position and a second endoscopic image of a second target portion captured at a second endoscopic position; identifying matching landmarks including two or more landmarks in the first endoscopic image that match corresponding two or more landmarks in the second endoscopic image; registering the first and second endoscopic images with respect to the matching landmarks in a coordinate system of the first image; generating the target map using at least the registered first and second images; 24. The method of any one of claims 21 to 23, comprising:
25. The step of registering the first and second endoscopic images includes: transforming the second image including one or more of scaling, translating, or rotating the second image; aligning the transformed second image and the first image with respect to the matching landmarks; 25. The method of claim 24, comprising:
26. 26. The method of claim 25, wherein transforming the second image comprises scaling the second image by a scaling factor based on a ratio of a distance between two of the matching landmarks in the first image to a distance between the two corresponding corresponding landmarks in the second image.
27. 26. The method of claim 25, wherein transforming the second image comprises scaling the second image by a scaling factor based on a ratio of a geometric feature of an aiming beam footprint in the first image to a geometric feature of an aiming beam footprint in the second image.
28. 26. The method of claim 25, wherein transforming the second image includes correcting for a change in endoscope orientation between the first and second images, the endoscope orientation indicating a tilt of a tip of an endoscope relative to a target site.
29. capturing real-time images of the target treatment site using the imaging system from an unknown endoscopic position during an endoscopic procedure; identifying matching landmarks, including two or more landmarks in the target map that match corresponding two or more landmarks in the real-time image; registering the real-time image to the target map using the matching landmarks; tracking a position of a tip of an endoscope based on said registration of said real-time images; 29. The method of any one of claims 21 to 28, further comprising:
30. 30. The method of claim 29, wherein identifying matching landmarks is based on one or more ratios of distances between landmarks in the real-time image and one or more ratios of distances between landmarks in the target map.
Citation Information
Patent Citations
Optical tomographic image obtaining apparatus and optical tomographic image obtaining method
JP2010046216A
Combined use of intraluminal data and extraluminal imaging
JP2015519957A
Coregistering images of a region of interest during several conditions using a landmark subsurface feature
US20130070069A1
Method of HUB communication
US20190201136A1
Endoscope device
WO2020153186A1
