Image reconstruction and endoscope tracking

Abstract

Systems, devices, and methods for endoscopic mapping of a target and tracking of an endoscope position within a subject during a procedure are disclosed. An example system includes an imaging system configured to capture endoscopic images of a target, the endoscopic images including a footprint of an aiming beam directed at the target; and a video processor configured to: identify one or more landmarks from the captured endoscopic images and determine respective positions of the one or more landmarks relative to the footprint of the aiming beam; and generate a target map by integrating a plurality of endoscopic images based on the landmarks identified from one or more of the plurality of endoscopic images. The target map can be used to track the endoscope position during an endoscopic procedure.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 055,936, filed July 24, 2020, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This article generally relates to endoscopy, and more specifically to systems and methods for endoscopic mapping of targets and tracking of the position of the endoscope during surgery. Background Technology

[0004] Endoscopes are typically used to access a subject's internal location, providing doctors with visual access. An endoscope is usually inserted into the patient's body, directing light towards the target being examined (e.g., a target anatomical structure or object) and collecting the light reflected from that object. The reflected light carries information about the object being examined and can be used to create an image of the object. Some endoscopes include a working channel through which the operator can perform suction, or pass instruments such as brushes, biopsy needles, or forceps, or perform minimally invasive surgery to remove unwanted tissue or foreign bodies from the patient's body.

[0005] Some endoscopes include, or can be used with, laser or plasma systems to deliver surgical laser energy to target anatomical structures or objects, such as soft or hard tissue. Examples of laser treatments include ablation, coagulation, vaporization, and fragmentation. In lithotripsy applications, lasers are used to break down stone structures in areas where stones form, such as the kidneys, gallbladder, and ureters, or to ablate large stones into smaller fragments.

[0006] Video systems have been used to assist physicians or technicians in visualizing the surgical site and guiding the endoscope during endoscopic procedures. Image-guided endoscopy typically requires locating the endoscope in a coordinate system of the target area and tracking its movement. Accurate mapping of the target area and effective endoscope positioning and tracking can improve the precision of endoscopic manipulation during endoscopic procedures, enhance the interventional capabilities of physicians or technicians, and improve the effectiveness of treatments such as laser therapy. Summary of the Invention

[0007] Systems, devices, and methods for endoscopic mapping of a target and tracking of an endoscope position within a subject during a procedure are described herein. An example system includes an imaging system configured to capture endoscopic images of a target, the endoscopic images including a footprint of an aiming beam directed at the target; and a video processor configured to: identify one or more landmarks from the captured endoscopic images and determine their respective positions with respect to the footprint of the aiming beam; and generate a target map by integrating a plurality of the endoscopic images based on the landmarks identified from one or more of the plurality of endoscopic images. The target map can be used to track the endoscope position during an endoscopic procedure.

[0008] Example 1 is a system for endoscopic mapping of a target. The system includes an imaging system configured to capture endoscopic images of a target, the endoscopic images including a footprint of an aiming beam directed at the target; and a video processor configured to: identify one or more landmarks from the captured endoscopic images and determine their respective positions with respect to the footprint of the aiming beam; and generate a target map by integrating a plurality of the endoscopic images based on the landmarks identified from one or more of the plurality of endoscopic images.

[0009] In Example 2, the subject matter of Example 1 optionally includes the video processor can be configured to identify a type of tissue at the location of the aiming beam and label the footprint of the aiming beam using a visual identifier indicative of the identified type of tissue.

[0010] In Example 3, the subject matter of any one or more of Examples 1-2 optionally includes a spectrometer communicatively coupled to the video processor, the spectrometer configured to measure one or more spectral properties of an illumination light signal reflected from the target; wherein the video processor is configured to identify a type of tissue at the location of the aiming beam based on the one or more spectral properties and label the footprint of the aiming beam using a visual identifier indicative of the identified type of tissue.

[0011] In Example 4, the subject matter of any one or more of Examples 2-3 optionally includes the video processor can be configured to identify the type of tissue as normal tissue or abnormal tissue.

[0012] In Example 5, the subject matter of any one or more of Examples 2-4 optionally includes the video processor can be configured to label the footprint of the aiming beam with different colors to indicate different types of tissue.

[0013] In Example 6, the subject matter of any one or more of Examples 1-5 optionally includes the video processor can be configured to identify the one or more landmarks from the endoscopic images based on a change in brightness of pixels of the endoscopic images.

[0014] In Example 7, the subject matter of Example 6 optionally includes one or more landmarks represented as line segments or intersecting line segments in the endoscopic images.

[0015] In Example 8, the subject matter of any one or more of Examples 1-7 optionally includes a video processor that can be configured to: select a subset of landmarks from the identified landmarks in the one or more of the plurality of endoscopic images based on whether laser energy was activated at the target sites where the identified landmarks are located; and generate the target map by integrating the plurality of endoscopic images based on the selected subset of landmarks.

[0016] In Example 9, the subject matter of any one or more of Examples 1-8 optionally includes a plurality of endoscopic images that can include images of various sites of a target, including a first endoscopic image of a first target site captured from a first endoscopic position and a second endoscopic image of a second target site captured from a second endoscopic position, wherein the video processor is configured to: identify matching landmarks that include corresponding two or more landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; align the first endoscopic image and the second endoscopic image about the matching landmarks in a coordinate system of the first image; and generate the target map using at least the aligned first image and the second image.

[0017] In Example 10, the subject matter of Example 9 optionally includes a video processor that can be configured to: transform the second image including one or more of scaling, translating, or rotating the second image; and align the transformed second image and the first image about the matching landmarks.

[0018] In Example 11, the subject matter of Example 10 optionally includes the transformation of the second image that can include a matrix multiplication by a transformation matrix.

[0019] In Example 12, the subject matter of any one or more of Examples 10-11 optionally includes a video processor that can be configured to scale the second image using a scaling factor that is based on a ratio of a distance between two of the matching landmarks in the first image to a distance between the corresponding two landmarks in the second image.

[0020] In Example 13, the subject matter of any one or more of Examples 10-12 optionally includes a video processor that can be configured to scale the second image by a scaling factor that is based on a ratio of a geometric feature of the aimed beam footprint in the first image to a geometric feature of the aimed beam footprint in the second image.

[0021] In Example 14, the subject matter of any one or more of Examples 10-13 optionally include the video processor can be configured to transform the second image to account for a change in endoscope orientation between the first image and the second image, the endoscope orientation indicating a degree of tilt of an endoscope tip relative to a target site.

[0022] In Example 15, the subject matter of Example 14 optionally includes the video processor can be configured to detect the change in endoscope orientation using a first slope between two of the matching landmarks in the first image and a second slope between the respective two landmarks in the second image.

[0023] In Example 16, the subject matter of any one or more of Examples 14-15 optionally include the video processor can be configured to detect the change in endoscope orientation using a first geometric feature of the aiming beam footprint in the first image and a second geometric feature of the aiming beam footprint in the second image.

[0024] In Example 17, the subject matter of Example 16 optionally includes at least one of the first aiming beam footprint or the second aiming beam footprint can have a shape of an ellipse having a major axis and a minor axis, and at least one of the first geometric feature or the second geometric feature can include a ratio of a length of the major axis to a length of the minor axis.

[0025] In Example 18, the subject matter of any one or more of Examples 1-17 optionally include the endoscope tracking system configured to: identify matching landmarks from real-time images of a target surgical site captured by the imaging system during an endoscopic procedure from an unknown endoscope position, the matching landmarks including corresponding two or more landmarks in the target map that match two or more landmarks in the real-time images; register the real-time images to the target map using the matching landmarks; and track the endoscope tip position based on the registration of the real-time images.

[0026] In Example 19, the subject matter of Example 18 optionally include the endoscope tracking system can be configured to identify the matching landmarks based on one or more ratios of distances between landmarks in the real-time images and one or more ratios of distances between landmarks in the target map.

[0027] In Example 20, the subject matter of any one or more of Examples 18-19 optionally include the endoscope tracking system can be configured to generate an indication of a change in tissue type at the target site.

[0028] Example 21 is a method of endoscopic mapping of a target. The method includes: directing a targeting beam at the target; capturing, via an imaging system, endoscopic images of the target, the endoscopic images including a targeting beam footprint; identifying, via a video processor, one or more landmarks from the captured endoscopic images and determining respective positions of the one or more landmarks relative to the targeting beam footprint; and generating, via the video processor, a target map by integrating a plurality of the endoscopic images based on the landmarks identified from one or more of the plurality of endoscopic images.

[0029] In Example 22, the subject matter of Example 21 optionally includes identifying a type of tissue at a location of the targeting beam using an illumination light signal reflected from the target; and marking the targeting beam footprint using a visual identifier indicative of the identified type of tissue.

[0030] In Example 23, the subject matter of any one or more of Examples 21-22 optionally includes identifying the one or more landmarks from the endoscopic images is based on a change in brightness of pixels of the endoscopic images.

[0031] In Example 24, the subject matter of any one or more of Examples 21-23 optionally includes wherein the plurality of endoscopic images includes images of various portions of the target, including a first endoscopic image of a first target portion captured at a first endoscopic position and a second endoscopic image of a second target portion captured at a second endoscopic position, the method including: identifying matching landmarks, the matching landmarks including corresponding two or more landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; aligning the first endoscopic image and the second endoscopic image about the matching landmarks in a coordinate system of the first image; and generating the target map using at least the aligned first image and the second image.

[0032] In Example 25, the subject matter of Example 24 optionally includes aligning the first endoscopic image and the second endoscopic image includes: transforming the second image including one or more of scaling, translating, or rotating the second image; and aligning the transformed second image and the first image about the matching landmarks.

[0033] In Example 26, the subject matter of Example 25 optionally includes transforming the second image includes: scaling the second image by a scaling factor, the scaling factor based on a ratio of a distance between two of the matching landmarks in the first image to a distance between the two corresponding landmarks in the second image.

[0034] In Example 27, the subject matter of any one or more of Examples 25-26 can optionally include that transforming the second image includes scaling the second image by a scaling factor that is based on a ratio of a geometric feature of the aiming beam footprint in the first image to a geometric feature of the aiming beam footprint in the second image.

[0035] In Example 28, the subject matter of any one or more of Examples 25-27 can optionally include that transforming the second image includes accounting for a change in endoscope orientation between the first image and the second image, the endoscope orientation indicating a tilt of an endoscope tip relative to the target site.

[0036] In Example 29, the subject matter of Example 28 can optionally include that the change in endoscope orientation is detected using a first slope between two of the matching landmarks in the first image and a second slope between the respective two landmarks in the second image.

[0037] In Example 30, the subject matter of any one or more of Examples 28-29 can optionally include that the change in endoscope orientation is detected using a first geometric feature of the aiming beam footprint in the first image and a second geometric feature of the aiming beam footprint in the second image.

[0038] In Example 31, the subject matter of any one or more of Examples 21-30 can optionally include, during an endoscopic procedure, capturing, using an imaging system, a real-time image of a surgical site of a target from an unknown endoscope position; identifying matching landmarks, the matching landmarks including corresponding two or more landmarks in the target map that match two or more landmarks in the real-time image; registering the real-time image to the target map using the matching landmarks; and tracking an endoscope tip position based on the registration of the real-time image.

[0039] In Example 32, the subject matter of Example 31 can optionally include that the matching landmarks are identified based on one or more ratios of distances between the landmarks in the real-time image and one or more ratios of distances between the landmarks in the target map.

[0040] Example 33 is at least one non-transitory machine -readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: directing an aiming beam at a target; capturing an endoscopic image of the target, the endoscopic image including an aiming beam footprint; identifying one or more landmarks from the captured endoscopic image and determining respective positions of the one or more landmarks relative to the aiming beam footprint; and generating a target map by integrating a plurality of endoscopic images based on the landmarks identified from one or more of the plurality of endoscopic images.

[0041] In Example 34, the subject matter of Example 33 optionally includes wherein the instructions cause the machine to perform operations further comprising: identifying a tissue type at a location of the aiming beam footprint; and labeling the aiming beam footprint using a visual identifier indicative of the identified tissue type.

[0042] In Example 35, the subject matter of any one or more of Examples 33-34 optionally includes wherein the instructions cause the machine to perform operations further comprising: identifying the one or more landmarks from the endoscopic image based on a change in brightness of pixels of the endoscopic image.

[0043] In Example 36, the subject matter of any one or more of Examples 33-35 optionally includes wherein the plurality of endoscopic images comprise images of various portions of the target, including a first endoscopic image of a first target portion captured at a first endoscopic position and a second endoscopic image of a second target portion captured at a second endoscopic position, and wherein the instructions cause the machine to perform operations further comprising: identifying matching landmarks, the matching landmarks comprising corresponding two or more landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; aligning the first endoscopic image and the second endoscopic image about the matching landmarks in a coordinate system of the first image; and generating the target map using at least the aligned first image and the second image.

[0044] In Example 37, the subject matter of Example 36 optionally includes wherein the operation of aligning the first endoscopic image and the second endoscopic image comprises: transforming the second image, including one or more of scaling, translating, or rotating the second image; and aligning the transformed second image and the first image about the matching landmarks.

[0045] In Example 38, the subject matter of Example 37 optionally includes wherein the operation of transforming the second image comprises: scaling the second image by a scaling factor, the scaling factor based on a ratio of a distance between two of the matching landmarks in the first image to a distance between the two corresponding landmarks in the second image.

[0046] In Example 39, the subject matter of any one or more of Examples 37-38 optionally includes wherein the operation of transforming the second image comprises: scaling the second image by a scaling factor, the scaling factor based on a ratio of a geometric feature of the aiming beam footprint in the first image to a geometric feature of the aiming beam footprint in the second image.

[0047] In Example 40, the subject matter of any one or more of Examples 37-39 optionally includes wherein the operation of transforming the second image comprises: accounting for a change in endoscope orientation between the first image and the second image, the endoscope orientation indicative of a tilt of an endoscope tip relative to the target portion.

[0048] In Example 41, the subject matter of Example 40 optionally includes, wherein the instructions cause the machine to perform operations further comprising detecting a change in the endoscope orientation using a first slope between two of the matching landmarks in the first image and a second slope between the corresponding two landmarks in the second image.

[0049] In Example 42, the subject matter of any one or more of Examples 40-41 optionally includes, wherein the instructions cause the machine to perform operations further comprising detecting a change in the endoscope orientation using a first geometric feature of the aiming beam footprint in the first image and a second geometric feature of the aiming beam footprint in the second image.

[0050] In Example 43, the subject matter of any one or more of Examples 33-42 optionally includes, wherein the instructions cause the machine to perform operations further comprising, during the endoscopic procedure, capturing, using the imaging system, a real-time image of a surgical site of the target from an unknown endoscope position; identifying matching landmarks comprising corresponding two or more landmarks in the target map that match two or more landmarks in the real-time image; registering the real-time image to the target map using the matching landmarks; and tracking the endoscope tip position based on the registration of the real-time image.

[0051] In Example 44, the subject matter of any one or more of Examples 33-43 optionally includes, wherein the operation of identifying the matching landmarks is based on one or more ratios of distances between the landmarks in the real-time image and one or more ratios of distances between the landmarks in the target map.

[0052] This Summary is an overview of some of the teachings of the present application and is not intended to be exclusive or limiting. Further details of the present subject matter can be found in the DETAILED DESCRIPTION and the appended claims. Other aspects of the disclosure will be apparent to those of ordinary skill in the art upon reviewing the detailed description and the appended claims. Each of the drawings figures is intended to be illustrative and not restrictive in nature. The scope of the disclosure is defined by the appended claims and their legal equivalents. BRIEF DESCRIPTION OF DRAWINGS

[0053] The various embodiments are illustrated by way of example in the figures. Such embodiments are illustrative of the subject innovation but are not meant to be exhaustive or limiting in nature.

[0054] Figure 1 FIG. 1 is a diagram illustrating an example of a medical system for endoscopic surgery.

[0055] Figure 2 FIG. 2 is a diagram illustrating an example of a system shown in FIG. 1. Figure 1 FIG. 3 is a diagram illustrating an example of a portion of the system shown in FIG. 1. FIG. 4 is a diagram illustrating an example of a portion of the system shown in FIG. 1.

[0056] Figure 3 is a block diagram illustrating an example of an endoscope controller for controlling portions of a system as Figure 1

[0057] Figures 4A to 4F Examples of sequences of endoscopic images or video frames captured at different target sites and landmarks and the aiming beam footprints detected therefrom are shown.

[0058] Figure 5 Examples of target maps reconstructed from a plurality of endoscopic images or video frames captured at different target sites are shown.

[0059] Figures 6A to 6F is a diagram illustrating the effect of endoscope orientation on endoscopic image features and correction for changes in endoscope orientation from one endoscopic image to another.

[0060] Figure 7 Examples of identifying matching landmarks between a real-time image and a reconstructed target map and registering the real-time image to the target map with respect to the matching landmarks are shown.

[0061] Figure 8 is a flowchart illustrating a method of endoscopic mapping of a target within a subject during a procedure.

[0062] Figure 9 is a flowchart illustrating an example of a method for endoscope tracking using a reconstructed target map.

[0063] Figure 10 is a block diagram illustrating an example machine that can perform any one or more of the techniques (e.g., methods) discussed herein. DETAILED DESCRIPTION

[0064] Minimally invasive endoscopic surgery is a surgical procedure in which a rigid or flexible endoscope is introduced into a target region of a subject’s body through a natural orifice or a small incision in the skin. Additional surgical tools, such as a laser fiber, can be introduced into the subject’s body through similar ports, with the endoscope used to provide visual feedback to the surgeon of the surgical site and the surgical tools.

[0065] ​Endoscopic surgery can include a pre-operative phase and an intra-operative phase. The pre-operative phase involves acquiring images or video frames of a target anatomical structure or object using an imaging system (e.g., a video camera) and reconstructing a map using the images or video frames. The map can be used for diagnostic assessment or for endoscopic surgery planning. During the intra-operative phase, an endoscope can be introduced into a target surgical site. An operator can move and turn the endoscope distal tip and acquire real-time images of the surgical site via an imaging system (e.g., located at the endoscope distal tip). The position and orientation of a surgical tool (e.g., a laser fiber) at the endoscope distal tip can be monitored and tracked throughout the surgery.

[0066] A conventional intra-operative tracking involves a freehand technique whereby a surgeon views the surgical area on a monitor that displays real-time images or video of the surgical area without an automatic tracking or navigation system. This approach fails to establish a relationship between images that facilitates tracking of the position and orientation of an endoscopic surgical tool relative to a target. Another approach involves a navigation-based tracking system (e.g., optical or electromagnetic tracking system) that tracks the position and orientation of the endoscopic surgical tool. An image registration procedure can be performed to align the real-time images with the target map. A fiducial marker visible on the real-time images is used as a reference to guide the surgeon with real-time feedback of the position and orientation of the endoscopic surgical tool. The fiducial marker can be an external object attached to the patient or can be an internal anatomical fiducial. External fiducials can lack consistency in position and add complexity to the system. The use of internal fiducials often imposes restrictions on the physical movement of the endoscope, such as requiring the scope to touch the anatomical fiducial marker during surgery, which can lengthen the surgery time. Conventional navigation-based endoscopic tracking systems can also suffer from degraded tracking performance in the presence of image distortion or deformation, such as due to changes in camera position, viewing direction, and orientation relative to the target surface (e.g., skew or tilt). For at least the reasons described above, the present inventors have recognized an unmet need for an improved endoscopic mapping and tracking system that is more robust to image distortion or deformation when used during endoscopic surgery.

[0067] Described herein are systems, devices, and methods for endoscopic mapping of a target and tracking of an endoscope position within a subject during surgery. An exemplary system includes an imaging system configured to capture endoscopic images of a target, the images including a footprint of an aiming beam directed at the target; and a video processor configured to identify one or more landmarks from the captured endoscopic images and determine their respective positions relative to the aiming beam footprint, and generate a target map by integrating a plurality of endoscopic images based on the landmarks identified from one or more of the plurality of endoscopic images. The target map can be used to track the endoscope position during surgery.

[0068] Systems, devices, and methods according to various embodiments discussed herein can provide improved endoscopic mapping of a target and tracking of an endoscope position during endoscopic surgery. According to various embodiments of the present disclosure, various image features can be generated from endoscopic images, including, for example, landmarks and their positions relative to a targeting beam footprint, spatial relationships between landmarks, shape and geometric properties of a targeting beam footprint, and so on. Image registration, reconstruction of a target map, and endoscopic tracking during endoscopic surgery based on these image features described herein are more resilient to image rotation, magnification, minification, changes in camera position, changes in viewing direction or endoscope orientation. With improved navigation and endoscopic tracking, an operator’s intervention capability and precision of endoscopic operation can be enhanced, surgery time can be reduced, and overall surgery effectiveness, patient safety, and system reliability can be improved.

[0069] The subject matter discussed herein can be applied to various endoscopic applications, including but not limited to arthroscopy, bronchoscopy, colonoscopy, laparoscopy, neuroendoscopy, and endoscopic cardiac surgery. Examples of endoscopic cardiac surgery include but are not limited to endoscopic coronary artery bypass, endoscopic mitral and aortic valve repair and replacement. In this document, “endoscopic” is broadly defined herein as a property of images acquired by any type of endoscope having the capability of imaging from inside a body. For the purposes of the present invention, examples of endoscopes include but are not limited to any type of mirror (e.g., endoscope, arthroscope, bronchoscope, cholangioscope, colonoscope, cystoscope, duodenoscope, gastroscope, hysteroscope, laparoscope, laryngoscope, neuroscope, otoscope, push enteroscope, rhinolaryngoscope, sigmoidoscope, sinuscope, thoracoscope, etc.) of any flexibility, as well as any device similar to a mirror equipped with an image system (e.g., a nested cannula capable of imaging). Imaging is local, and surface images can be optically obtained through fiber optics, lenses, or miniaturized (e.g., CCD-based) imaging systems. For the purposes of the present invention, examples of fluoroscopes include but are not limited to X-ray imaging systems.

[0070] Figure 1 FIG. 1 is a diagram showing an example of a medical system 100 for endoscopic surgery. The system 100 includes an endoscope 102, an endoscope controller 103, a light source 104, a laser device 106, and a display 108. Figure 2 A schematic diagram of portions of the system 100 is shown in FIG. 2. The endoscope 102 can include an insertion portion 110 at a distal end portion of the endoscope 102 and an operating unit 107 at a proximal end portion. The insertion portion 110 can be configured to be inserted into a target site of a subject, capture images of a target 101, and optionally perform a procedure therein. The insertion portion 110 can be formed using illumination fibers (light guides), electrical cables, optical fibers, etc. In FIG. 2, the endoscope 102 is shown as a flexible endoscope, but the endoscope 102 can be a rigid endoscope. Figure 1In the illustrated example, the insertion portion 110 includes a distal end portion 110a, a bendable bending portion 110b, and a flexible tube portion 110c provided on the proximal end portion side of the bending portion 110b, the distal end portion 110a containing an imaging unit, the bendable bending portion 110b including a plurality of bendable members, and the flexible tube portion 110c being flexible.

[0071] Referring to Figure 2 , the distal end portion 110a can be provided with a light guide 120 configured to be coupled to the light source 104 and to project illumination light 230 onto the target 101 via an illumination lens 122. The distal end portion 110a can include a viewing unit such as an imaging system 115 configured to image the target 101. The imaging system 115 can include an image sensor 116 and an associated lens system 118. An example of the image sensor 116 can include a CCD or CMOS camera sensitive at ultraviolet (UV), visible (VIS), or infrared (IR) wavelengths. The endoscope 102 can include an insertion port 107b coupled to a treatment tool channel 102a located inside the endoscope 102 and extending along the insertion portion 110. An optical path 112 disposed within the channel 102a through the insertion port 107b has a proximal end operatively connected to the laser device 106 and extending distally from a distal opening 102b of the channel 102a. Laser energy (e.g., a treatment beam 240a or an aiming beam 240b) can be transmitted through the optical path 112 and emitted from a distal end 112a of the optical path 112 and directed toward the target 101. The endoscope 102 can selectively include an air / water supply nozzle (not shown) at the distal end portion 110a.

[0072] The operation unit 107 can be configured to be held by an operator. The operation unit 107 can be located at a proximal end portion of the endoscope 102 and configured to communicate with the endoscope controller 103 and the light source 104 via a flexible general-purpose cable 114 extending from the operation unit 107. As Figure 1 illustrated, the operation unit 107 includes a bending knob 107a for bending the bending portion 110b in the up-down direction and the left-right direction, a treatment tool insertion port 107b through which a treatment tool such as a medical forceps or the optical path 112 is inserted into a body cavity of a subject, and a plurality of switches 107c for operating peripheral devices such as the endoscope controller 103, the light source 104, an air supply device, a water supply device, or a gas supply device. A treatment tool such as the optical path 112 can be inserted from the treatment tool insertion port 107b and pass through the channel l02a so that its distal end is exposed from an opening 102b (see Figure 2 ) of the channel 102a at the distal end of the insertion portion 110.

[0073] The endoscope controller 103 can control the operation of one or more elements of the system 100, such as the light source 104, the laser device 106, or the display 108 that displays an image of the target 101 based on an image or video signal sensed by the imaging system 115. The distal tip of the endoscope 102 can be positioned and oriented such that the aiming beam 240b is directed to a target location within a field of view (FOV) of the imaging system 115; and the endoscopic image includes a footprint of the aiming beam 240b. Although the aiming beam 240b is shown as a laser beam emitted from a laser energy source, other light sources can be used to produce the aiming beam that propagates along an optical fiber. The endoscope controller 103 can impose image processing on the endoscopic image, and reconstruct a map of the target by integrating multiple endoscopic images. In some examples, the endoscope controller 103 can use the reconstructed target map to position and track the endoscope tip during endoscopic surgery. Examples of the endoscope controller 103, including endoscopic mapping of a target and tracking of an endoscope position, will be discussed below, for example, with reference to Figure 3

[0074] The universal cable 114 includes an optical fiber, an electrical cable, or the like. The universal cable 114 can be branched at its proximal end. One end of the branched end is a connector 114a, and the other proximal end of the branched end is a connector 114b. The connector 114a is attachable to / detachable from a connector of the endoscope controller 103. The connector 114b is attachable to / detachable from the light source 104. The universal cable 114 propagates illumination light from the light source 104 to the distal portion 110a via the connector 114b and the light guide 120. In addition, the universal cable 114 can send an image or a video signal captured by the imaging system 115 to the endoscope controller 103 via a signal line 124 (see Figure 2 ) in the cable and via the connector 114a. The endoscope controller 103 performs image processing on the image or the video signal output from the connector 114a, and controls at least part of the components that constitute the system 100.

[0075] ​The light source 104 can generate illumination light when the endoscope 102 is used for surgery. The light source 104 can include, for example, a xenon lamp, a light emitting diode (LED), a laser diode (LD), or any combination thereof. In an example, the light source 104 can include two or more light sources that emit light having different illumination characteristics (referred to as illumination modes). Under the control of the endoscope controller 103, the light source 104 emits light, which is supplied to the endoscope 102 connected via the connector 114b and the light guide of the general-purpose cable 114, as illumination light for inside the subject as an object. The illumination mode can be a white light illumination mode or a specific light illumination mode such as a narrow-band imaging mode, an autofluorescence imaging mode, or an infrared imaging mode. The specific light illumination can concentrate and intensify light of a specific wavelength, for example, to better visualize superficial microvessels and mucosal surface structures to enhance subtle contrasts of irregularities of the mucosa.

[0076] The display 108 includes, for example, a liquid crystal display, an organic electroluminescent display, or the like. The display 108 can display information including an endoscopic image of a target, which is subjected to image processing by the endoscope controller 103 via the video cable 108a. In some examples, one or more endoscopic images can each include a footprint of the aiming beam 240b. The operator can observe and track the behavior of the endoscope inside the subject by operating the endoscope 102 while observing the image displayed on the display 108.

[0077] The laser device 106 is used for the optical path 112 of, for example, a laser fiber. Referring to Figure 2 , the laser device 106 can include one or more energy sources (e.g., the first energy source 202 and the second energy source 204) for generating laser energy coupled to the proximal end of the optical path 112. In an example, the user can select the energy source, for example, via a button 106a (see Figure 1 ) or a footswitch (not shown) on the laser device 106, by software or a user interface on the display 108, or other manual or automatic input known in the art.

[0078] The first energy source 202 can be optically coupled to the optical path 112 and configured to deliver a therapeutic beam 240a to the target 101 through the optical path 112. By way of example and not limitation, the first energy source 202 can include a thulium laser for generating laser light to be delivered to the target tissue through the optical path 112 to operate in different treatment modes such as a cutting (ablation) mode and a coagulation (hemostasis) mode. Other energy sources known in the art for such tissue treatment or any other treatment mode can also be used for the first energy source 202, such as Ho:YAG, Nd:YAG, and CO2, and others known in the art.

[0079] The second energy source 204 can be optically coupled to the optical path 112 and configured to direct the aiming beam 240b toward the target 101 through the optical path 112. Although the aiming beam 240b is shown as a laser beam emitted from a laser source, other light sources can be used to produce the aiming beam 240b that travels along the optical fiber. The aiming beam 240b can be emitted while the target is illuminated by the illumination light 230. In some examples, the second energy source 204 can emit at least two different aiming beams, where a first aiming beam has at least one characteristic that is different from a second aiming beam. Such different characteristics can include wavelength, power level, and / or emission pattern. For example, a first aiming beam can have a wavelength in the range of 500 nm to 550 nm, while a second aiming beam can have a wavelength in the range of 635 nm to 690 nm. The characteristics of the different aiming beams can be selected based on the visibility of the aiming beam in the image processed by the endoscope controller 103 and displayed on the display 108 under certain illumination modes provided by the light source 104.

[0080] The laser device 106 can include a controller 206 that includes hardware, such as a microprocessor, that controls the operation of the first energy source 202 and the second energy source 204. In the example shown, the controller 206 is in communication with the first energy source 202 and the second energy source 204. The controller 206 can be in communication with the light source 104 and the endoscope 102. The controller 206 can be in communication with the display 108 and the user interface 110. The controller 206 can be in communication with the spectrometer 208. Figure 2 In the example shown, in response to the illumination light 230, light reflected from the target 101 can enter the optical path 112 from the distal end 112a. The optical path 112, which is configured to transmit the laser beam, can also serve as a passageway to transmit the reflected light back to the laser device 106. The beamsplitter 205 can collect the reflected light, separating it from the laser beam that is being delivered to the target 101 via the same optical path 112. The laser device 106 can include a spectrometer 208 that is operatively coupled to the beamsplitter 205 and configured to detect the reflected light that comes out of the beamsplitter. Alternatively, the reflected light can be directed through an optical path (e.g., an optical fiber) that is separate from the optical path 112. The spectrometer 208 can be operatively coupled to the dedicated optical path and detect the light that is reflected therefrom.

[0081] The spectrometer 208 can measure one or more spectral characteristics from the sensed reflected signal. Examples of the spectrometer 208 can include a Fourier transform infrared (FTIR) spectrometer, a Raman spectrometer, a UV-VIS spectrometer, a UV-VIS-IR spectrometer, or a fluorescence spectrometer, among others. The spectral characteristics can include characteristics such as reflectance, reflectance spectrum, absorption index, and the like. The spectral characteristics can be indicative of a class of structures (e.g., anatomical tissue or calculus) or a specific structure type that is indicative of a chemical composition of the target.

[0082] Figure 3is a block diagram illustrating an example of an endoscope controller 103 for use in the system 100. The endoscope controller 103 includes hardware, e.g., a microprocessor, for performing operations in accordance with various examples described herein. The endoscope controller 103 can include a device controller 310, a video processor 320, an endoscope tracker 330, and a memory 340. The device controller 310 can control the operation of one or more components of the system 100, e.g., the endoscope 102, the display 108, the light source 104, or the laser device 106.

[0083] The video processor 320 can receive an image or video signal from the imaging system 115 through the signal line 124, and process the image or video signal to produce an image or video frame that can be displayed on the display 108. In some examples, multiple endoscopic images, e.g., video frames, can be generated and displayed on the display 108. When the endoscope tip and the imaging system 115 remain stationary, multiple endoscopic images can be taken at the same site of the target 101 while the distal tip 112a of the light path 112, e.g., a laser fiber, can be moved and direct the laser beam to different locations of the target. In some examples, multiple endoscopic images of the same target site can be taken at different times. The images taken at later times can be registered with the images of the same target site taken previously, e.g., through image transformation and / or image alignment processing. The registered images can be used to determine the change of tissue state at the target site. Additionally or alternatively, multiple endoscopic images at different sites of the target 101 can be taken, e.g., when the endoscope distal tip is translated over the target 101. During the endoscope translation, the endoscope distal tip can be moved and positioned at different endoscope locations {L1, L2,..., L N}(i.e., respective positions of the endoscope distal tip), which can be done manually by the operator or automatically by the endoscope actuator. The imaging system 115, under the control of the endoscope controller 103, can take a series of images (or video frames) {G1, G2,..., G N} at respective target sites {S1, S2,..., S N} that collectively cover a substantial surface area of the target 101. For example, when the lens system 118 is positioned and oriented at the endoscope location L i , an endoscopic image G i of the target site S i that falls within the FOV of the imaging system 115 can be taken. When the endoscope distal tip is moved to a different endoscope location L j , another endoscopic image G j at a different target site S j that falls within the FOV of the imaging system 115 can be taken. For example, in accordance with the following, e.g., with reference toFigures 4A to 4F and Figure 5 The video processor 320 can integrate the resulting images {G1, G2,..., G N} to create a map of the target 101.

[0084] As described above, the endoscope tip can be positioned and oriented such that the aiming beam 240b falls within the FOV of the imaging system 115, and the aiming beam footprint can be captured in the endoscope image (e.g., G i ). In an example, the video processor 320 can color the aiming beam footprint with a different color than the background of the endoscope image. The video processor 320 can identify the location where the aiming beam 240b is currently illuminating by matching the color of the aiming beam 240b to the color of the pixels of the endoscope image.

[0085] The video processor 320 can include a landmark detector 321 configured to detect one or more landmarks from the endoscope image. The landmarks can be created manually by an operator or identified automatically using image processing algorithms. In an example, the landmark detector 321 can detect landmarks based on changes in brightness of the pixels of the endoscope image. In an example, the landmark detector 321 can detect landmarks using edge detection constrained by a minimum contrast threshold and a number of pixels between similar positive and negative contrast slopes. The detected landmarks can indicate blood vessels. The edge detection can involve detecting a transition from light to dark pixel brightness that indicates the beginning of a blood vessel segment, and detecting a subsequent transition from dark to light pixel brightness that indicates the end of a blood vessel segment. Other criteria can be applied to confirm the detection of a blood vessel. For example, if the subsequent transition occurs with at least a user-defined number of dark pixels, and it is flanked by a threshold number of light pixels, then an edge defined by the transition between light and dark pixels can be used as a landmark, provided that those points along one or both edges persist for at least a length greater than another threshold. In another example, if a linear regression of the pixels of an edge produces a line with an R-squared or other goodness of fit greater than a target threshold (e.g., 0.8 in an example), then the detected edge is determined to be a blood vessel.

[0086] In the endoscope map, the landmarks can have different morphologies. In an example, a landmark can be represented as a line segment in the endoscope image. In another example, a landmark can be represented as two or more line segments intersecting at a point in the endoscope image, which is referred to as an intersecting landmark, or a point landmark. In some examples, for two non-intersecting and non-parallel line segments that are close to each other (e.g., within a particular distance range), the landmark detector 321 can algorithmically extend one line segment until it intersects the other line segment to create a point landmark.

[0087] The landmark detector 321 can locate landmarks in the coordinate system of the endoscope image relative to the aiming beam footprint. For example, the position of a point-like landmark in the endoscope image can be represented by a vector between the point-like landmark and the aiming beam footprint or by distances along the X and Y axes in the coordinate system. In some examples, the landmark detector 321 can determine spatial relationships between landmarks (e.g., distances and slopes between landmarks) in the coordinate system of the endoscope image. Information about the landmarks and their positions, the aiming beam footprint, and the spatial relationships between landmarks can be stored in the memory 340 and used for endoscope image registration, target map reconstruction, or endoscope tracking during an endoscopic procedure according to various examples discussed herein.

[0088] The landmark detector 321 can select a subset of the detected landmarks to store in the memory 340 or for use in applications such as image registration, target map reconstruction, or endoscope tracking. In examples, the subset of landmarks can be selected based on the positions of the landmarks (e.g., the spatial distribution of landmarks in the endoscope image). For example, landmarks that are spread out in the endoscope image can be more easily selected than clusters of landmarks that are closely spaced in the endoscope image. In another example, the subset of landmarks can be selected based on whether the laser energy was activated at the target site where the landmark is located. Because the laser energy can affect the accuracy and consistency of landmark detection, in examples, landmarks that were not activated by the laser energy can be more advantageously selected than another landmark that was activated by the laser energy.

[0089] In some examples, the endoscope controller 103 can control the light source 104 or the illumination lens 122 to produce specific illumination conditions of the target 101 to improve landmark detection and localization. For example, the light source 104 can provide blue or green illumination to increase the contrast of the target 101 on the endoscope image and more clearly define vessels that are less likely to move or change over time. This allows for more consistent landmark detection and localization under slightly different illumination conditions. In examples, the endoscope controller 103 can temporarily change the illumination, for example, turning on green or blue light sources to best identify landmarks, and revert to normal illumination mode after the landmarks are identified.

[0090] The video processor 320 can include a target identifier 322 configured to identify a target type at the aiming beam location of the target 101. In an example, the identification of the target type can be based on one or more spectral properties of the illumination light reflected from the target 101. The spectral properties can be measured using the spectrometer 208. The identified target type can include an anatomical tissue type or a stone type. Examples of the stone type can include a stone or a stone fragment in different stone forming sites such as urinary system, gallbladder, nasal cavity, gastrointestinal tract, stomach, or tonsil. Examples of the anatomical tissue type can include soft tissue (e.g., muscle, tendon, ligament, blood vessel, fascia, skin, fat, and fibrous tissue), hard tissue such as bone, connective tissue such as cartilage, etc. In an example, the target identifier 323 can identify the tissue type at the aiming beam location of the target 101 as normal and abnormal tissue, or mucosal or muscle tissue based on the characteristics of the reflected illumination signal.

[0091] The video processor 320 can label (e.g., display on the display 108) the aiming beam footprint in the endoscopic images using a visual identifier indicative of the identified tissue type. In an example, the visual identifier can include a color code such that the aiming beam footprint can be colored differently to indicate different tissue types. For example, if the target site is identified as normal tissue, the aiming beam footprint can be colored green, or if the target site is identified as abnormal tissue (e.g., cancer), the aiming beam footprint can be colored red. In an example, the video processor 320 can label the aiming beam footprint with a visual identifier indicative of a change in the tissue type at the target site over time (e.g., from normal to abnormal, or vice versa), for example, by using a color different from the color representative of normal tissue or abnormal tissue. In another example, the video processor 320 can label the aiming beam footprint with a visual identifier indicative of a treatment status at the target site. For example, if the target site has been treated (e.g., with a laser), the aiming beam footprint can be represented with a dot different from the color representative of normal tissue or abnormal tissue.

[0092] Figures 4A to 4F A sequence of endoscopic images (e.g., video frames) taken at the target 101 (e.g., the interior of a kidney, bladder, urethra, or ureter, and other anatomical structures of interest) is shown by way of example and not limitation as {G1, G2,..., G N The endoscopic images can be displayed on the display 108. As described above, the endoscopic images at the same target site can be taken at different times, or the endoscopic images at different target sites {S1, S2,... S N The target identifier 322 can identify the target type at the target sites {S1, S2,... SN The target type at the corresponding aiming beam position. The video processor 320 can mark the aiming beam footprint in the corresponding endoscopic image using a corresponding visual identifier (e.g., color) that identifies the target type, detect and locate landmarks in the endoscopic image, and integrate the endoscopic image into the target map of target 101 based on the landmarks identified from the endoscopic image.

[0093] like Figures 4A to 4F The endoscopic images 410-460 shown are generated as the distal end of the endoscope is translated over the target 101, during which time the distal end of the endoscope is manually or automatically moved and positioned at different endoscopic locations. Figure 4A An endoscopic image 410 is shown, which includes a graphical representation of the illuminated target region S1 falling within the field of view (FOV) of the imaging system 115, and a circular aiming beam footprint 412. The aiming beam footprint 412 is tinted green to indicate that the tissue at the aiming beam location is normal tissue. In this example, image 410 also shows an image 413 of the distal end 112a of the optical path 112 (e.g., a laser fiber) and an image 414 of the distal portion of the endoscope 102.

[0094] Image 410 also includes, for example, landmarks 415A-415C detected by target recognizer 322. In this example, landmarks 415B and 415C are each represented by two line segments that intersect to form a point landmark, and landmark 415A is represented by two line segments that intersect by an algorithm (e.g., by projecting one line segment onto another) to form a point landmark. The positions of landmarks 415A-415C can be determined by target recognizer 322 (see above). Figure 3 (As discussed). Images 410, including information about the aiming beam footprint 412 and landmarks 415A-415C, can be stored in memory 340.

[0095] As the distal endoscope is moved manually or automatically to a new endoscope position, images can be generated such as... Figure 4BAnother endoscopic image 420 is shown. The new endoscopic image 420 includes a graphical representation of the new illuminated target site S2 corresponding to the new endoscopic position and a new aiming beam footprint 422. Since the tissue at the current aiming beam location is identified as normal tissue, the aiming beam footprint 422 is colored green. If new landmarks are detected from the current endoscopic image, the new landmarks can be included in the image. In this example, no new landmarks are detected from the endoscopic image 420. If the previous aiming beam footprint 412 and the previously generated landmarks 415A-415C are within the FOV of the imaging system at the current endoscopic position, the previous aiming beam footprint 412 and the previously generated landmarks 415A-415C can be retained in the current image 420.

[0096] If the movement of the distal tip is made in small step sizes, the illuminated target sites S1 and S2 can overlap, such that both endoscopic images 410 and 420 can cover a common region of the target 101 (as shown in FIGS. 4B and 4C, respectively). Figure 4A and Figure 4B One or more matching landmarks can be identified from the endoscopic images 410 and 420. Such matching landmarks can be used to align the images 410 and 420 to reconstruct a map of the target, according to various examples to be discussed below.

[0097] The endoscopic translation process can continue, and additional endoscopic images can be generated. Figure 4C An image 430 is shown that includes a graphical representation of the new illuminated target site S3 and a new aiming beam footprint 432 colored red to indicate that abnormal tissue is identified at the current aiming beam location. New landmarks 435A-435B can be detected from the current endoscopic image. The previous aiming beam footprint and the previously generated landmarks (e.g., 415A-415C) are still within the FOV of the imaging system at the current endoscopic position, and can be retained in the image 430.

[0098] Figure 4DAn image 440 is shown that includes a graphical representation of the newly illuminated target site S4 and a new aiming beam footprint 442 colored green to indicate that normal tissue is identified at the current aiming beam location. New landmarks 445A-445C can be detected from the current endoscope image. The new aiming beam footprint (including its location and color representative of the tissue type) and the new landmarks (including their locations relative to the aiming beam footprint) as well as the previous aiming beam footprint and previously generated landmarks can be stored in the memory 340. The previous aiming beam footprint and previously generated landmarks (e.g., 435B) that fall within the FOV of the imaging system at the current endoscope location can be saved in the image 440.

[0099] The endoscope distal tip can be moved manually or automatically along a specific path or following a specific pattern such that the endoscope images produced during the translation process can collectively provide a panoramic coverage of the substantial surface area of the target 101. As a non-limiting example, Figures 4A to 4F An image 440 is shown that includes a graphical representation of the newly illuminated target site S4 and a new aiming beam footprint 442 colored green to indicate that normal tissue is identified at the current aiming beam location. New landmarks 445A-445C can be detected from the current endoscope image. The new aiming beam footprint (including its location and color representative of the tissue type) and the new landmarks (including their locations relative to the aiming beam footprint) as well as the previous aiming beam footprint and previously generated landmarks can be stored in the memory 340. The previous aiming beam footprint and previously generated landmarks (e.g., 435B) that fall within the FOV of the imaging system at the current endoscope location can be saved in the image 440. Figures 4A to 4D The endoscope distal tip is moved vertically up, during which endoscope images at various target sites can be taken. Figure 4E An image 450 is shown that includes a graphical representation of the newly illuminated target site S5 and a new aiming beam footprint 452 colored green to indicate that normal tissue is identified at the current aiming beam location. New landmarks 455A-455B can be detected from the current endoscope image. The previous aiming beam footprint and previously generated landmarks that fall within the FOV of the imaging system at the current endoscope location are retained in the image 450.

[0100] After the upward vertical motion, the endoscope distal tip takes a horizontal motion to the right, during which endoscope images at various target sites can be taken. Figure 4F An image 460 is shown that includes a graphical representation of the illuminated target site S6 (which includes a portion of the previously visited illuminated site captured in the image 410) and a new aiming beam footprint 462 colored green to indicate that normal tissue is identified at the current aiming beam location. New landmarks 465A-465B can be detected from the current endoscope image. The previous aiming beam footprint and previously generated landmarks that fall within the FOV of the imaging system at the current endoscope location are retained in the image 460. This includes the previously generated landmark 435A that once fell outside of the endoscope images 440 and 450.

[0101] Returning to Figure 3The video processor 320 may include a target image generator 323, which is configured to integrate multiple endoscopic images (or video frames) {G1, G2, ..., G...} of various target parts of the target 101 stored in the memory 340. N To reconstruct the target image, refer to the above text. Figures 4A to 4F The stored endoscopic images G discussed i This may include a graphical representation of the illuminated target area, the aiming beam footprint (including its location and color representing the tissue type), and one or more landmarks (including their positions relative to the aiming beam footprint and the spatial relationships between the landmarks). The target image generator 323 can perform image registration to align the stored endoscopic image {G1, G2, ..., G...} based on the landmarks with relative positions. N Image registration may include identifying matching landmarks, including from the first endoscopic image (e.g., at the first target site S). i Image G taken at location i Two or more landmarks identified from a second endoscopic image (e.g., at different second target sites S). j Image G taken at location j Two or more identified landmarks match, and the target image generator 323 aligns the second image with the first image relative to the identified matching landmarks. For example, it can be used in... Figure 4B Image 420 and Figure 4C Matching landmarks 415A-415C, present in both images 430, align image 430 with image 420. The aligned images can then be stitched together with respect to the matching landmarks to reconstruct the target image. In some examples, landmark detector 321 can adjust the landmark detection algorithm (e.g., reduce the edge detection threshold) to allow for the identification of more landmarks from the endoscopic images. Multiple landmarks can increase the probability of identifying matching landmarks between images and improve the accuracy of image alignment.

[0102] Figure 5 An example of a target map 500 for target 101 (e.g., a solid region of the bladder) is shown. In addition to the stitched images, the reconstructed map may additionally include one or more of the following: a set of landmarks identified from multiple endoscopic images, a targeting beam footprint, a target type identifier (e.g., a color code for the targeting beam footprint), landmark positions relative to the targeting beam footprint, or spatial relationships between landmarks. Target map 500 can be used to assist in medical diagnosis or treatment planning, such as locating and tracking the endoscope position during endoscopic procedures.

[0103] It may be used to reconstruct the target image from the endoscopic image (e.g., for reconstruction). Figure 5 Target Figure 500Figures 4A to 4F Geometric warping or distortion is introduced in the endoscopic images 410-460) due to the change in viewing direction (from the imaging system 115 at the distal tip of the endoscope towards the target) can cause image rotation. In some cases, geometric warping or distortion can be caused by a change in endoscope orientation. In this context, endoscope orientation refers to the tilt or skew of the endoscope tip relative to the surface of the target. Changes in endoscope orientation from one image to another can cause warping in length, shape, and other geometric properties. To correct for such warping or distortion, in some examples, the target map generator 323 can transform one image before aligning it with another image. Examples of image transformations can include one or more of scaling, translation, rotation, or shear transformation of the image in a coordinate system, as well as other rigid transformations, similarity-based transformations, or affine transformations. In examples, the images can be scaled by a scaling factor based on the inter-landmark distance measured from the two images, respectively, or by a scaling factor based on a geometric feature measured from the aiming beam footprints in the two images, respectively (e.g., described below with respect to Figure 7 In examples, changes in endoscope orientation can be corrected based on the slope between the landmarks measured from the two images, respectively, or based on a geometric feature measured from the aiming beam footprints in the two images, respectively (e.g., described below with respect to Figures 6A to 6F

[0104] The transformation can be implemented as a transformation matrix multiplied by the image data (e.g., a data array). The transformed image can be aligned with another image with respect to matching landmarks between the transformed image and the other image. In some examples, the alignment can be based on the slope of the landmarks with respect to each other. Such alignment can be insensitive to the distance between the landmarks (related to the difference in magnification, or the distance of the endoscope from the target). Image transformations according to various examples described herein can improve the robustness of target map reconstruction to differences in endoscopic image rotation, magnification, or shrinkage.

[0105] The landmarks in the transformed endoscopic images are saved in the memory 340 in their transformed state (as if on the two-dimensional projection surface of the target). Thus, the landmarks in the transformed endoscopic images are invariant to the non-uniformity, tilt, rotation, scaling, and other warping or distortion of the surface. The saved transformed endoscopic images can be integrated to form an integrated target map. The saved landmarks can be used as a basis for comparison of new images, or for transforming new images to the two-dimensional projection surface and registering the new images onto the saved target map (as described below with respect to Figure 7 ​​

[0106] Figures 6A to 6F are diagrams showing the effect of endoscope orientation on endoscope image characteristics and methods of correcting for different endoscope orientations between two endoscope images. The endoscope orientation correction methods discussed herein can be applied to image registration applications, such as registering a real-time intraoperative endoscope image to a target map (such as target map 500), which will be discussed below with reference to Figure 7 Endoscope orientation refers to the tilt or skew angle Θ of the lens system 118 relative to the surface of the target. For two endoscope images G i and G j , image properties such as landmark positions (e.g., distances to the footprint of the aiming beam) and inter-landmark spatial relationships (e.g., inter-landmark distances) are measured in the respective coordinate systems of the two images. If G i and G j are endoscope images of the same target site, by correcting for such differences in endoscope orientation, the image properties of endoscope images G i and G j are measured in the same coordinate system. Evaluating anatomical differences or similarities based on image properties (e.g., inter-landmark distances) between the two images is more robust to different imaging conditions. If G i and G j are endoscope images of different target sites (such as two images during endoscope translation in Figures 4A to 4F , by correcting for such differences in endoscope orientation, the inconsistencies between endoscope images G i and G j are reduced, and the integration of G i and G j (which is part of target map 500) can provide a more reliable representation of the extended surface area of target 101

[0107] Figure 6A A first endoscope orientation Θ1 is shown, in which the tip of endoscope 102 is perpendicular to the surface of target site 611 (i.e., Θ1 = 90 degrees), and lens system 118 is parallel to target site 611. Figure 6C An endoscope image 615 taken at endoscope orientation Θ1 is shown in Figure 6B A second endoscope orientation Θ2 is shown, in which the tip of endoscope 102 is tilted relative to target site 621 (i.e., Θ2 is an acute angle), and lens system 118 is not parallel to target site 621. Figure 6D An endoscope image 625 taken at endoscope orientation Θ2 is shown. Matching landmarks {M1, M2, M3} (e.g., in the form of intersecting line segments) can be identified from endoscope images 615 and 625 by landmark detector 321.

[0108] The target image generator 323 can use features generated from images 615 and 625 respectively to detect changes in endoscope orientation from θ1 to θ2, transform endoscope image 625 to correct for changes in endoscope orientation, and align the transformed image 625 with image 615 relative to matching landmarks {M1, M2, M3}. Figure 6C and Figure 6D As shown, due to the different orientations of the endoscopes, the spatial relationships (e.g., distances and relative positions) between the landmarks {M1, M2, M3} in image 615 may differ from the relative positions between the landmarks {M1, M2, M3} in image 625. In the example, the spatial relationships between the landmarks can be represented by the slope between two landmarks in the coordinate system (e.g., the slope k between landmarks M1 and M3). 13 Let ) represent the slope k 13 It can be calculated as the y-distance between M1 and M3 on the y-axis. 13 The distance x on the x-axis between M1 and M3 13 The ratio, i.e., k 13 =y 13 / x 13 To determine changes in endoscope orientation, the target image generator 323 can determine the first slope (e.g., k) between two landmarks in image 615. 13 =y 13 / x 13 The second slope (e.g., k) between the two identical landmarks in image 625 and... 13 '=y 13 ' / x 13 The relative slope (e.g., k) is compared. In the example shown, the relative slope (e.g., k) is compared. 13 With k 13 The ratio between ' and ' can indicate changes in endoscope orientation.

[0109] Alternatively, in some examples, the target map generator 323 may use geometric features generated from the aiming beam footprints in images 615 and 625, respectively, to detect changes in endoscope orientation. Figure 6A and Figure 6B The distal end 112a of the laser fiber (an example of optical path 112) is shown directing the aiming beam toward its respective target site. As shown in the respective endoscopic images 615 and 625, the resulting aiming beam footprints 612 and 622 have different geometric characteristics due to the different endoscope orientations. Corresponding to an endoscope orientation θ1 = 90°, Figure 6E The circular aiming beam footprint 612 with diameter d is shown. This corresponds to an endoscope orientation θ1 < 90°. Figure 6FAn elliptical aiming beam footprint 622 is shown, with a long axis 623 of length "a" and a short axis 624 of length "b". In an example, the target map generator 323 can determine the endoscope orientation using an elliptic axis length ratio R e = a / b. For the elliptical footprint 622, the elliptic axis length ratio R e > 1. A larger elliptic axis length ratio indicates a more tilted endoscope orientation. For a circular footprint 612 of diameter d, the long and short axes a = b = d, and the elliptic axis length ratio R e = 1. The target map generator 323 can determine a change in endoscope orientation based on a comparison between the elliptic axis length ratios computed from the aiming beam footprints 612 and 622, respectively, transform the endoscope image 625 to correct for the change in endoscope orientation, and align the transformed image 625 with the image 615 relative to the identified matching landmarks.

[0110] Referring back to Figure 3 , the endoscope tracker 330 can locate and track the endoscope tip during an endoscopic procedure using a pre-generated target map (e.g., the target map 500 as shown in Figure 5 ). The endoscope tracking can begin with capturing a real-time image or video signal from a surgical site of the target 101 using the imaging system 115, and generating real-time images or video frames using the video processor 320, which is similar to the discussion above regarding generating one of the images shown in Figures 4A to 4F to reconstruct the target map 500. The imaging system 115 can be positioned at an unknown endoscope location. The landmark detector 321 can identify one or more landmarks from the real-time image. The endoscope tracker 330 can register the real-time image of the target 101 with the pre-generated target map (e.g., the target map 500), and locate the site captured in the real-time image according to the target map.

[0111] The endoscope tracker 330 can determine a change in tissue state at the target site (e.g., a change from normal tissue to abnormal tissue, or a change from abnormal tissue to normal tissue). The endoscope tracker 330 can locate and track the endoscope tip during the procedure, for example, based on landmarks identified from the real-time image and stored landmarks associated with the target map. In an example, the endoscope tracker 330 can identify two or more matching landmarks between the landmarks of the target map and the landmarks of the real-time image, register the real-time image to the target map using the identified matching landmarks, and locate and track the endoscope tip based on the registration of the real-time image.

[0112] Figure 7An example of identifying matching landmarks between the live image 710 and the reconstructed target map 500 and registering the live image 710 to the target map 500 with respect to the matching landmarks is shown. In the example, matching landmarks can be identified based on the distance ratios (r) between the landmarks. As discussed above with reference to Figure 5 the target map 500 contains pairs of inter-landmark distances {d1, d2, d3,..., d K} where K represents the number of pairs of landmarks identified from the target map 500. Distance ratios {r1, r2,..., rM} can be calculated between any two of the K inter-landmark distances {d1, d2, d3,..., d K} where M represents the number of distance ratios.

[0113] According to one example, to identify matching landmarks, the endoscope tracker 330 can identify, among a set of intersecting landmarks (i.e., intersecting line segments) having respective intersection point locations, the intersecting landmark 711 in the live image that is closest to the current aiming beam footprint 701. Distances from the landmark 711 to other landmarks in the image 710 can be measured: a distance D1 to Pc, a distance D2 to P d , a distance D3 to P b , a distance D4 to P a , and so on. The endoscope tracker 330 can then calculate distance ratios (R) between distances originating from the same landmark (e.g., landmark 711 in the example): R1 = D1 / D2, R2 = D1 / D3, R3 = D1 / D4, and so on. The distance ratios {R1, R2, R3} can be compared to the distance ratios {r1, r2,..., rM} associated with the target map 500. When the distance ratios {R1, R2, R3} (corresponding to the originating landmark 711 in the live image 710) match the distance ratios {rx, ry, rz} (corresponding to the originating landmark P k ) in the target map 500, such that R1 = rx, R2 = ry, R3 = rz, then there is a high likelihood that the distances {D1, D2, D3, D4} match the distances {d1, d2, d3, d4}; and the landmarks {Pa, Pb, Pc, Pd} in the live image 710 match the landmarks {p1, p2, p3, p4} in the target map 500. The more distances that match, the higher the probability of landmark matching between the live image 710 and the map 500.

[0114] The endoscope tracker 330 can then determine the correspondence between distances {D1, D2, D3, D4} and distances {d1, d2, d3, d4} based on the ratio of the distances between the landmarks. For example, if D1 / D2 = d1 / d2, then D1 = d1 and D2 = d2. By checking various combinations until they all match, the endoscope tracker 330 can identify which distance in {D1, D2, D3, D4} corresponds to which distance in {d1, d2, d3, d4}. Since all matches are made relative to D1, if D1 is identified as d1, the remaining distances D2, D3, and D4 can be matched to d2, d3, and d4 respectively. The correspondence between {Pa, Pb, Pc, Pd} and {p1, p2, p3, p4} can also be determined through the correspondences established between D1 and d1, D2 and d2, D3 and d3, and D4 and d4.

[0115] Endoscope tracker 330 can register real-time image 710 to target image 500 using identified matching landmarks {Pa, Pb, Pc, Pd} that match landmarks {p1, p2, p3, p4} in target image 500. To correct for geometric distortions or deformations in the image, such as those caused by image magnification, reduction, rotation, or changes in endoscope orientation, endoscope tracker 330 can transform real-time image 710 or target image 500 in a manner similar to that discussed above, i.e., transforming a first endoscope image, aligning the first endoscope image with a second endoscope image, and reconstructing the panoramic target image using at least the transformed first and second images (as described above). Figures 4A to 4F (As discussed). Transformations can include one or more operations such as scaling, translation, or rotation. A transformation can be implemented as a transformation matrix multiplied by the image data (e.g., a data array) of the target image 500 in the coordinate system of the real-time image 710. Alternatively, the transformation can be applied to the real-time image 710.

[0116] like Figure 7 As shown, the transformation may include scaling Figure 500 by a scaling factor λ to correct for differences in image magnification or reduction between real-time image 710 and Figure 500. The scaled Figure 720 includes landmarks and their positions (e.g., relative distances to the aiming beam footprint) and inter-landmark distances, also scaled by the scaling factor λ. In the example, the scaling factor λ can be determined using the ratio of the distance between two matching landmarks (e.g., P1 and P2) in real-time image 710 to the distance between two corresponding landmarks (e.g., Pa and Pb) in Figure 500. In the example, the largest inter-landmark distance among the matching landmarks in real-time image 710 can be selected for calculating the distance ratio λ. For example, if D4 is the largest distance among {D1, D2, D3, D4}, then the scaling factor λ = D4 / d4.

[0117] The scaling factor λ can be determined alternatively or additionally by comparing the shape of the aiming beam footprint in real-time image 710 with the shape of the aiming beam footprint in Figure 500. In an example, the scaling factor can be determined using the ratio of the geometric features of the aiming beam footprint in real-time image 710 to the corresponding geometric features of the aiming beam footprint in Figure 500. For example, if real-time image 710 has a diameter of d... R Circular footprints (such as) Figure 6E As shown), while Figure 500 has a diameter of d. M For a circular footprint, the scaling factor λ = d R / d M In the example, if the real-time image 710 has an elliptical footprint (such as...), Figure 6F As shown), its major axis length is a. R The length of the minor axis is b R Figure 500 shows an elliptical footprint with a major axis length of a. M The length of the minor axis is b M Then the scaling factor λ = a R / a M , or = b R / b M .

[0118] The scaling factor λ calculated above assumes that the surface onto which the aiming beam is projected is flat. In some cases, the surface onto which the aiming beam is projected may not be perfectly flat, but rather have a three-dimensional shape. This may introduce variations in the calculated scaling factor λ. The system can adapt to variations in the ideal scaling factor λ. In the example, multiple aiming beam footprints can be captured when the aiming beam is pointed at different locations of a target part with a corresponding projection surface. The superposition of multiple aiming beam footprints can show variations in the shape of the aiming beam footprints. Geometric features (e.g., the diameter of a circular footprint or the length of the major or minor axis of an elliptical footprint) can be measured separately from the multiple aiming beam footprints, and the corresponding scaling factors can be calculated. The multiple scaling factors can be averaged or weighted to obtain the expected value of the scaling factor λ.

[0119] The scaling factor λ, as described above, is determined based on the assumption that the endoscope orientation remains substantially unchanged between real-time image 710 and Figure 500 (e.g., the tilt of the endoscope tip relative to the target surface is substantially the same). In cases where significantly different endoscope orientations exist, measures such as landmark position, inter-landmark distance, shape of the aiming beam footprint, and its geometric characteristics (e.g., the lengths of the major and minor axes) may be affected by the endoscope orientation. Real-time image 710 or Figure 500 can be transformed to correct for variations in endoscope orientation (e.g., according to information regarding...). Figures 6A to 6F (Description). Then the scaling factor λ can be determined based on the transformed image.

[0120] The scaled image 720 (including the landmarks therein) can be aligned with the target image 500 with respect to the identified matching landmarks (e.g., PI and Pa as shown in FIG. 730). The live image 710 can be translated toward the scaled image 720 such that Pa is at the same coordinate as PI of the scaled image 720 (denoted as PI(Pa)). Then, the scaled image 720 is rotated 730 by an angle a or ∠Pb-PI(Pa)-P2 clockwise. After the rotation, the matching landmark Pb is at the same coordinate as P2 of the scaled image 720 (denoted as P2(Pb)), as shown in the registered image 740. Since the scaling and rotation operations preserve the relative positions (e.g., angles) between the landmarks, the other matching landmarks P3 and P4 also overlap with the landmarks Pc and Pd on the scaled image 720, denoted as P3(Pc) and P4(Pd) in the registered image 740. Thus, the live image 710 is registered to the target image 500 with respect to the matching landmarks PI-P4 (corresponding to Pa-Pd in the image 710).

[0121] As described above, the registration of the live image taken during an endoscopic procedure with a target image can be used for various applications to improve the accuracy and efficiency of the endoscopic procedure. In an example, the image registration can assist an operator to identify a surgical site from a pre-generated target image in real time with improved accuracy. Because the target image stores the information of the endoscope tip position with respect to a plurality of stored landmarks, the image registration discussed herein can assist in locating and tracking the endoscope tip in real time throughout the procedure. For a target image that stores information about the target type (e.g., normal tissue or abnormal tissue) at various targeting beam positions, the endoscope tracker 330 can detect and track the change of tissue type at various target sites over time, or provide an assessment of the effectiveness of a treatment delivered at the target.

[0122] Figure 8 FIG. 8 is a flowchart illustrating a method 800 of endoscopic mapping of a target in a subject during a procedure. The method 800 can be implemented in and executed by a medical system (e.g., the system 100) for endoscopic procedures or variations thereof. Although the processes of the method 800 are depicted in a particular order, it is not required that the processes be performed in the particular order shown. In various examples, some of the processes can be performed in a different order than shown herein.

[0123] At 810, a targeting beam can be emitted from the endoscope tip and directed at a site of the target (e.g., a portion of the target 101). The targeting beam can be generated by a laser source, e.g., the second laser energy source 204. Alternatively, the targeting beam can be generated by other light sources and transmitted, e.g., via an optical fiber.

[0124] At 820, an image of the target site can be captured by an imaging system, e.g., imaging system 115. The image can be taken while the lens system 118 of the imaging system 115 is positioned at the endoscope position. The image can be taken while the target is illuminated by electromagnetic radiation in the optical range from UV to IR, also referred to as illumination light. The illumination light can be generated by a light source, e.g., light source 104, and transmitted to the target site via the light guide 120. In an example, the light source 104 can include two or more light sources that emit light with different illumination characteristics.

[0125] The aiming beam directed at the target site can fall within the FOV of the imaging system such that the image captured at the target site can include not only a graphical representation of the illuminated target (e.g., a surface of the target anatomy), but also the aiming beam footprint. This image can be displayed to the user, e.g., on the display 108 (as shown in any of FIGS. 1-3). The aiming beam footprint can be colored differently from the background of the endoscope image. In an example, the location of the aiming beam footprint can be identified from the endoscope image by matching the color of the aiming beam to the color of the pixels in the endoscope image. Figures 4A to 4F

[0126] The signal reflected from the target in response to the illumination light can be analyzed, e.g., using the spectrometer 208, to identify the target type at the aiming beam location of the target. In an example, based on the spectral characteristics of the reflected signal, the tissue at the target site can be identified as normal and abnormal tissue, or mucosal or muscle tissue, among other anatomical structure types. In some examples, the target site can be identified as one of a plurality of stone types with respective compositions.

[0127] The aiming beam footprint in the endoscope image can be marked with a visual identifier that indicates the tissue type at the aiming beam location of the target. In an example, the aiming beam footprint can be colored differently to indicate different tissue types. For example, if the target site is confirmed to be normal tissue, the aiming beam footprint can be colored green, or if the target site is confirmed to be abnormal tissue (e.g., cancer), the aiming beam footprint can be colored red. In some examples, the aiming beam footprint can be marked with an identifier to indicate a change in tissue type over time, or to indicate a treatment status.

[0128] At 830, one or more landmarks can be identified from the captured image of the target site, e.g., using the video processor 320, and landmark locations relative to the aiming beam footprint can be determined. In an example, the landmarks represent anatomical structures (e.g., blood vessels). The landmarks can be detected based on changes in the brightness of the pixels of the endoscope image. In an example, the landmarks can be detected using edge detection constrained by a contrast threshold, and the number of pixels between similar positive and negative contrast slopes.​

[0129] The position (e.g., X and Y distances) of one or more landmarks in the coordinate system of an endoscopic image can be determined relative to the aiming beam footprint in the same endoscopic image. In another example, landmark localization involves determining inter-landmark distances in the coordinate system of an endoscopic image. In some examples, a subset of detected landmarks can be selected based on the spatial distribution of landmarks in the endoscopic image. For example, the selected subset can include landmarks distributed in the endoscopic image (rather than a tightly-spaced cluster of landmarks at one region in the image). In another example, landmarks can be selected based on whether laser energy is activated on the landmark, as laser energy can distort such landmarks. For example, a landmark that is not activated by laser energy can be more favorably selected than another landmark that is activated by laser energy.

[0130] In some examples, the target can be illuminated by specific lighting conditions to improve landmark detection and localization. For example, the target can be illuminated by blue or green light to increase the contrast of the endoscopic image of the target and more clearly define vessels that are less likely to move or change over time. This allows for more consistent landmark detection and localization under slightly different illumination conditions.

[0131] At 840, a target map can be reconstructed, e.g., by integrating the multiple images based on the respective landmarks identified from the multiple images of the various portions of the target. For example, as the endoscope tip is translated over the target, either manually by an operator or automatically by an endoscope actuator, the endoscope distal tip is moved and positioned at different endoscope locations {L1, L2,..., L N} and the imaging system can capture a series of endoscopic images {G1, G2,..., G N} at respective multiple target portions {S1, S2,..., S N} of the target that fall within the FOV of the imaging system at the respective endoscope locations. Figures 4A to 4F An example of a sequence of images (or video frames) captured at different portions of a target is shown.

[0132] A target map can be reconstructed by integrating the multiple endoscopic images {G1, G2,..., G N}. The endoscopic images {G1, G2,..., G N} can be aligned with respect to landmarks identified from the images. In an example, between two endoscopic images G i and G j , matching landmarks can be identified that include two or more landmarks identified from image G i that match two or more landmarks identified from image G j . Then, image G iand G j The images can be aligned with respect to the identified matching landmarks.

[0133] In some examples, an endoscopic image (e.g., G i ) can be transformed prior to being aligned with another endoscopic image (e.g., G j ). The transformation can correct for geometric image distortion or deformation, such as image magnification or reduction caused by the endoscope distal tip being moved too close or too far from the target or body motion (e.g., breathing), image rotation caused by changes in the viewing direction of the imaging system toward the target, or distortion in length, shape, and other image characteristics caused by changes in the endoscope orientation. Examples of image transformations can include one or more of scaling, translation, rotation, or shear transformations of the image in a coordinate system, as well as other rigid, similarity-based, or affine transformations.

[0134] In an example, prior to aligning the first endoscopic image G i and the second endoscopic image G j , the image G j may be scaled by a scaling factor λ based on the ratio of the distance between the two matching landmarks in the image G i to the distance between the corresponding two landmarks in the image G j . In another example, the scaling factor λ can be determined based on the ratio of a geometric feature generated from the aiming beam footprint in the image G i to a geometric feature generated from the aiming beam footprint in the image G j . Examples of geometric features can include the diameter of a circular footprint, or the long (or short) axis length of an elliptical footprint (as discussed above with reference to Figure 7 ).

[0135] In an example, prior to aligning the first endoscopic image G i and the second endoscopic image G j , a change in endoscope orientation between the images G i and G j may be detected and corrected. In an example, the change in endoscope orientation can be determined based on a comparison of a first slope between two matching landmarks in the image G i and a second slope between the two matching landmarks in the image G j . In another example, the change in endoscope orientation can be determined based on a comparison of a first geometric feature of an aiming beam footprint in the image G i to a geometric feature of an aiming beam footprint in the image G jdetermined based on a comparison between the first and second geometric features of the aiming beam footprints in the images. In an example, at least one of the first or second aiming beam footprints has an elliptical shape with a major axis and a minor axis, and at least one of the first or second geometric features can include a ratio of the major axis length to the minor axis length of the elliptical shaped aiming beam footprint (as discussed above with reference to Figures 6A to 6F

[0136] For example Figure 5 As shown, the transformed images can be aligned and integrated into a reconstructed target map. The reconstructed map can include one or more of the following: a set of landmarks identified from the plurality of endoscopic images, aiming beam footprints generated during tissue mapping, target type identifiers (e.g., color-coded aiming beam footprints), positions of landmarks relative to aiming beam footprints, or relative positions between landmarks. The target map (including information about landmarks and aiming beam footprints) can be stored in memory 340. At 850, the target map can be used to locate and track the endoscope tip during endoscopic surgery (as discussed below with reference to Figure 9 Additionally or alternatively, the target map can be used to determine changes in tissue status at a target site (e.g., from normal to abnormal, or vice versa).

[0137] Systems and methods of image reconstruction and endoscope tracking according to various embodiments discussed herein can accommodate tolerances around which measurements are made. Comparisons or equations described herein or inferred from the description herein are not limited to exact equality. In an example, systems and methods described herein can first compare values for exact equality, but then gradually expand a tolerance around each calculation to determine an overlapping portion, which is then treated as equal. For example, when comparing anatomical maps from the same patient at two different times (which can be weeks, months, or years apart), the algorithm can gradually expand the tolerance window from the ideal value to a limit value, or can create a larger region around each landmark and then check for overlap using standard statistical methods (such as a t-test or Mann-Whitney median comparison) with a target confidence of, for example, 5 to 25%. In an example, the tolerance window or confidence interval can be between 0% and X% of the distance between each pair of landmarks being compared, where X% can be 20% in one example, or 25% in another example. Alternatively, the tolerance window or confidence interval can be gradually increased until a suitable match can be obtained between a substantial number of landmarks, for example, approximately 70-100% of the landmarks are found to be a match.

[0138] Figure 9 ​is a flowchart illustrating an example of a method 900 for endoscope tracking using a reconstructed target map (e.g., a target map generated using method 800). At 910, during an endoscopic procedure, real-time images of a surgical site of a target site can be captured, e.g., via an imaging system positioned at an unknown endoscope position. At 920, matching landmarks can be identified, the matching landmarks including two or more landmarks in the target map that respectively match two or more landmarks in the real-time images. The matching landmarks can be identified based on a distance ratio between the landmarks (as described above with reference to Figure 7 the discussion). At 930, the real-time images can be registered to the target map using the identified matching landmarks. The registration can include image transformations (e.g., translation, scaling, rotation, etc.) and image alignment (as described above with reference to Figure 7 the discussion). At 940, the endoscope tip can be positioned and tracked based on the registration of the real-time images. Because the target map stores position information of the endoscope tip relative to the plurality of landmarks, the endoscope tracker 330 can position and track the endoscope tip in real-time based on the landmarks on the registered images throughout the procedure. For a target map that stores information about tissue types (e.g., normal tissue or abnormal tissue at different target sites, e.g., as indicated by the aiming beam footprint), the endoscope tracker 330 can detect and track changes in tissue types at different sites of the target over time or effectiveness of a treatment delivered thereto.

[0139] Figure 10 A block diagram of an example machine 1000 is generally shown upon which any one or more of the techniques (e.g., methodologies) discussed herein can perform. Portions of the description can apply to a computing framework of portions of the system 100 (e.g., the endoscope controller 103).

[0140] In alternative implementations, the machine 1000 can operate as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 1000 can act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 1000 can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations, etc.

[0141] As described herein, examples may include logic or multiple components or mechanisms, or may be operated by logic or multiple components or mechanisms. A circuit group is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic, etc.). The members of a circuit group may vary over time and with changes in the underlying hardware. A circuit group includes members that can perform a specified operation individually or in combination during operation. In the example, the hardware of the circuit group may be invariably designed to perform the specified operation (e.g., hardwired). In the example, the hardware of the circuit group may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) to encode instructions for the specified operation, and the variably connected physical components may include computer-readable media that are physically modified (e.g., magnetically grounded, electrically grounded, movable placement of massless particles, etc.). When the physical components are connected, the underlying electrical properties of the hardware components are changed, for example, from an insulator to a conductor or from a conductor to an insulator. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create portions of the circuit group in the hardware via variable connections to perform the specified operation during operation. Therefore, when the device is in operation, the computer-readable medium is communicatively coupled to other components of the circuit group members. In the example, any component of the physical components can be used in more than one member of more than one circuit group. For example, in operation, an execution unit can be used at one point in time in a first circuit of a first circuit group, and can be reused at different times by a second circuit in the first circuit group or by a third circuit in the second circuit group.

[0142] The machine (e.g., computer system) 1000 can include a hardware processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1004 and a static memory 1006, some or all of which can communicate with one another via an interlink (e.g., bus) 1008. The machine 1000 can further include a display unit 1010 (e.g., a raster display, a vector display, a holographic display, etc.), an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In an example, the display unit 1010, input device 1012 and UI navigation device 1014 can be a touch screen display. The machine 1000 can additionally include a storage device (e.g., drive unit) 1016, a signal generation device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1021, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. The machine 1000 can include an output controller 1028, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0143] The storage device 1016 can include a machine readable medium 1022 on which is stored one or more sets of data structures or instructions 1024 (e.g., software) embodying any one or more of the techniques or functions described herein or by any one or more of the techniques or functions described herein. The instructions 1024 can also reside, completely or at least partially, within the main memory 1004, within static memory 1006, or within the hardware processor 1002 during execution thereof by the machine 1000. In an example, one or any combination of the hardware processor 1002, the main memory 1004, the static memory 1006, or the storage device 1016 can constitute machine readable media.

[0144] While the machine readable medium 1022 is illustrated as a single medium, the term“machine readable medium” can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 1024.

[0145] The term "machine-readable medium" can include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 1000 and that cause the machine 1000 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples can include solid-state memories, and optical and magnetic media. In examples, a massed machine-readable medium includes a machine-readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed machine-readable media are not transitory propagating signals. Specific examples of massed machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0146] The instructions 1024 can further be transmitted or received over a communications network 1026 using a transmission medium via the network interface device 1020 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks can include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi®, IEEE 802.16 family of standards known as WiMax®), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 1020 can include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network 1026. In an example, the network interface device 1020 can include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying the instructions for execution by the machine 1000, and includes digital or analog communications signals or other intangible media to facilitate communication of such software. The instructions 1024 can further be transmitted or received over a communications network 1026 using a transmission medium via the network interface device 1020 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks can include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi®, IEEE 802.16 family of standards known as WiMax®), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 1020 can include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network 1026. In an example, the network interface device 1020 can include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying the instructions for execution by the machine 1000, and includes digital or analog communications signals or other intangible media to facilitate communication of such software.

[0147] Supplementary notes

[0148] ​The above detailed description includes reference to the accompanying drawings, which form part of the detailed description. For illustration, the drawings show specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Furthermore, the inventors contemplate examples using any combination or substitution of these elements (or one or more aspects of these elements) shown or described with respect to a particular example (or one or more aspects of a particular example) or with respect to other examples shown or described herein (or one or more aspects of other examples).

[0149] In this document, as is common in patent documents, terms without quantifiers are used to include one or more, unrelated to any other instance or use of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to indicate a non-exclusive "or", such that "A or B" includes "A but not B", "B but not A", and "A and B". In this document, the terms "comprising" and "in..." are used as concise English equivalents to the corresponding terms "including" and "wherein". Furthermore, in the appended claims, the terms "comprising" and "including" are open-ended, meaning that a system, apparatus, article, composition, formulation, or process that includes elements other than those listed after this term in a claim is still considered to fall within the scope of that claim. Additionally, in the following claims, the terms "first", "second", and "third", etc., are used merely as designations and are not intended to impose numerical requirements on their objects.

[0150] The above description is intended to be illustrative and not restrictive. For example, the examples (or one or more aspects of the examples) described above can be used in combination with each other. Other embodiments can be used by those skilled in the art after consulting the above description. An abstract is provided to enable the reader to quickly determine the nature of the technical disclosure. The abstract is submitted under the understanding that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the specific embodiments described above, various features may be combined to organize the present disclosure. This should not be construed as meaning that all unclaimed disclosed features are necessary for any claim. Rather, the subject matter of the invention may lie in fewer than all features of a particular disclosed embodiment. Therefore, the appended claims are thus incorporated into the specific embodiments as examples or embodiments, wherein each claim is an independent, separate embodiment, and such embodiments are contemplated to be combined with each other in various combinations or substitutions. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A system for endoscopic mapping of a target, the system comprising: An imaging system configured to capture endoscopic images of the target, the endoscopic images including the footprint of a targeting beam directed at the target; as well as Video processor, the video processor being configured to: Identify one or more landmarks from the captured endoscopic images and determine the corresponding positions of the one or more landmarks relative to the trail of the aiming beam; as well as A target map is generated by integrating the multiple endoscopic images based on landmarks identified from one or more of the multiple endoscopic images.

2. The system according to claim 1, wherein, The video processor is configured to identify the tissue type at the location of the aiming beam and to mark the aiming beam footprint using a visual identifier that indicates the identified tissue type.

3. The system according to any one of claims 1-2, comprising a spectrometer communicatively coupled to the video processor, the spectrometer being configured to measure one or more spectral characteristics of an illumination light signal reflected from the target; in, The video processor is configured to identify the tissue type at the location of the aiming beam based on one or more spectral characteristics, and to mark the aiming beam footprint using a visual identifier that indicates the identified tissue type.

4. The system according to claim 2, wherein, The video processor is configured to identify the tissue type as normal or abnormal tissue.

5. The system according to claim 2, wherein, The video processor is configured to use different colors to mark the aiming beam footprint to indicate different tissue types.

6. The system according to any one of claims 1-2, wherein, The video processor is configured to identify one or more landmarks from the endoscope image based on changes in the brightness of pixels in the endoscope image.

7. The system according to claim 6, wherein, The one or more landmarks are represented as line segments or intersecting line segments in the endoscopic image.

8. The system according to any one of claims 1-2, wherein, The video processor is configured to: Based on whether the laser energy is activated at each target location where the identified landmarks are located, a subset of landmarks is selected from the landmarks identified in one or more of the plurality of endoscopic images; as well as The target image is generated by integrating the multiple endoscopic images based on a selected subset of landmarks.

9. The system according to any one of claims 1-2, wherein, The plurality of endoscopic images include images of various parts of the target, including a first endoscopic image of a first target part captured from a first endoscopic position and a second endoscopic image of a second target part captured from a second endoscopic position, wherein the video processor is configured to: Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; Align the first and second endoscopic images with respect to the matching landmark in the coordinate system of the first endoscopic image; and The target image is generated using at least aligned first and second endoscopic images.

10. The system according to claim 9, wherein, The video processor is configured to: Transforming the second endoscopic image includes scaling, translating, or rotating the second endoscopic image, or performing one or more of these operations. Align the transformed second endoscopic image and the first endoscopic image with respect to the matching landmark.

11. The system according to claim 10, wherein, The transformation of the second endoscopic image includes matrix multiplication performed by the transformation matrix.

12. The system according to claim 10, wherein, The video processor is configured to scale the second endoscopic image using a scaling factor based on the ratio of the distance between two matching landmarks in the first endoscopic image to the distance between two corresponding landmarks in the second endoscopic image.

13. The system according to claim 10, wherein, The video processor is configured to scale the second endoscopic image using a scaling factor based on the ratio of the geometry of the aiming beam footprint in the first endoscopic image to the geometry of the aiming beam footprint in the second endoscopic image.

14. The system according to claim 10, wherein, The video processor is configured to transform the second endoscopic image to correct for changes in endoscopic orientation between the first and second endoscopic images, the endoscopic orientation indicating the tilt of the endoscope tip relative to the target site.

15. The system according to claim 14, wherein, The video processor is configured to detect changes in the endoscope orientation using a first slope between two landmarks in the first endoscopic image and a second slope between two corresponding landmarks in the second endoscopic image.

16. The system according to claim 14, wherein, The video processor is configured to detect changes in the endoscope orientation using a first geometric feature of the first aiming beam footprint in the first endoscope image and a second geometric feature of the second aiming beam footprint in the second endoscope image.

17. The system according to claim 16, wherein, At least one of the first or second aiming beam footprint has an elliptical shape, the ellipse having a major axis and a minor axis; and At least one of the first geometric feature or the second geometric feature includes the ratio of the length of the major axis to the length of the minor axis.

18. The system according to any one of claims 1-2, comprising an endoscope tracking system, the endoscope tracking system being configured to: Identify matching landmarks from real-time images of the surgical site of the target captured from an unknown endoscopic location by the imaging system during endoscopic surgery, the matching landmarks including two or more corresponding landmarks in the target image that match two or more landmarks in the real-time image; The real-time image is registered to the target image using the matching delimiters; and Based on the registration of the real-time images, the position of the endoscope tip is tracked.

19. The system according to claim 18, wherein, The endoscopic tracking system is configured to identify the matching landmarks based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

20. The system according to claim 18, wherein, The endoscopic tracking system is configured to generate indications of changes in tissue type at the target site.

21. A method for endoscopic mapping of a target, the method comprising: Point the aiming beam at the target; An endoscopic image of the target is captured via an imaging system, the endoscopic image including the footprint of the aiming beam; The video processor identifies one or more landmarks from the captured endoscopic images and determines the corresponding positions of the one or more landmarks relative to the trail of the aiming beam. as well as The target image is generated by integrating the plurality of endoscopic images via the video processor based on landmarks identified from one or more of the plurality of endoscopic images.

22. The method of claim 21, comprising: The tissue type at the location of the aiming beam is identified using the illumination light signal reflected from the target; as well as The target beam footprint is marked using a visual identifier that indicates the type of tissue being identified.

23. The method according to any one of claims 21-22, wherein, Identifying one or more landmarks from the endoscopic image is based on changes in the brightness of the pixels in the endoscopic image.

24. The method according to any one of claims 21-22, wherein, The plurality of endoscopic images include images of various parts of the target, including a first endoscopic image of a first target part captured at a first endoscopic position and a second endoscopic image of a second target part captured at a second endoscopic position, the method comprising: Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; Align the first and second endoscopic images with respect to the matching landmark in the coordinate system of the first endoscopic image; and The target image is generated using at least aligned first and second endoscopic images.

25. The method according to claim 24, wherein, Aligning the first endoscopic image and the second endoscopic image includes: Transforming the second endoscopic image includes scaling, translating, or rotating the second endoscopic image, or performing one or more of these operations. Align the transformed second endoscopic image and the first endoscopic image with respect to the matching landmark.

26. The method of claim 25, wherein, Transforming the second endoscopic image includes scaling the second endoscopic image by a scaling factor based on the ratio of the distance between two matching landmarks in the first endoscopic image to the distance between two corresponding landmarks in the second endoscopic image.

27. The method according to claim 25, wherein, Transforming the second endoscopic image includes scaling the second endoscopic image by a scaling factor based on the ratio of the geometry of the aiming beam footprint in the first endoscopic image to the geometry of the aiming beam footprint in the second endoscopic image.

28. The method according to claim 25, wherein, Transforming the second endoscopic image includes correcting for a change in endoscopic orientation between the first and second endoscopic images, the endoscopic orientation indicating the tilt of the endoscope tip relative to the target site.

29. The method according to any one of claims 21-22, further comprising: During endoscopic surgery, the imaging system is used to capture real-time images of the target surgical site from an unknown endoscopic location; Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the target image that match two or more landmarks in the real-time image; The real-time image is registered to the target image using the matching delimiters; as well as Based on the registration of the real-time images, the position of the endoscope tip is tracked.

30. The method according to claim 29, wherein, The identification and matching of landmarks is based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

Images