Image reconstruction and endoscopic tracking

Abstract

To provide devices for endoscopic mapping and tracking of endoscope locations during a procedure.SOLUTION: Systems, devices and methods for endoscopic mapping of a target and tracking of endoscope locations inside a subject's body during a procedure are disclosed. An exemplary system comprises: an imaging system configured to capture an endoscopic image of the target that includes a footprint of an aiming beam directed at the target; and a video processor configured to identify one or more landmarks from the captured endoscopic image and determine their respective locations relative to the aiming beam footprint, and generate a target map by integrating a plurality of endoscopic images on the basis of landmarks identified from one or more of the endoscopic images. The target map can be used to track endoscope locations during an endoscopic procedure.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 055,936, filed on July 24, 2020, the content of which is hereby incorporated by reference in its entirety.

[0002] This specification generally relates to endoscopy, and more specifically, to systems and methods for endoscopic mapping of targets and tracking of endoscopic positions during procedures.

Background Art

[0003] Endoscopes are typically used to provide access to internal locations of a subject so that a physician can have visual access. An endoscope is usually inserted into a patient's body, sends light to a target (e.g., a target anatomical structure or object to be examined), and collects the light reflected from the target. The reflected light carries information about the object being examined and can be used to create an image of the object. Some endoscopes include a working channel through which an operator can perform suction, or pass instruments such as brushes, biopsy needles, or forceps, or perform minimally invasive surgery to remove unwanted tissue or foreign objects from the patient's body.

[0004] Some endoscopes can include or be used with a laser or plasma system for delivering surgical laser energy to the anatomical structure or object of a target, such as soft or hard tissue. Examples of laser treatments include ablation, coagulation, vaporization, fragmentation, etc. In lithotripsy applications, lasers have been used to break down stone structures in the kidneys, gallbladder, ureters, etc., or to excise large stones into smaller fragments, among other stone - forming regions.

[0005] Video systems have been used to help physicians or technicians visualize the treatment site and navigate the endoscope during endoscopic procedures. Image-guided endoscopy generally requires locating the endoscope and tracking its movement in the coordinate system of the target area. Accurate mapping of the target area, as well as efficient endoscope positioning and tracking, can improve the precision of endoscopic manipulation during endoscopic procedures, enhance the physician's or technician's ability to intervene, and potentially improve the effectiveness of treatments (e.g., laser treatments). [Overview of the Initiative] [Means for solving the problem]

[0006] This specification describes systems, devices, and methods for endoscopic mapping of a target during a procedure and tracking the position of an endoscope within a subject's body. An exemplary system comprises an imaging system configured to capture an endoscopic image of a target, including the footprint of a aiming beam directed at the target, and a video processor configured to identify one or more landmarks from the captured endoscopic image, determine the position of each relative to the aiming beam footprint, and generate a target map by integrating multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images. The target map can be used to track the position of the endoscope during an endoscopic procedure.

[0007] Example 1 is a system for endoscopic mapping of a target. The system comprises an imaging system configured to capture an endoscopic image of a target, wherein the endoscopic image includes the footprint of a targeting beam directed at the target, and a video processor configured to identify one or more landmarks from the captured endoscopic image and determine their respective positions relative to the targeting beam footprint, and to generate a target map by integrating multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images.

[0008] In Example 2, the subject of Example 1 optionally includes a video processor that can be configured to identify tissue types at the location of the aiming beam and to mark the aiming beam footprint with a visual identifier indicating the identified tissue type.

[0009] In Example 3, one or more subjects from Examples 1-2 optionally include a spectrometer communicatively coupled to a video processor, the spectrometer configured to measure one or more spectral characteristics of an illumination light signal reflected from a target, and the video processor configured to identify a tissue type at the location of the aiming beam based on the one or more spectral characteristics and to mark the aiming beam footprint with a visual identifier indicating the identified tissue type.

[0010] In Example 4, one or more subjects from Examples 2 to 3 optionally include a video processor that can be configured to identify tissue types as normal or abnormal tissue.

[0011] In Example 5, one or more subjects from Examples 2 to 4 optionally include a video processor that can be configured to mark aiming beam footprints in different colors to indicate different tissue types.

[0012] In Example 6, one or more subjects from Examples 1 to 5 optionally include a video processor that can be configured to identify one or more landmarks from an endoscopic image based on changes in the brightness of pixels in the endoscopic image.

[0013] In Example 7, the subject of Example 6 optionally includes one or more landmarks represented in the endoscopic image as line segments or intersecting line segments.

[0014] In Example 8, one or more subjects from Examples 1 to 7 optionally include a video processor that can be configured to select a subset of landmarks from landmarks identified from one or more of a plurality of endoscopic images, based on whether laser energy is activated at each target site where the identified landmarks are located, and to generate a target map by integrating the plurality of endoscopic images based on the selected subset of landmarks.

[0015] In Example 9, one or more subjects from Examples 1 to 8 optionally include a plurality of endoscopic images that include images of various parts of the target, including a first endoscopic image of a first target area captured from a first endoscopic position and a second endoscopic image of a second target area captured from a second endoscopic position, and the video processor is configured to identify matching landmarks including two or more landmarks in the first endoscopic image that match two or more corresponding landmarks in the second endoscopic image, to align the first and second endoscopic images with respect to the matching landmarks in the coordinate system of the first image, and to generate a target map using at least the aligned first and second images.

[0016] In Example 10, the subject of Example 9 optionally includes a video processor that can be configured to transform a second image, including one or more of scaling, translation, or rotation of the second image, and to align the transformed second image and the first image with respect to matching landmarks.

[0017] In Example 11, the subject of Example 10 optionally includes a second image transformation which may include matrix multiplication by a transformation matrix.

[0018] In Example 12, one or more subjects from Examples 10 to 11 optionally include a video processor that can be configured to scale the second image using a magnification factor based on the ratio of the distance between two of the matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0019] In Example 13, one or more subjects from Examples 10 to 12 optionally include a video processor that can be configured to scale the second image by a magnification based on the ratio of the geometric features of the aiming beam footprint in the first image to the geometric features of the aiming beam footprint in the second image.

[0020] In Example 14, one or more subjects from Examples 10 to 13 optionally include a video processor that can be configured to transform the second image in order to compensate for a change in the orientation of the endoscope between the first image and the second image, where the orientation of the endoscope indicates the inclination of the tip of the endoscope relative to the target site.

[0021] In Example 15, the subject of Example 14 optionally includes a video processor which can be configured to detect changes in the orientation of the endoscope using a first gradient between two of the matching landmarks in the first image and a second gradient between the corresponding two landmarks in the second image.

[0022] In Example 16, one or more subjects from Examples 14-15 optionally include a video processor that can be configured to detect changes in the orientation of the endoscope using a first geometric feature of the aiming beam footprint in a first image and a second geometric feature of the aiming beam footprint in a second image.

[0023] In Example 17, the subject of Example 16 optionally includes at least one of a first or second aiming beam footprint which may have an elliptical shape with major and minor axes, and at least one of a first or second geometric feature which may include a ratio of the length of the major axis to the length of the minor axis.

[0024] In Example 18, one or more subjects from Examples 1 to 17 optionally include an endoscope tracking system configured to identify matching landmarks from real-time images of the treatment site of a target captured by an imaging system from an unknown endoscope position during an endoscopic procedure, including two or more landmarks in a target map that match two or more corresponding landmarks in the real-time images, register the real-time images in the target map using the matching landmarks, and track the position of the tip of the endoscope based on the registration of the real-time images.

[0025] In Example 19, the subject of Example 18 optionally includes an endoscopic tracking system that can be configured to identify matching landmarks based on one or more ratios of distances between landmarks in a real-time image and one or more ratios of distances between landmarks in a target map.

[0026] In Example 20, one or more subjects from Examples 18-19 optionally include an endoscopic tracking system that can be configured to generate a display of changes in tissue type at a target site.

[0027] Example 21 is a method for endoscopic mapping of a target. The method comprises the steps of: directing a aiming beam towards a target; capturing an endoscopic image of the target via an imaging system, wherein the endoscopic image includes the footprint of the aiming beam; identifying one or more landmarks from the captured endoscopic image via a video processor and determining the position of each of the one or more landmarks relative to the aiming beam footprint; and generating a target map by integrating multiple endoscopic images via a video processor based on the landmarks identified from one or more of the multiple endoscopic images.

[0028] In Example 22, the subject of Example 21 optionally includes the steps of identifying a tissue type at the position of the aiming beam using illumination light signals reflected from the target, and marking the aiming beam footprint using a visual identifier indicating the identified tissue type.

[0029] In Example 23, one or more themes from Examples 21 to 22 optionally include the step of identifying one or more landmarks from an endoscopic image based on changes in the brightness of pixels in the endoscopic image.

[0030] In Example 24, any one or more themes from Examples 21 to 23 optionally include a plurality of endoscopic images including images of various parts of a target, which include a first endoscopic image of a first target area captured at a first endoscopic position and a second endoscopic image of a second target area captured at a second endoscopic position, and the method comprises the steps of: identifying matching landmarks which include two or more landmarks in the first endoscopic image that match two or more corresponding landmarks in the second endoscopic image; aligning the first and second endoscopic images with respect to the matching landmarks in the coordinate system of the first image; and generating a target map using at least the aligned first and second images.

[0031] In Example 25, the subject of Example 24 optionally includes a step of aligning the first and second endoscopic images, which includes the step of transforming the second image, which involves one or more of scaling, translation, or rotation of the second image, and the step of aligning the transformed second image and the first image with respect to a matching landmark.

[0032] In Example 26, the subject of Example 25 optionally includes a step of transforming a second image, which includes scaling the second image by a magnification factor based on the ratio of the distance between two of the matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0033] In Example 27, one or more subjects from Examples 25-26 optionally include a step of transforming a second image, which includes scaling the second image by a magnification based on the ratio of the geometric features of the aiming beam footprint in the first image to the geometric features of the aiming beam footprint in the second image.

[0034] In Example 28, one or more subjects from Examples 25 to 27 optionally include a step of transforming a second image, which includes a step of correcting for changes in the orientation of the endoscope between the first image and the second image, where the orientation of the endoscope indicates the inclination of the tip of the endoscope relative to the target site.

[0035] In Example 29, the subject of Example 28 optionally includes the step of detecting a change in the orientation of the endoscope using a first gradient between two of the matching landmarks in the first image and a second gradient between the corresponding two landmarks in the second image.

[0036] In Example 30, one or more subjects from Examples 28 to 29 optionally include the step of detecting a change in the orientation of the endoscope using a first geometric feature of the aiming beam footprint in a first image and a second geometric feature of the aiming beam footprint in a second image.

[0037] In Example 31, one or more subjects from Examples 21 to 30 optionally include the steps of capturing a real-time image of the target treatment site from an unknown endoscopic position using an imaging system during an endoscopic procedure, identifying matching landmarks including two or more landmarks in a target map that match two or more corresponding landmarks in the real-time image, registering the real-time image in the target map using the matching landmarks, and tracking the position of the tip of the endoscope based on the registration of the real-time image.

[0038] In Example 32, the subject of Example 31 optionally includes the step of identifying matching landmarks based on one or more ratios of distances between landmarks in a real-time image and one or more ratios of distances between landmarks in a target map.

[0039] Embodiment 33 is at least one non-temporary machine-readable storage medium which, when performed by one or more processors of the machine, causes the machine to perform an operation comprising: directing a aiming beam at a target; capturing an endoscopic image of the target, wherein the endoscopic image includes a footprint of the aiming beam; identifying one or more landmarks from the captured endoscopic image and determining the position of each of the one or more landmarks relative to the aiming beam footprint; and generating a target map by integrating the multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images.

[0040] In Example 34, the subject of Example 33 optionally includes causing the machine to perform an operation further comprising identifying a tissue type at the location of the aiming beam and marking the aiming beam footprint with a visual identifier indicating the identified tissue type.

[0041] In Example 35, one or more subjects from Examples 33 to 34 optionally include an instruction causing a machine to perform an operation further comprising identifying one or more landmarks from an endoscopic image based on changes in the brightness of pixels in the endoscopic image.

[0042] In Example 36, one or more subjects from Examples 33 to 35 optionally include a plurality of endoscopic images that include images of various parts of a target, including a first endoscopic image of a first target site captured at a first endoscopic position and a second endoscopic image of a second target site captured at a second endoscopic position, wherein the command causes the machine to perform operations further comprising: identifying matching landmarks including two or more landmarks in the first endoscopic image that match two or more corresponding landmarks in the second endoscopic image; aligning the first and second endoscopic images with respect to the matching landmarks in the coordinate system of the first image; and generating a target map using at least the aligned first and second images.

[0043] In Example 37, the subject of Example 36 optionally includes the operation of aligning the first and second endoscopic images by transforming the second image, which involves one or more of scaling, translation, or rotation of the second image, and aligning the transformed second image and the first image with respect to a matching landmark.

[0044] In Example 38, the subject of Example 37 optionally includes scaling the second image by a magnification factor based on the ratio of the distance between two of the matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0045] In Example 39, one or more subjects from Examples 37-38 optionally include a step of scaling the second image by a magnification based on the ratio of the geometric features of the aiming beam footprint in the first image to the geometric features of the aiming beam footprint in the second image.

[0046] In Example 40, one or more themes from Examples 37 to 39 are such that the operation of transforming the second image is to correct for a change in the orientation of the endoscope between the first image and the second image, wherein the orientation of the endoscope optionally includes indicating the inclination of the tip of the endoscope relative to the target site.

[0047] In Example 41, the subject of Example 40 optionally includes causing the command to perform an operation which further comprises detecting a change in the orientation of the endoscope using a first gradient between two of the matching landmarks in a first image and a second gradient between the corresponding two landmarks in a second image.

[0048] In Example 42, one or more subjects from Examples 40 to 41 optionally include a command to cause the machine to perform an operation further comprising detecting a change in the orientation of the endoscope using a first geometric feature of the aiming beam footprint in a first image and a second geometric feature of the aiming beam footprint in a second image.

[0049] In Example 43, one or more subjects from Examples 33 to 42 optionally include causing a command to perform an operation further comprising: capturing a real-time image of the target treatment site from an unknown endoscopic position during an endoscopic procedure; identifying matching landmarks including two or more landmarks in a target map that match two or more corresponding landmarks in the real-time image; registering the real-time image in the target map using the matching landmarks; and tracking the position of the tip of the endoscope based on the registration of the real-time image.

[0050] In Example 44, one or more themes from Examples 33 to 43 optionally include an operation to identify matching landmarks based on one or more ratios of distances between landmarks in a real-time image and one or more ratios of distances between landmarks in a target map.

[0051] This abstract is a summary of some of the teachings of this application and is not intended to be an exclusive or exhaustive representation of the subject matter. Further details relating to the subject matter are set forth in the detailed description and the attached claims. Other aspects of this disclosure will become apparent to those skilled in the art upon reading and understanding the following detailed description and viewing the drawings that form part thereof. None of these should be constrained. The scope of this disclosure is defined by the attached claims and their legal equivalents.

[0052] Various embodiments are shown as examples in the accompanying drawings. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the subject matter. [Brief explanation of the drawing]

[0053] [Figure 1] This figure shows an example of a medical system for use in endoscopic procedures. [Figure 2] Figure 1 is a schematic diagram of a part of the system shown. [Figure 3] This block diagram shows an example of an endoscopic controller for controlling various parts of the system shown in Figure 1. [Figure 4A] This figure shows an example of a sequence of endoscopic images or video frames captured at different target sites and landmarks, as well as the targeting beam footprint detected therefrom. [Figure 4B] This figure shows an example of a sequence of endoscopic images or video frames captured at different target sites and landmarks, as well as the targeting beam footprint detected therefrom. [Figure 4C] This figure shows an example of a sequence of endoscopic images or video frames captured at different target sites and landmarks, as well as the targeting beam footprint detected therefrom. [Figure 4D]This figure shows an example of a sequence of endoscopic images or video frames captured at different target sites and landmarks, as well as the targeting beam footprint detected therefrom. [Figure 4E] This figure shows an example of a sequence of endoscopic images or video frames captured at different target sites and landmarks, as well as the targeting beam footprint detected therefrom. [Figure 4F] This figure shows an example of a sequence of endoscopic images or video frames captured at different target sites and landmarks, as well as the targeting beam footprint detected therefrom. [Figure 5] This figure shows an example of a target map reconstructed from multiple endoscopic images or video frames captured at different target sites. [Figure 6A] This figure illustrates, as an example, the influence of the endoscope's orientation on endoscopic image features, and the correction for changes in the endoscope's orientation between endoscopic images. [Figure 6B] This figure illustrates, as an example, the influence of the endoscope's orientation on endoscopic image features, and the correction for changes in the endoscope's orientation between endoscopic images. [Figure 6C] This figure illustrates, as an example, the influence of the endoscope's orientation on endoscopic image features, and the correction for changes in the endoscope's orientation between endoscopic images. [Figure 6D] This figure illustrates, as an example, the influence of the endoscope's orientation on endoscopic image features, and the correction for changes in the endoscope's orientation between endoscopic images. [Figure 6E] This figure illustrates, as an example, the influence of the endoscope's orientation on endoscopic image features, and the correction for changes in the endoscope's orientation between endoscopic images. [Figure 6F] This figure illustrates, as an example, the influence of the endoscope's orientation on endoscopic image features, and the correction for changes in the endoscope's orientation between endoscopic images. [Figure 7]This figure shows an example of recognizing matching landmarks between a real-time image and a reconstructed target map, and registering the real-time image with respect to the matching landmarks in the target map. [Figure 8] This flowchart illustrates a method for endoscopic mapping of targets within a subject's body during treatment. [Figure 9] This flowchart illustrates an example of a method for endoscopic tracking using a reconstructed target map. [Figure 10] This is a block diagram illustrating an exemplary machine on which any one or more of the techniques (e.g., methodologies) discussed herein may be performed. [Modes for carrying out the invention]

[0054] Minimally invasive endoscopic surgery is a surgical procedure in which a rigid or flexible endoscope is introduced into a target area of ​​the subject's body through a natural opening or a small incision in the skin. To provide the surgeon with visual feedback of the surgical site and surgical instruments, the endoscope can be used to introduce additional surgical instruments, such as laser fibers, into the subject's body through similar ports.

[0055] Endoscopic surgery may include preoperative and intraoperative phases. The preoperative phase includes acquiring images or video frames of the target anatomical structure or object using an imaging system (e.g., a video camera) and reconstructing a map using those images or video frames. The map may be used for diagnostic evaluation or endoscopic surgical planning. In the intraoperative phase, the endoscope can be introduced into the target treatment site. The surgeon can move and rotate the distal end of the endoscope and acquire real-time images of the treatment site via an imaging system positioned at the distal end of the endoscope. The position and orientation of surgical tools (e.g., laser fibers) at the distal end of the endoscope can be monitored and tracked throughout the procedure.

[0056] One conventional intraoperative tracking technique involves a freehand method in which the surgeon views the surgical field on a monitor displaying real-time images or video of the surgical field without an automated tracking or navigation system. This technique cannot establish image-to-image relationships that facilitate tracking the position and orientation of the endoscopic surgical tool relative to the target. Another technique involves a navigation-based tracking system, such as an optical or electromagnetic tracking system, that tracks the position and orientation of the endoscopic surgical tool. An image registration procedure can be performed to align real-time images to a target map. Reference markers displayed on the real-time images are used as a reference to guide the surgeon with real-time feedback on the position and orientation of the endoscopic surgical tool. Reference markers can be external objects attached to the patient or internal anatomical references. External references may lack positional consistency and increase the complexity of the system. Using internal references may generally restrict the physical movement of the endoscope, such as requiring the scope to touch the anatomical reference during the procedure, potentially increasing procedure time. Conventional navigation-based endoscopic tracking systems may also suffer from reduced tracking performance if there is image distortion or deformation due to changes in the camera's position, line of sight, and orientation (e.g., tilt or skew) relative to the target plane. For at least the reasons stated above, the inventors recognized an unmet need for an improved endoscopic mapping and tracking system that is more robust against image distortion or deformation when used in endoscopic procedures.

[0057] Described herein are systems, devices, and methods for endoscopic mapping of a target and tracking the position of an endoscope within a subject's body during a procedure. An exemplary system comprises an imaging system configured to capture an endoscopic image of a target, including the footprint of a aiming beam directed at the target, and a video processor configured to identify one or more landmarks from the captured endoscopic image, determine their respective positions relative to the aiming beam footprint, and generate a target map by integrating multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images. The target map can be used to track the position of the endoscope during a procedure.

[0058] The systems, devices, and methods according to various embodiments discussed herein can provide improved endoscopic mapping and tracking of the target during endoscopic procedures. According to various examples of this disclosure, various image features can be generated from endoscopic images, including, for example, landmarks and their positions relative to the aiming beam footprint, spatial relationships between landmarks, and the shape and geometric properties of the aiming beam footprint. Image registration, target map reconstruction, and endoscopic tracking based on these image features described herein are more resilient to rotation, zooming, panning, changes in camera position, line of sight, or changes in endoscopic orientation during endoscopic procedures. Improved navigation and endoscopic tracking can enhance the operator's intervention capabilities and the precision of endoscopic movements, reduce procedure time, and improve overall procedure effectiveness, patient safety, and system reliability.

[0059] The subjects described herein include, but are not limited to, arthroscopy, bronchoscopy, colonoscopy, laparoscopy, neuroendoscopy, and endoscopic cardiac surgery, and can be applied to a variety of endoscopic applications. Examples of endoscopic cardiac surgery include, but are not limited to, endoscopic coronary artery bypass surgery and endoscopic mitral and aortic valve repair and replacement. In this specification, “endoscopic” is broadly defined as the characterization of images obtained by any type of endoscope having the ability to image from inside the body. Examples of endoscopes for the purposes of the present invention include, but are not limited to, any type of flexible or rigid scope (e.g., endoscopes, arthroscopes, bronchoscopes, cholangioscopy, colonoscopes, cystoscopes, duodenoscopes, gastroscopy, hysteroscopes, laparoscopes, laryngoscopy, neuroscopy, otoscopes, push enteroscopes, nasopharyngoscopes, sigmoidoscopy, sinusoscopes, thoracoscopy, etc.) and any scope-like devices equipped with an imaging system (e.g., nesting cannulas with imaging). The imaging is localized, and the surface image can be acquired optically using optical fibers, lenses, or a miniaturized (e.g., CCD-based) imaging system. Examples of fluorescence fluoroscopy for the purposes of the present invention include, but are not limited to, X-ray imaging systems.

[0060] Figure 1 shows an example of a medical system 100 for use in endoscopic procedures. The system 100 comprises an endoscope 102, an endoscope controller 103, a light source 104, a laser device 106, and a display 108. A schematic diagram of part of the system 100 is shown in Figure 2. The endoscope 102 may include an insertion section 110 at its distal end and an operating unit 107 at its proximal end. The insertion section 110 can be inserted into a target site of a patient, capture an image of the target 101, and optionally perform a procedure within it. The insertion section 110 can be formed using an illumination fiber (illumination guide), an electrical cable, an optical fiber, etc. In the example shown in Figure 1, the insertion section 110 has a distal end 110a containing an imaging unit, a bendable curved section 110b including a plurality of curved pieces, and a flexible tube section 110c provided on the proximal end side of the curved section 110b.

[0061] Referring to Figure 2, the distal end 110a may include an illumination guide 120 coupled to a light source 104 and configured to project illumination light 230 onto the target 101 via an illumination lens 122. The distal end 110a may include an observation unit, such as an imaging system 115 configured to image the target 101. The imaging system 115 may include an image sensor 116 and an associated lens system 118. An example of the image sensor 116 may include a CCD or CMOS camera that is sensitive to ultraviolet (UV), visible (VIS), or infrared (IR) wavelengths. The endoscope 102 may include an insertion port 107b located inside the endoscope 102 and coupled to a treatment tool channel 102a that extends along the insertion section 110. An optical path 112, positioned within the channel 102a through the insertion port 107b, has a proximal end operably connected to a laser device 106 and extends distally from the distal end opening 102b of the channel 102a. Laser energy, such as the treatment beam 240a or the aiming beam 240b, is transmitted through the optical path 112 and emitted from the distal end 112a of the optical path 112, and can be directed towards the target 101. The endoscope 102 may optionally include an air / water supply nozzle (not shown) at its distal end 110a.

[0062] The operating unit 107 may be configured to be held by the surgeon. The operating unit 107 may be located at the proximal end of the endoscope 102 and is configured to communicate with the endoscope controller 103 and the light source 104 via a flexible universal cord 114 extending from the operating unit 107. As shown in Figure 1, the operating unit 107 includes a bending knob 107a for bending the bending section 110b vertically and horizontally, a therapeutic tool insertion port 107b for inserting a therapeutic tool such as medical forceps or an optical path 112 into the body cavity of the subject, and a number of switches 107c for operating peripheral devices such as the endoscope controller 103, the light source 104, an air supply device, a water supply device, or a gas supply device. The therapeutic tool, such as the optical path 112, is inserted through the channel 102a from the therapeutic tool insertion port 107b so that its distal end is exposed at the distal end of the insertion section 110 through the opening 102b of the channel 102a (see Figure 2).

[0063] The endoscope controller 103 can control the operation of one or more elements of the system 100, such as a display 108 that displays an image of the target 101 based on an image or video signal sensed by a light source 104, a laser device 106, or an image sensor 116. The distal end of the endoscope 102 may be positioned and oriented so that the aiming beam 240b is directed towards the target position in the field of view (FOV) of the imaging system 115, and the endoscope image includes the footprint of the aiming beam 240b. Although the aiming beam 240b is shown as a laser beam emitted from a laser energy source, other light sources can be used to generate a aiming beam that moves along an optical fiber. The endoscope controller 103 can reconstruct a map of the target by applying image processing to the endoscope image and integrating multiple endoscope images. In some examples, the endoscope controller 103 can use the reconstructed target map to locate and track the tip of the endoscope during an endoscopic procedure. An example of the endoscope controller 103, including endoscopic mapping of a target and tracking of the endoscope position, is discussed below with reference to, for example, Figure 3.

[0064] The universal cord 114 includes illumination fibers, cables, etc. The universal cord 114 may branch at its proximal end. One end of the branched end is connector 114a, and the other proximal end of the branched end is connector 114b. Connector 114a is detachable from the connector of the endoscope controller 103. Connector 114b is detachable from the light source 104. The universal cord 114 propagates illumination light from the light source 104 to the distal end 110a via connector 114b and the illumination guide 120. Furthermore, the universal cord 114 can transmit images or video signals captured by the imaging system 115 to the endoscope controller 103 via signal lines 124 (see Figure 2) within the cord and via connector 114a. The endoscope controller 103 performs image processing of the image or video signals output from connector 114a and controls at least some of the components constituting the system 100.

[0065] The light source 104 can generate illumination light while the endoscope 102 is being used in a procedure. The light source 104 may include, for example, a xenon lamp, a light-emitting diode (LED), a laser diode (LD), or any combination thereof. In one example, the light source 104 may include two or more light sources that emit light having different illumination characteristics, called illumination modes. Under the control of the endoscope controller 103, the light source 104 emits light and supplies it to the endoscope 102, which is connected via the illumination guide of the connector 114b and the universal cord 114, as illumination light inside the body of the subject being treated. The illumination mode may be a white light illumination mode, or a special light illumination mode such as a narrowband imaging mode, an autofluorescence imaging mode, or an infrared imaging mode. Special light illumination can concentrate and intensify light of a specific wavelength, for example, to better visualize surface microvessels and mucosal surface structures to highlight the subtle contrast of mucosal irregularities.

[0066] The display 108 includes, for example, a liquid crystal display, an organic electroluminescent display, etc. The display 108 can display information, including endoscopic images of a target subject, for image processing by the endoscope controller 103 via the video cable 108a. In some examples, one or more endoscopic images may each include the footprint of the aiming beam 240b. The surgeon can observe and track the behavior of the endoscope inside the subject's body by operating the endoscope 102 while viewing the images displayed on the display 108.

[0067] The laser device 106 is intended for use with an optical path 112, such as a laser fiber. Referring to Figure 2, the laser device 106 may include one or more energy sources, such as a first energy source 202 and a second energy source 204, to generate laser energy coupled to the proximal end of the optical path 112. In one example, the user may select an energy source via a button 106a on the laser device 106 (see Figure 1) or a foot switch (not shown), through a user interface on software or a display 108, or through other manual or automatic inputs known in the art.

[0068] The first energy source 202 may be optically coupled to the optical path 112 and configured to deliver a therapeutic beam 240a to the target 101 through the optical path 112. The first energy source 202 may include, but is not limited to, a thulium laser used to generate laser light for delivery to the target tissue through the optical path 112 in order to operate in different therapeutic modes such as cutting (excision) mode and coagulation (hemostasis) mode. Other energy sources known in the art for such treatment of tissue, or for any other therapeutic mode, such as Ho:YAG, Nd:YAG, and CO2, as well as others known in the art, may also be used for the first energy source 202.

[0069] A second energy source 204 may be optically coupled to the optical path 112 and configured to direct a targeting beam 240b towards the target 101 through the optical path 112. Although the targeting beam 240b is shown as a laser beam emitted from a laser source, other light sources may be used to generate the targeting beam 240b traveling along the optical fiber. The targeting beam 240b may be emitted when the target is illuminated by illumination light 230. In some examples, the second energy source 204 may emit at least two different targeting beams, the first targeting beam having at least one characteristic different from the second targeting beam. Such different characteristics may include wavelength, power level, and / or emission pattern. For example, the first targeting beam may have a wavelength in the range of 500 nm to 550 nm, and the second targeting beam may have a wavelength in the range of 635 nm to 690 nm. The characteristics of different aiming beams can be processed by the endoscope controller 103 and selected based on the visibility of the aiming beam in the image displayed on the display 108 under a specific illumination mode provided by the light source 104.

[0070] The laser device 106 may include a controller 206, which has hardware such as a microprocessor, to control the operation of the first energy source 202 and the second energy source 204. In the example shown in Figure 2, in response to the illumination light 230, the light reflected from the target 101 can enter the optical path 112 from the distal end 112a. The optical path 112, configured to transmit the laser beam, can also be used as a path for returning the reflected light to the laser device 106. A splitter 205 can collect the reflected light and split it from the laser beam delivered to the target 101 via the same optical path 112. The laser device 106 may include a spectrometer 208 operably coupled to the splitter 205 and configured to detect the light reflected from the splitter. Alternatively, the reflected light may be guided through an optical path (e.g., an optical fiber) separated from the optical path 112. The spectrometer 208 can be operably coupled to a dedicated optical path and detect the light reflected from there.

[0071] The spectrometer 208 can measure one or more spectral properties from the sensed reflected signal. Examples of the spectrometer 208 may include, among others, a Fourier transform infrared (FTIR) spectrometer, a Raman spectrometer, a UV-VIS spectrometer, a UV-VIS-IR spectrometer, or a fluorescence spectrometer. Spectral properties may include properties such as reflectance, reflectance spectrum, and absorption index. Spectral properties may indicate a structural category (e.g., anatomical tissue or calculus) or a specific structural type that indicates the chemical composition of the target.

[0072] Figure 3 is a block diagram showing an example of an endoscope controller 103 for use in system 100. The endoscope controller 103 comprises hardware such as a microprocessor for performing operations according to various examples described herein. The endoscope controller 103 may include a device controller 310, a video processor 320, an endoscope tracking device 330, and memory 340. The device controller 310 can control the operation of one or more components of system 100, such as an endoscope 102, a display 108, a light source 104, or a laser device 106.

[0073] The video processor 320 receives image or video signals from the imaging system 115 via signal lines 124 and can process the image or video signals to generate images or video frames that can be displayed on the display 108. In some examples, multiple endoscopic images (e.g., video frames) may be generated and displayed on the display 108. Multiple endoscopic images may be captured at the same location on the target 101 when the tip of the endoscope and the imaging system 115 remain stationary, while the distal end 112a of the optical path 112 (e.g., laser fiber) moves, directing the laser beam to different locations on the target. In some examples, multiple endoscopic images of the same target location may be captured at different times. Laterally captured images may be registered with previously captured images of the same target location through processes such as image transformation and / or image registration. The registered images may be used to determine changes in the tissue state at the target location. Additionally or alternatively, multiple endoscopic images may be captured at different locations on the target 101, such as when the distal end of the endoscope pans across the target 101. During endoscopy panning, the distal end of the endoscope is moved manually by the surgeon or automatically by the endoscope actuator to various endoscope positions {L1, L2, ..., L N It can be moved and positioned at} (i.e., at the distal end of the endoscope). The imaging system 115, under the control of the endoscope controller 103, jointly covers the substantial surface area of ​​target 101 with each target site {S1, S2, ..., S N In} a sequence of images (or video frames) {G1, G2, ..., G N For example, an endoscopic image G of the target area Si within the FOV of the imaging system 115 can be captured. i The lens system 118 is positioned at endoscope position L i It can be photographed when positioned and oriented in a different endoscopic position L. j When moved, another endoscopic image G j However, different target areas S fall within the FOV of the imaging system 115. jcan be captured. The video processor 320 creates a map of the target 101, for example, according to various examples further described below with reference to FIGS. 4A to 4F and FIG. 5, and can integrate the resulting images {G1, G2, …, G N}

[0074] As described above, the tip of the endoscope is positioned and oriented such that the aiming beam 240b fits within the FOV of the imaging system 115, and the aiming beam footprint can be captured in the endoscope image (e.g., Gi). In one example, the video processor 320 can color the aiming beam footprint with a color different from the background of the endoscope image. The video processor 320 can identify where the aiming beam 240b is currently irradiating by matching the color of the aiming beam 240b to the color of the pixels of the endoscope image.

[0075] The video processor 320 may include a landmark detector 321 configured to detect one or more landmarks from an endoscopic image. Landmarks may be manually created by the surgeon or automatically identified using an image processing algorithm. In one example, the landmark detector 321 may detect landmarks based on changes in the brightness of pixels in the endoscopic image. In one example, the landmark detector 321 may detect landmarks using edge detection constrained by a minimum contrast threshold, as well as the number of pixels between similar positive and negative contrast gradients. Detected landmarks may indicate blood vessels. Edge detection may include detecting a light-to-dark transition in pixel brightness indicating the start of a blood vessel segment, and a subsequent dark-to-light transition in pixel brightness indicating the end of a blood vessel segment. Additional criteria may be applied to confirm the detection of blood vessels. For example, if the subsequent transition occurs in at least a user-defined number of dark pixels and is bounded on both sides by a threshold number of bright pixels, the edge defined by the transition between the bright and dark pixels may be used as a landmark, provided that those points along one or both edges extend for a length at least longer than another threshold. In another example, a detected edge is identified as a blood vessel if a linear regression of the edge's pixels generates a straight line with an R-squared or other goodness-of-fit greater than the target threshold, such as 0.8 in this example.

[0076] Landmarks can take different forms in an endoscopic map. In one example, a landmark may be represented as a line segment in an endoscopic image. In another example, a landmark may be represented in an endoscopic image as two or more line segments that intersect at a point called an intersection landmark or point landmark. In some examples, for two adjacent, non-intersecting, and non-parallel line segments (such as within a certain distance range), the landmark detector 321 may algorithmically extend one line segment until it intersects the other, in order to create a point landmark.

[0077] The landmark detector 321 can locate landmarks relative to the aiming beam footprint in the coordinate system of the endoscopic image. For example, the position of a point landmark in the endoscopic map may be represented by a vector between the point landmark and the aiming beam footprint, or by distances along the X and Y axes in the coordinate system. In some examples, the landmark detector 321 may determine spatial relationships between landmarks, such as distances and gradients, in the coordinate system of the endoscopic image. According to various examples, information about landmarks and their positions, aiming beam footprints, and spatial relationships between landmarks is stored in the memory 340 and used for endoscopic image registration, target map reconstruction, or endoscopic tracking during endoscopic procedures, according to the various examples described herein.

[0078] The landmark detector 321 may select a subset of detected landmarks for storage in the memory 340, or for applications such as image registration, target map reconstruction, or endoscopic tracking. For example, a subset of landmarks may be selected based on the location of the landmarks, such as the spatial distribution of landmarks in the endoscopic image. For instance, landmarks spread across the entire endoscopic image may be favorably selected over clusters of densely packed landmarks in the endoscopic image. In another example, a subset of landmarks may be selected based on whether laser energy is activated at the target site where the landmarks are located. Since laser energy can affect the accuracy and consistency of landmark detection, in one example, a landmark not activated by laser energy may be favorably selected over another landmark that is activated by laser energy.

[0079] In some examples, the endoscope controller 103 may control the light source 104 or illumination lens 122 to generate special illumination conditions for the target 101 in order to improve landmark detection and localization. For example, the light source 104 may provide blue or green illumination to increase the contrast of the endoscopic image of the target 101 and to more clearly define vascular systems that are less likely to move or change over time. This allows for more consistent landmark detection and localization under slightly different illumination conditions. In one example, the endoscope controller 103 may temporarily change the illumination, such as turning on a green or blue light source to optimally identify a landmark, and then return to the normal illumination mode after the landmark has been identified.

[0080] The video processor 320 may include a target identifier 322 configured to identify the target type at the aiming beam position of the target 101. In one example, the identification of the target type may be based on one or more spectral characteristics of the illumination light reflected from the target 101. The spectroscopic characteristics may be measured using a spectrometer 208. The identified target type may include an anatomical tissue type or a calculus type. Examples of calculus types may include calculus or calculus fragments in various calculus-forming areas such as the urinary tract, gallbladder, nasal passage, gastrointestinal tract, stomach, or tonsils. Examples of anatomical tissue types may include, among other things, soft tissues (e.g., muscles, tendons, ligaments, blood vessels, fascia, skin, fat, and fibrous tissue), hard tissues such as bone, and connective tissues such as cartilage. In one example, the target identifier 322 may identify the tissue type at the aiming beam position of the target 101 as normal and abnormal tissue, or mucous membrane or muscle tissue, based on the characteristics of the reflected illumination signal.

[0081] The video processor 320 may mark the targeting beam footprint in the endoscopic image so that it is displayed on the display 108, using a visual identifier that indicates the identified tissue type. For example, the visual identifier may include a color code so that the targeting beam footprint can be colored with different colors to indicate different tissue types. For example, the targeting beam footprint may be colored green if the target site is recognized as normal tissue, or red if the target site is recognized as abnormal tissue (e.g., cancer). For example, the video processor 320 may mark the targeting beam footprint using a visual identifier that indicates a change in tissue type over time at the target site (e.g., from normal to abnormal, or vice versa), such as by using a different color than the color that represents normal or abnormal tissue. In another example, the video processor 320 may mark the targeting beam footprint using a visual identifier that indicates the treatment status at the target site. For example, if the target site is being treated (e.g., with laser treatment), the targeting beam footprint may be represented by a dot of a different color than the color that represents normal or abnormal tissue.

[0082] Figures 4A to 4F are sequences of endoscopic images (e.g., video frames) taken at target 101, including, but not limited to, other anatomical structures of interest, such as the inside of the kidney, bladder, urethra, or ureter {G1, G2, ..., G N This shows}. The endoscopic image may be displayed on display 108. As described above, the endoscopic image may be taken at the same target site at different times, or at different target sites {S1, S2, ...S} as the tip of the endoscope pans across target 101. N It may be photographed in}. The target identifier 322 is the target region {S1, S2, ...S NAt the aiming beam position corresponding to}, the target type can be identified. The video processor 320 can mark the aiming beam footprint in the corresponding endoscopic image with a visual identifier (e.g., color) that identifies the corresponding target type, detect and locate landmarks from the endoscopic image, and integrate the endoscopic image into a target map of target 101 based on the landmarks identified from the endoscopic image.

[0083] As shown in Figures 4A to 4F, endoscopic images 410 to 460 are generated as the tip of the endoscope pans across the target 101, during which time the distal end of the endoscope is moved manually or automatically to different endoscopic positions. Figure 4A shows endoscopic image 410, which includes a graphic representation of the irradiated target area S1 entering the FOV of the imaging system 115, and a circular aiming beam footprint 412. The aiming beam footprint 412 is colored green to indicate that the tissue at the aiming beam position is normal tissue. In this example, image 413 of the distal end 112a of the optical path 112 (e.g., laser fiber) and image 414 of the distal portion of the endoscope 102 are also shown in image 410.

[0084] Image 410 also includes landmarks 415A–415C, such as those detected by target identifier 322. In this example, landmarks 415B and 415C are represented by two line segments that intersect to form a point landmark, while landmark 415A is represented by two line segments that algorithmically intersect (for example, by projecting one line segment toward the other) to form a point landmark. As described above with reference to Figure 3, the locations of landmarks 415A–415C may be determined by target identifier 322. Image 410, containing information on the aiming beam footprint 412 and landmarks 415A–415C, may be stored in memory 340.

[0085] When the distal end of the endoscope is moved manually or automatically to a new endoscopic position, another endoscopic image 420 may be generated, as shown in Figure 4B. The new endoscopic image 420 includes a graphic representation of the new illuminated target site S2 corresponding to the new endoscopic position, and a new aiming beam footprint 422. The aiming beam footprint 422 is colored green because the tissue in the current aiming beam is recognized as normal tissue. New landmarks may be included in the image if detected from the current endoscopic image. In this example, no new landmarks are detected from the endoscopic image 420. The previous aiming beam footprint 412 and previously generated landmarks 415A-415C may be retained in the current image 420 if they are located within the imaging system's FOV at the current endoscopic position.

[0086] If the distal end movement is performed in small step sizes, the illuminated target regions S1 and S2 will overlap, and both endoscopic images 410 and 420 may cover a common region of target 101, as shown in Figures 4A and 4B. One or more matching landmarks may be identified from endoscopic images 410 and 420. Such matching landmarks may be used to align images 410 and 420 to reconstruct a map of the target, according to the various examples described below.

[0087] The endoscopic panning process may be continued, and additional endoscopic images may be generated. Figure 4C shows image 430, which includes a graphic representation of the newly illuminated target area S3, and the new aiming beam footprint 432 is colored red to indicate that abnormal tissue has been recognized at the current aiming beam position. New landmarks 435A-435B may be detected from the current endoscopic image. Previous aiming beam footprints and previously generated landmarks (e.g., 415A-415C) may still be within the imaging system's FOV at the current endoscopic position and may be retained in image 430.

[0088] Figure 4D shows image 440, which includes a graphic representation of the newly illuminated target area S4, and the new aiming beam footprint 442 is colored green to indicate that normal tissue is recognized at the current aiming beam position. New landmarks 445A-445C may be detected from the current endoscopic image. The new aiming beam footprint (including its position and color representing the tissue type) and new landmarks (including their positions relative to the aiming beam footprint), as well as previous aiming beam footprints and previously generated landmarks, may be stored in memory 340. Previous aiming beam footprints and previously generated landmarks (e.g., 435B) within the imaging system's FOV at the current endoscopic position may be retained in image 440.

[0089] The distal end of the endoscope may be moved manually or automatically along a specific path or according to a specific pattern, so that the endoscopic images generated during the panning process can together provide panoramic coverage of the substantial surface area of ​​target 101. As a non-limiting example, Figures 4A–4F show rectangular paths indicated by the aiming beam footprint in the corresponding endoscopic images. After a horizontal movement to the left (as shown in Figures 4A–4D), the distal end of the endoscope moves vertically upward, during which endoscopic images may be taken at each target site. Figure 4E shows image 450 containing a graphic representation of the newly illuminated target site S5, where the new aiming beam footprint 452 is colored green to indicate that normal tissue is recognized at the current aiming beam position. New landmarks 455A–455B may be detected from the current endoscopic image. Previous aiming beam footprints and previously generated landmarks within the FOV of the imaging system at the current endoscopic position are retained in image 450.

[0090] After an upward vertical movement, the distal end of the endoscope performs a horizontal movement to the right, during which endoscopic images can be acquired at each target site. Figure 4F shows image 460, which includes a graphic representation of the illuminated target site S6, including some of the previously visited illuminated sites captured in image 410, and the new aiming beam footprint 462 is colored green to indicate that normal tissue is recognized at the current aiming beam position. New landmarks 465A-465B can be detected from the current endoscopic image. Previous aiming beam footprints and previously generated landmarks within the imaging system's FOV at the current endoscopic position are retained in image 460. This includes the previously generated landmark 435A, which once went outside the endoscopic images 440 and 450.

[0091] Returning to Figure 3, the video processor 320 processes multiple endoscopic images (or video frames) {G1, G2, ..., G} of various target sites of target 101 stored in memory 340. N It may include a target map generator 323 configured to reconstruct the target map by integrating}. As described above with reference to Figures 4A to 4F, the stored endoscopic image G i This may include a graphic representation of the illuminated target area, a aiming beam footprint (including its location and color representing the tissue type), and one or more landmarks (including their positions relative to the spatial relationship between the aiming beam footprint and the landmarks). The target map generator 323 stores endoscopic images {G1, G2, ..., G} based on the landmarks. N Image registration may be performed to align the} with its relative position. Image registration is performed on a second endoscopic image (e.g., a different second target site S). j Image G taken at j A first endoscopic image (for example, a first target site S) that matches two or more landmarks identified from the first endoscopic image. i Image G taken at iThis may include identifying matching landmarks, including two or more landmarks identified from the endoscopic images, and aligning the second image with respect to the identified matching landmarks. For example, image 430 in Figure 4C may be aligned with image 420 in Figure 4B using matching landmarks 415A-415C present in both images 420 and 430. The aligned images can then be stitched together with respect to the matching landmarks to reconstruct a target map. In some examples, the landmark detector 321 may adjust the landmark detection algorithm (e.g., lower the edge detection threshold) to allow more landmarks to be identified from the endoscopic images. Multiple landmarks can increase the likelihood of identifying matching landmarks between images and improve the accuracy of image alignment.

[0092] Figure 5 shows an example of a target map 500 for a target 101, such as a substantial area of ​​the bladder. In addition to the stitched images, the reconstructed map may further include one or more of the following: a set of landmarks identified from multiple endoscopic images, a aiming beam footprint, a target type identifier (e.g., a color code for the aiming beam footprint), the location of the landmarks relative to the aiming beam footprint, or spatial relationships between landmarks. The target map 500 may be used to assist in medical diagnosis or treatment planning, such as locating and tracking the endoscope during an endoscopic procedure.

[0093] Geometric distortion or deformation may be introduced into the endoscopic images used to reconstruct the target map (for example, endoscopic images 410-460 in Figures 4A-4F used to reconstruct target map 500 in Figure 5), resulting in inconsistencies in common areas between images. For example, moving the tip of the endoscope closer to or further away from the target, or body movements (e.g., breathing), may cause the image to enlarge or shrink. Changes in the observation direction (direction from the imaging system 115 at the distal end of the endoscope toward the target) may cause image rotation. In some cases, changes in the orientation of the endoscope may cause geometric distortion or deformation. In this specification, the orientation of the endoscope refers to the inclination or distortion of the tip of the endoscope relative to the target plane. Changes in the orientation of the endoscope from one image to another may cause distortion of length, shape, and other geometric properties. To compensate for such distortion or deformation, in some examples, the target map generator 323 may transform the image before aligning it with another image. Examples of image transformations may include one or more of the following transformations of images in a coordinate system: scaling, translation, rotation, or shearing, among other stiffness-based, similarity-based, or affine transformations. In one example, an image may be scaled by a magnification based on the distance between landmarks measured from two images, as described below with respect to Figure 7, or by a magnification based on geometric features measured from the aiming beam footprints in two images, as described below. In one example, changes in the orientation of the endoscope may be corrected, as described below, based on the gradient between landmarks measured from two images, as described below with respect to Figures 6A to 6F, or by geometric features measured from the aiming beam footprints in two images, as described below.

[0094] The transformation can be implemented as a transformation matrix multiplied by the image data (e.g., a data array). The transformed image can be aligned with another image with respect to matching landmarks between the two images. In some examples, the alignment can be based on the gradients of multiple landmarks relative to each other. Such alignment may not be affected by the distance between landmarks (differences in magnification or the distance of the endoscope from the target). The image transformations by various examples described herein can improve the robustness of target map reconstruction against differences in rotation, magnification, or reduction of endoscopic images.

[0095] Landmarks in the transformed endoscopic image can often be stored in memory 340 in their transformed state, as if they were on the two-dimensional projection plane of the target. Therefore, landmarks in the transformed endoscopic image are invariant to surface heterogeneity, distortion, rotation, and scaling, among other distortions or deformations. Stored transformed endoscopic images can be integrated to form an integrated target map. Stored landmarks can serve as a basis for comparing new images, or as a basis for transforming a new image into a two-dimensional projection plane and registering the new image in the stored target map, for example, as described below with reference to Figure 7.

[0096] Figures 6A to 6F illustrate the effect of endoscope orientation on endoscopic image characteristics and a method for correcting different endoscope orientations between two endoscopic images. The endoscope orientation correction method discussed herein can be applied to image registration applications, such as registering real-time intraoperative endoscopic images to a target map (e.g., target map 500), as described below with reference to Figure 7. Endoscope orientation refers to the tilt angle or skew angle θ of the lens system 118 with respect to the target plane. Two endoscopic images G taken with different endoscope orientations. i and G jRegarding this, image characteristics such as the location of landmarks (e.g., distance to the aiming beam footprint) and the spatial relationships between landmarks (e.g., distance between landmarks) are measured in the coordinate systems of the two images. i and G j If the endoscopic images are of the same target site, the difference in orientation of the endoscope can be corrected for the endoscopic image G. i and G j The image properties are measured in the same coordinate system. Evaluating anatomical differences or similarities based on image properties between two images (e.g., distance between landmarks) is more robust to different image conditions. i and G j If the endoscopic images are of different target sites (for example, two of the images in Figures 4A to 4F during endoscopic panning), the endoscopic image G can be corrected for such differences in the orientation of the endoscope. i and G j The discrepancy between (as part of target map 500) G can be reduced. i and G j Integration with this may provide a more reliable representation of the expanded surface area of ​​target 101.

[0097] Figure 6A shows a first endoscope orientation θ1, where the tip of the endoscope 102 is perpendicular to the surface of the target site 611 (i.e., θ1 = 90 degrees) and the lens system 118 is parallel to the target site 611. Figure 6C shows an endoscopic image 615 taken in endoscope orientation θ1. Figure 6B shows a second endoscope orientation θ2, where the tip of the endoscope 102 is tilted relative to the target site 621 (i.e., θ2 is acute) and the lens system 118 is not parallel to the target site 621. Figure 6D shows an image 625 taken in endoscope orientation θ2. Coincident landmarks {M1, M2, M3}, such as the shapes of intersecting line segments, can be identified by the landmark detector 321 from endoscopic images 615 and 625.

[0098] The target map generator 323 can use features generated from images 615 and 625, respectively, to detect changes in the orientation of the endoscope from θ1 to θ2, transform the endoscope image 625 to compensate for the change in the orientation of the endoscope, and align the transformed image 625 with image 615 with respect to matching landmarks {M1, M2, M3}. As shown in Figures 6C and 6D, differences in the orientation of the endoscope may cause the spatial relationships between landmarks {M1, M2, M3} in image 615 (e.g., distance and relative position) to appear different from the relative positions between landmarks {M1, M2, M3} in image 625. In one example, the spatial relationship between landmarks is determined by the gradient k between landmarks M1 and M3. 13 This can be represented by the gradient between two landmarks in the coordinate system, which is the distance y-axis between M1 and M3. 13 And the distance x along the x-axis between M1 and M3 13 The ratio to, that is, k 13 =y 13 / x 13 It can be calculated as follows. To determine the change in the orientation of the endoscope, the target map generator 323 calculates the first gradient between two landmarks in image 615 (for example, k 13 =y 13 / x 13 ) and the second gradient between the same two landmarks in image 625 (for example, k 13 '=y 13 ' / x 13 It can be compared with '). In the illustrated example, k 13 and k 13 The relative gradient, such as the ratio between ' and ', may indicate a change in the orientation of the endoscope.

[0099] Additionally or alternatively, in some examples, the target map generator 323 may detect changes in the orientation of the endoscope using geometric features generated from the aiming beam footprints in images 615 and 625, respectively. Figures 6A and 6B show the distal end 112a of a laser fiber (an example of an optical path 112) directing the aiming beam to the respective target site. The resulting aiming beam footprints 612 and 622 have different geometric properties due to the difference in the orientation of the endoscope, as shown in the respective endoscope images 615 and 625. Corresponding to the orientation of the endoscope θ1 = 90°, Figure 6E shows a circular aiming beam footprint 612 with diameter d. Corresponding to the orientation of the endoscope θ1 < 90°, Figure 6F shows an elliptical aiming beam footprint 622 with a major axis 623 of length "a" and a minor axis 624 of length "b". In one example, the target map generator 323 uses an elliptical axis length ratio R e The orientation of the endoscope can be determined using = a / b. For an elliptical footprint 622, the ratio of the lengths of the elliptical axes is R e >1. A larger ratio of the lengths of the elliptical axes indicates that the endoscope is tilted. In the case of a circular footprint 612 with diameter d, the major and minor axes are a=b=d, and the ratio of the lengths of the elliptical axes is R e = 1. The target map generator 323 can determine changes in the orientation of the endoscope based on a comparison between the elliptic axis length ratios calculated from the aiming beam footprints 612 and 622, respectively, transform the endoscope image 625 to correct for changes in the orientation of the endoscope, and align the transformed image 625 with image 615 with respect to identified matching landmarks.

[0100] Returning to Figure 3, the endoscopic tracking device 330 may locate and track the tip of the endoscope during an endoscopic procedure using a pre-generated target map (e.g., target map 500 shown in Figure 5). Endoscopic tracking may begin with capturing real-time images or video signals from the treatment site of target 101 using the imaging system 115 and generating real-time images or video frames using the video processor 320, similar to what was described above regarding generating endoscopic images (e.g., one of those shown in Figures 4A to 4F) for reconstructing target map 500. The imaging system 115 may be positioned at an unknown endoscopic location. The landmark detector 321 can identify one or more landmarks from the real-time images. The endoscopic tracking device 330 can register the real-time images to a pre-generated target map of target 101 (e.g., target map 500) and from the target map, pinpoint the areas captured in the real-time images.

[0101] The endoscopic tracking device 330 can determine changes in the tissue state at the target site (for example, a change from normal tissue to abnormal tissue, or vice versa). The endoscopic tracking device 330 can locate and track the tip of the endoscope during the procedure based on landmarks identified from real-time images and stored landmarks associated with the target map. For example, the endoscopic tracking device 330 can recognize two or more matching landmarks between landmarks in the target map and landmarks in the real-time image, register the real-time image in the target map using the recognized matching landmarks, and locate and track the tip of the endoscope based on the registration of the real-time image.

[0102] Figure 7 shows an example of recognizing matching landmarks between a real-time image 710 and a reconstructed target map 500, and registering the real-time image 710 in the target map 500 with respect to the matching landmarks. In one example, matching landmarks may be recognized based on the distance ratio (r) between landmarks. As described above with reference to Figure 5, the target map 500 uses pairwise landmark distances {d1, d2, d3, ..., d K The distance ratio {r1, r2, ..., rM} is the distance between K landmarks {d1, d2, d3, ..., d K It can be calculated between any two of the following, where M represents the number of distance ratios.

[0103] For example, to recognize a matching landmark, the endoscope tracking device 330 can identify the intersection landmark 711 that is closest to the current aiming beam footprint 701 in the real-time image from a set of intersection landmarks (i.e., intersection line segments) each having an intersection position. The distance from landmark 711 to other landmarks in image 710 can be measured, where D1 is the distance to Pc, D2 is the distance to Pd, D3 is the distance to Pb, D4 is the distance to Pa, and so on. The endoscope tracking device 330 can then calculate the distance ratios (R) between distances arising from the same landmark, such as landmark 711 in this example, R1=D1 / D2, R2=D1 / D3, R3=D1 / D4, and so on. The distance ratios {R1, R2, R3} can be compared to the distance ratios {r1, r2, ..., rM} associated with the target map 500. Distance ratio {R1, R2, R3} (corresponding to the original landmark 711 in real-time image 710) is distance ratio {rx, ry, rz} (original landmark P in target map 500) kIf the distances correspond to R1=rx, R2=ry, and R3=rz, then the distances {D1, D2, D3, D4} are likely to coincide with the distances {d1, d2, d3, d4}, and the landmarks {Pa, Pb, Pc, Pd} in the real-time image 710 are likely to coincide with the landmarks {p1, p2, p3, p4} in the target map 500. The longer the distance at which a coincidence is possible, the higher the probability that the landmarks will coincide between the real-time image 710 and the map 500.

[0104] Next, the endoscope tracking device 330 may determine the correspondence between distances {D1, D2, D3, D4} and distances {d1, d2, d3, d4} based on the ratio of distances between landmarks. For example, if D1 / D2 = d1 / d2, it may be determined that D1 = d1 and D2 = d2. By checking various combinations until all match, the endoscope tracking device 330 may identify which distances in {D1, D2, D3, D4} correspond to which distances in {d1, d2, d3, d4}. Since all matches were made for D1, identifying D1 as d1, the remaining distances D2, D3, and D4 may match d2, d3, and d4. The established correspondences between D1 and d1, between D2 and d2, between D3 and d3, and between D4 and d4 can also determine the correspondences between {Pa, Pb, Pc, Pd} and {P1, P2, P3, P4}.

[0105] The endoscope tracking device 330 may register the real-time image 710 in the target map 500 using identified matching landmarks {Pa, Pb, Pc, Pd} in the real-time image 710 that match landmarks {P1, P2, P3, P4} in the target map 500. To correct geometric distortion or deformation of the image due to image scaling, rotation, or changes in the orientation of the endoscope, the endoscope tracking device 330 may transform the real-time image 710 or the target map 500 in a similar manner to that described above with respect to the transformation of the first endoscope image, align it with the second endoscope image, and reconstruct the panoramic target map using at least the transformed first and second images as described above with reference to Figures 4A to 4F. The transformation may include one or more of scaling, translation, or rotation, among other operations. The transformation may be implemented as a transformation matrix multiplied by the image data (e.g., data array) of the target map 500 in the coordinate system of the real-time image 710. Alternatively, the transformation may be applied to the real-time image 710.

[0106] As shown in Figure 7, the transformation may include scaling map 500 by a magnification λ to compensate for different image enlargement or shrinkage between the real-time image 710 and map 500. The scaled map 720 includes landmarks with their positions (e.g., relative distance to the aiming beam footprint) and inter-landmark distances also scaled by the magnification λ. In one example, the magnification λ may be determined using the ratio of the distance between two matching landmarks in the real-time image 710 (e.g., P1 and P2) to the distance between two corresponding landmarks in map 500 (e.g., Pa and Pb). In one example, the largest inter-landmark distance may be selected from the matching landmarks in the real-time image 710 to calculate the distance ratio λ. For example, if D4 is the largest distance among {D1, D2, D3, D4}, then the magnification λ = D4 / d4.

[0107] The magnification λ may, alternatively or additionally, be determined by comparing the shape of the aiming beam footprint in the real-time image 710 with the shape of the aiming beam footprint in the map 500. For example, the magnification λ may be determined by using the ratio of the geometric features of the aiming beam footprint in the real-time image 710 to the corresponding geometric features of the aiming beam footprint in the map 500. For example, if the real-time image 710 has a diameter d R It has a circular footprint (as shown in Figure 6E), and map 500 has a diameter d M If it has a circular footprint, the magnification λ = d R / d M For example, in the real-time image 710, the length of the major axis is a R The length of the minor axis is b R It has an elliptical footprint (as shown in Figure 6F), and map 500 has a major axis length of a M The length of the minor axis is b M If it has an elliptical footprint, the magnification is λ=a R / a M , or λ=b R / b M That is the case.

[0108] The magnification λ calculated above assumes that the surface onto which the aiming beam is projected is planar. In some cases, the aiming beam projection surface may not be perfectly flat, but instead have a three-dimensional shape. This may cause a change in the calculated magnification λ. The system may need to account for the change from the ideal magnification λ. For example, multiple aiming beam footprints may be captured when the aiming beam is directed to slightly different positions on the target area on each projection surface. When multiple aiming beam footprints are superimposed, changes in shape in the aiming beam footprint may become apparent. Geometric features (e.g., the diameter of a circular footprint or the length of the major or minor axis of an elliptical footprint) can be measured from each of the multiple aiming beam footprints, and a corresponding multiple magnification can be calculated. To obtain the expected value of the magnification λ, an average or weighted average of the multiple magnifications may be performed.

[0109] The above determination of the magnification λ is based on the assumption that the orientation of the endoscope does not substantially change between the real-time image 710 and the map 500 (for example, the skew of the endoscope tip relative to the target plane is substantially the same). If substantially different endoscope orientations exist, metrics such as the location of landmarks, the distance between landmarks, the shape of the aiming beam footprint, and their geometric properties (e.g., the lengths of the major and minor axes) may be affected by the endoscope orientation. The real-time image 710 or map 500 may be transformed to compensate for changes in the endoscope orientation, as described in Figures 6A to 6F, etc. The magnification λ may then be determined from the transformed image.

[0110] As shown in Figure 730, a scaled map 720 (containing landmarks) can be aligned with a target map 500 with respect to identified matching landmarks such as P1 and Pa. The real-time image 710 can be translated toward the scaled map 720 so that Pa is at the same coordinates as P1 on the scaled map 720, indicated by P1(Pa). The scaled map 720 can then be rotated clockwise by an angle α, i.e., ∠Pb-P1(Pa)-P2 (730). After rotation, the matching landmark Pb is at the same coordinates as P2 on the scaled map 720, indicated by P2(Pb), as shown in the registered map 740. Because the scaling and rotation operations preserve the relative positions (e.g., angles) between landmarks, other matching landmarks P3 and P4 also overlap with landmarks Pc and Pd on the scaled map 720, indicated by P3(Pc) and P4(Pd) on the registered map 740. Therefore, the real-time image 710 is registered in the target map 500 with respect to matching landmarks P1 to P4 (corresponding to Pa to Pd in ​​image 710).

[0111] As described above, the registration of real-time images taken during endoscopic procedures into a target map can be used in a variety of applications to improve the accuracy and efficiency of endoscopic procedures. For example, image registration can assist the operator in identifying the treatment site in real time from a pre-generated target map with improved accuracy. Since the target map stores information on the position of the endoscope tip relative to multiple stored landmarks, the image registration discussed herein can be useful in locating and tracking the endoscope tip in real time throughout the procedure. In the case of a target map that stores information on the target type (e.g., normal or abnormal tissue) at various aiming beam positions of the target, the endoscopic tracking device 330 may detect and track changes in tissue type over time at various target sites, or provide an evaluation of the effectiveness of the treatment delivered to the target.

[0112] Figure 8 is a flowchart illustrating Method 800 for endoscopic mapping of targets within a subject's body during treatment. Method 800 can be implemented and performed by a medical system for use in endoscopic procedures, such as System 100 or a variation thereof. Although the processes of Method 800 are depicted in one flowchart, they do not need to be performed in a specific order. In various examples, some of the processes may be performed in an order different from the order shown herein.

[0113] In 810, the aiming beam is emitted from the tip of the endoscope and directed towards a part of the target, such as a portion of the target 101. The aiming beam may be generated by a laser source, such as a second laser energy source 204. Alternatively, the aiming beam may be generated by another light source and transmitted via an optical fiber or the like.

[0114] In 820, an image of the target site can be captured by an imaging system such as the imaging system 115. The image can be taken when the lens system 118 of the imaging system 115 is positioned at the endoscope location. An image can be taken when the target is illuminated with electromagnetic radiation (also called illumination light) within the optical range from UV to IR. The illumination light can be generated by a light source such as the light source 104 and transmitted to the target site via the illumination guide 120. In one example, the light source 104 may include two or more light sources that emit light with different illumination characteristics.

[0115] The aiming beam directed at the target site is contained within the imaging system's field of view (FOV), and therefore, the image captured at the target site may include not only a graphic representation of the illuminated target (e.g., the surface of the target's anatomical structure) but also the footprint of the aiming beam. The image may be displayed to the user on a display 108 or the like, as shown in any of Figures 4A to 4F. The aiming beam footprint may be colored using a different color from the background of the endoscopic image. In one example, the location of the aiming beam footprint may be identified from the endoscopic image by matching the color of the aiming beam to the color of a pixel in the endoscopic image.

[0116] The signal reflected from the target in response to the illumination light can be analyzed, for example, using a spectrometer 208, to identify the type of target at the target's aiming beam position. In one example, based on the spectroscopic characteristics of the reflected signal, the tissue at the target site can be identified as normal and abnormal tissue, or mucous membrane or muscle tissue, among other anatomical structural types. In some examples, the target site can be identified as one of several calculus types, each with its own composition.

[0117] In endoscopic images, the targeting beam footprint may be marked with a visual identifier indicating the tissue type at the targeting beam position. In one example, the targeting beam footprint may be colored with different colors to indicate different tissue types. For instance, the targeting beam footprint may be colored green if the target site is recognized as normal tissue, or red if the target site is recognized as abnormal tissue (e.g., cancer). In some examples, the targeting beam footprint may be marked with identifiers to indicate changes in tissue type over time or to indicate the treatment status.

[0118] In 830, one or more landmarks are identified from the captured image of the target site, and the position of the landmarks relative to the aiming beam footprint can be determined, for example, using the video processor 320. In one example, the landmarks represent anatomical structures (e.g., blood vessels). Landmarks can be detected based on changes in the brightness of pixels in the endoscopic image. In one example, landmarks can be detected using edge detection constrained by a contrast threshold, as well as the number of pixels between similar positive and negative contrast gradients.

[0119] The location of one or more landmarks, such as the x and y distances in the coordinate system of an endoscopic image, can be determined relative to the aiming beam footprint in the same endoscopic image. In another example, landmark localization includes determining the distances between landmarks in the coordinate system of the endoscopic image. In some examples, a subset of detected landmarks may be selected based on the spatial distribution of landmarks in the endoscopic image. For example, the selected subset may include landmarks distributed throughout the entire endoscopic image (as opposed to clusters of closely spaced landmarks in one region of the image). In yet another example, landmarks may be selected based on whether laser energy is activated on a landmark, since laser energy can distort such landmarks. For example, a landmark not activated by laser energy may be favorably selected over another landmark that is activated by laser energy.

[0120] In some cases, targets can be illuminated under special lighting conditions to improve landmark detection and localization. For example, targets may be illuminated with blue or green lighting to enhance the contrast of the endoscopic image of the target and to more clearly define vascular systems that are less likely to move or change over time. This allows for more consistent landmark detection and localization under slightly different lighting conditions.

[0121] In 840, the target map can be reconstructed by integrating multiple images of various parts of the target based on the respective landmarks identified from multiple images. For example, when the tip of the endoscope pans across the target, either manually by the surgeon or automatically by the endoscopic actuator, the distal end of the endoscope moves, and various endoscopic positions {L1, L2, ..., L N The imaging system is positioned at each endoscope position, and the imaging system captures multiple target areas {S1, S2, ..., S} that fall within the imaging system's FOV. N In}, endoscopic images {G1, G2, ..., GN A sequence of} can be captured. Examples of sequences of images (or video frames) captured at different parts of the target are shown in Figures 4A to 4F.

[0122] Multiple endoscopic images {G1, G2, ..., G N The target map can be reconstructed by integrating the endoscopic images {G1, G2, ..., G N The} can be aligned with respect to landmarks identified from the image. In one example, two endoscopic images G i and G j Between, Image G j Image G matching two or more landmarks identified from i Matching landmarks can be identified, including two or more landmarks identified from image G. i and G j It can be aligned with respect to identified matching landmarks.

[0123] In some cases, endoscopic images (for example, G i ) is another endoscopic image (for example, G j The image may be transformed before alignment. Transformations can correct geometric image distortions or deformations, such as image scaling caused by moving the distal end of the endoscope closer to or further away from the target, body movement (e.g., breathing), image rotation caused by changes in the line of sight from the imaging system to the target, or distortion of length, shape, and other image properties caused by changes in the orientation of the endoscope. Examples of image transformations may include one or more of the scaling, translation, rotation, or shear transformations of the image in a coordinate system, among other rigid, similarity-based, or affine transformations.

[0124] In one example, the first endoscopic image G i and the second endoscopic image G j Before aligning them, image G j This is image G iThe distance between two of the matching landmarks in [reference], and the image G j can be scaled by a magnification factor λ based on the ratio of the distance between two corresponding landmarks in the image G i and the image G j In another example, the magnification factor λ can be determined based on the ratio of geometric features generated from the aiming beam footprint in the image G

[0125] In one example, before aligning the first endoscopic image G i with the second endoscopic image G j changes in the orientation of the endoscope between the images G i and G j can be detected and corrected. In one example, the change in the orientation of the endoscope can be determined based on a comparison of a first gradient between two of the matching landmarks in the image G i and a second gradient between two matching landmarks in the image G j In another example, the change in the orientation of the endoscope can be determined based on a comparison between a first geometric feature of the aiming beam footprint in the image G i and a second geometric feature of the aiming beam footprint in the image G j In one example, at least one of the first or second aiming beam footprints has an elliptical shape with a major axis and a minor axis, and at least one of the first or second geometric features can include the ratio of the length of the major axis to the length of the minor axis of the elliptical aiming beam footprint, as described above with reference to FIGS. 6A-6F.

[0126] The converted images can be aligned and integrated into a reconstructed target map, as shown in Figure 5. The reconstructed map may include one or more of the following: multiple endoscopic images, aiming beam footprints generated during the tissue painting process, target type identifiers (such as color-coded aiming beam footprints), the position of landmarks relative to the aiming beam footprints, or a set of landmarks identified from the relative positions between landmarks. The target map, including information on landmarks and aiming beam footprints, can be stored in memory 340. In 850, the target map can be used to locate and track the tip of the endoscope during endoscopic procedures, as described later with reference to Figure 9. Additionally or alternatively, the target map may be used to determine changes in tissue state at the target site (e.g., changes from normal to abnormal or vice versa).

[0127] The image reconstruction and endoscopic tracking systems and methods discussed herein, in various examples, can accommodate tolerances for their measurements. Comparisons or formulas described or inferred from the description herein are not limited to perfect equivalence. In one example, the systems and methods described herein can first compare values ​​for perfect equality, then gradually increase the tolerance around each calculation to identify overlaps that are then treated as equal. For example, when comparing anatomical maps of the same patient from two different points in time, which may differ by week, month, or year, the tolerance range can also be gradually increased from ideal to limit, or a larger area can be created around each landmark, and then overlaps are checked using standard statistical methods such as a median t-test or Mann-Whitney comparison with a target confidence level of 5–25%. In one example, the tolerance range or confidence interval may be a range from 0% to X% of the distance between each pair of landmarks being compared, where X% may be 20% in one example and 25% in another. Alternatively, the tolerance or confidence interval can be gradually increased until a satisfactory match is obtained among the majority of landmarks, such as when approximately 70-100% of the landmarks are found to match.

[0128] Figure 9 is a flowchart showing an example of a method 900 for endoscopic tracking using a reconstructed target map, such as one generated using method 800. In 910, during endoscopic procedure, a real-time image of the treatment site of the target site may be captured via an imaging system positioned at an unknown endoscopic location. In 920, matching landmarks may be identified, including two or more landmarks in the target map that correspond to two or more landmarks in the real-time image. Matching landmarks may be identified based on the distance ratio between landmarks, as described above with reference to Figure 7. In 930, the real-time image may be registered in the target map using the identified matching landmarks. Registration may include image transformations (e.g., translation, scaling, rotation, etc.) and image alignment, as described above with reference to Figure 7. In 940, the tip of the endoscope may be localized and tracked based on the registration of the real-time image. Since the target map stores information about the position of the tip of the endoscope with respect to multiple landmarks, the endoscopic tracking device 330 can localize and track the tip of the endoscope in real time throughout the procedure based on the landmarks on the registered image. With respect to a target map that stores information about tissue types (for example, normal or abnormal tissue at different target sites, as indicated by the footprint of the targeting beam), the endoscopic tracking device 330 can detect and track changes in tissue type over time at various sites of the target, or the effectiveness of the treatment delivered thereto.

[0129] Figure 10 generally shows a block diagram of an exemplary machine 1000 capable of performing any one or more of the techniques (e.g., methodologies) discussed herein. Parts of this description may be applied to the computing framework of various parts of system 100, such as the endoscope controller 103.

[0130] In alternative embodiments, machine 1000 may operate as a standalone device or may be connected to other machines (e.g., networked). In a networked deployment, machine 1000 may operate as a server machine, a client machine, or both in a server-client network environment. For example, machine 1000 may function as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 1000 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be performed by that machine. Furthermore, although only one machine is shown, the term “machine” shall be interpreted as including any set of machines that individually or collectively execute a set (or set of) instructions that perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

[0131] As described in this book, examples may include or be operated by logic or many components or mechanisms. A circuit set is a collection of circuits implemented on a tangible entity that includes hardware (e.g., simple circuits, gates, logic, etc.). The membership of a set of circuits can be flexible over time and in response to variations in the underlying hardware. A circuit set includes members that can perform specified operations, either individually or in combination. In one example, the hardware of a circuit set may be poorly designed to perform a particular operation (e.g., hardwired). In one example, the hardware of a circuit set may include various connected physical components (e.g., execution units, transistors, simple circuits, etc.) that include a physically modified computer-readable medium (e.g., a magnetically, electrically, or movable arrangement of invariant mass particles) to encode a particular operation instruction. When connecting physical components, the underlying electrical properties of the hardware components are changed from insulator to conductor, or vice versa. These instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create members of a circuit configured in the hardware via variable connections to perform a specific part of an operation during operation. Thus, the computer-readable medium is communicatively coupled to other components of the circuit set members when the device is operating. In one example, any one physical component may be used in multiple members of multiple circuit sets. For example, during operation, an execution unit may be used in a first circuit of a first circuit set at one point in time, and then reused by a second circuit in the first circuit set, or by a third circuit in the second circuit set at a different time.

[0132] The machine (e.g., a computer system) 1000 may include a hardware processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 1004, and static memory 1006, some or all of which may communicate with each other via an interlink (e.g., a bus) 1008. The machine 1000 may further include a display unit 1010 (e.g., a raster display, a vector display, a holographic display, etc.), an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In one example, the display unit 1010, the input device 1012, and the UI navigation device 1014 may be touchscreen displays. Machine 1000 may further include a storage device (e.g., a drive unit) 1016, a signal generating device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1021 such as a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 1000 may also include an output controller 1028 for communicating with or controlling one or more peripheral devices (e.g., a printer, a card reader, etc.) via a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near-field communication (NFC)) connection.

[0133] The storage device 1016 may include a machine-readable medium 1022 in which one or more sets of data structures or instructions 1024 (e.g., software) that embody or utilize any one or more of the techniques or functions described herein are stored. The instructions 1024 may also reside entirely or at least partially in the main memory 1004, the static memory 1006, or the hardware processor 1002 while being executed by the machine 1000. In one example, one or any combination of the hardware processor 1002, the main memory 1004, the static memory 1006, or the storage device 1016 may constitute the machine-readable medium.

[0134] Although the machine-readable medium 1022 is shown as a single medium, the term “machine-readable medium” may include a single or multiple mediums configured to store one or more instructions 1024 (for example, a centralized or distributed database, and / or associated caches and servers).

[0135] The term “machine-readable medium” may include any medium capable of storing, encoding, or carrying instructions for execution by machine 1000, causing machine 1000 to perform any one or more of the techniques of the Disclosure, or storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable mediums may include solid-state memory, as well as optical and magnetic media. In one example, a machine-readable medium with mass comprises a number of particles having immutable (e.g., stationary) mass. Thus, a machine-readable medium with mass is not a transient propagating signal. Specific examples of machine-readable mediums with mass may include non-volatile memory such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPSOM)) and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0136] Instruction 1024 may be further transmitted or received via the communication network 1026 using a transmission medium via the network interface device 1020, utilizing any one of numerous transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Illustrative communication networks may include, among others, local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), public telephone (POTS) networks, and wireless data networks (e.g., the IEEE 802.11 series of standards known as WiFi®, the IEEE 802.16 series of standards known as WiMax®), the IEEE 802.15.4 series of standards, and peer-to-peer (P2P) networks. For example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to the communication network 1026. For example, the network interface device 1020 may include multiple antennas for wireless communication using at least one of the single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” is interpreted to include any intangible medium capable of storing, encoding, or carrying instructions for execution by machine 1000, and includes digital or analog communication signals or other intangible mediums for facilitating the communication of such software.

[0137] Supplementary note The above detailed description includes references to the accompanying drawings, which form part of the detailed description. The drawings illustrate specific embodiments in which the present invention can be carried out. These embodiments are also referred to herein as “examples.” Such examples may include elements in addition to those illustrated or described. However, the inventors also intend examples in which only the illustrated or described elements are provided. Furthermore, the inventors also intend examples in which any combination or permutation of the illustrated or described elements (or one or more embodiments thereof) is used in reference to a particular example (or one or more embodiments thereof) or in reference to other examples (or one or more embodiments thereof) illustrated or described herein.

[0138] In this specification, the terms “a” or “an” are used to include one or more, regardless of any other examples or uses of “at least one” or “one or more,” as is common in patent documents. In this specification, the term “or” is used to refer to non-exclusive “or,” such that “A or B” includes “A but not B,” “B but not A,” and “A and B.” In this specification, the terms “including” and “in which” are used as plain English equivalents of the terms “comprising” and “wherein,” respectively. Furthermore, in the following claims, the terms “including” and “comprising” are unrestricted, meaning that any system, device, article, composition, formulation, or process that includes elements in addition to those listed after such terms in the claim is still considered to be within the scope of that claim. Moreover, in the following claims, terms such as “first,” “second,” “third,” etc., are used merely as labels and are not intended to impose numerical requirements on their subjects.

[0139] The above description is intended to be illustrative and not limiting. For example, the above examples (or one or more of their embodiments) may be used in combination with one another. Other embodiments may be used by those skilled in the art who consider the above description. The abstract is provided so that readers may quickly confirm the nature of the technical disclosure. The abstract is submitted with the understanding that it is not to be used to interpret or limit the scope or meaning of the claims. Also, in the embodiments for carrying out the above invention, various features may be grouped together to simplify the disclosure. This should not be interpreted as meaning that any disclosed features not claimed are essential to the claims. Rather, the subject matter of the invention may lie in fewer features than all the features of a particular disclosed embodiment. Accordingly, the following claims are incorporated into the embodiments for carrying out the invention as examples or embodiments, and each claim stands independently as a separate embodiment, and such embodiments are intended to be able to be combined with one another in various combinations or permutations. The scope of the invention should be determined by referring to the appended claims, together with the entire scope of equivalents to which such claims are granted. [Explanation of symbols]

[0140] 100 Healthcare Systems 101 Targets 102 Endoscope 102a Therapeutic Tool Channel 102b Distal end opening 103 Endoscope Controller 104 Light source 106 Laser device 107 Operating Unit 107a Curved knob 107b Insertion port, treatment tool insertion port 107c Switch 108 displays 108a video cable 110 Insertion part 110a Distal end 110b Curved section 110c Flexible tube section 112 Optical Path 112a Distal end 114 Universal Code 114a connector 114b connector 115 Imaging System 116 Image Sensors 118 Lens System 120 Lighting Guide 122 Illumination Lens 124 signal line 202 First energy source 204 Second energy source, second laser energy source 205 Splitter 206 Controller 208 Spectrometer 230 Illumination Light 240a therapeutic beam 240b Aiming Beam 310 Device Controllers 320 Video Processors 321 Landmark Detector 322 Target Identifier 323 Target Map Generator 330 Endoscopic tracking device 340 memory 410-460 Endoscopic images 410 images 412 Aiming Beam Footprint 413 images 414 images 415A Landmark 415B Landmark 415C Landmark 420 Endoscopic Images 422 Aiming Beam Footprint 430 images 432 Aiming Beam Footprint 435A~435B New Landmark 440 images 442 Aiming Beam Footprint 445A~445C New Landmark 450 images 452 Aiming Beam Footprint 455A~455B New Landmark 460 images 462 Aiming Beam Footprint 465A~465B New Landmark 500 Target Map 611 Target site 612 Aiming Beam Footprint 615 Endoscopic images 621 Target site 622 Aiming beam footprint, elliptical footprint 625 Endoscopic Images 701 Aiming Beam Footprint 710 Real-time images 711 Intersection Landmark 720 Maps Figure 730 740 Maps 800 ways 1000 machines 1002 Hardware Processor 1004 Main Memory 1006 Static Memory 1008 Interlink 1010 Display Unit 1012 Alphanumeric input device 1014 User Interface (UI) Navigation Devices 1016 Storage Devices 1018 Signal Generating Devices 1020 Network Interface Device 1021 Sensor 1022 Machine-readable media 1024 instructions 1026 Communication Network 1028 Output Controller

Claims

1. A system for endoscopic mapping of targets, An imaging system configured to generate an endoscopic image of the target, Multiple landmarks are identified from the aforementioned endoscopic images, Determine the spatial relationships between the multiple landmarks identified in the endoscopic image of the target. A target map is generated by integrating multiple endoscopic images based on landmarks identified from the spatial relationships between multiple endoscopic images and the determined landmarks, wherein the multiple endoscopic images include images of various parts of the target. A video processor configured to perform the following: A system equipped with these features.

2. Identifying matching landmarks in the target map that match corresponding landmarks in the real-time image of the target generated by the imaging system during endoscopic procedure, Registering the real-time image to the target map using the matching landmark, Based on the registration of the real-time images, the position of the tip of the endoscope is tracked, The system further comprises an endoscopic tracking system configured to perform the following: The system according to claim 1.

3. The system according to claim 2, wherein the endoscopic tracking system is configured to identify matching landmarks based on one or more ratios of distances between landmarks in the real-time image and one or more ratios of distances between landmarks in the target map.

4. The system according to claim 2, wherein the endoscopic tracking system is configured to generate a display of changes in tissue type at the target site.

5. The system according to claim 1, wherein the video processor is further configured to determine the position of each of the identified landmarks relative to the footprint of a targeting beam emitted from a laser source or light source, directed toward the target, and captured by the imaging system.

6. The video processor is equipped with a spectrometer that is communicatively coupled to it, and the spectrometer is configured to measure one or more spectral characteristics of the illumination light signal reflected from the target, The video processor is configured to identify a tissue type at the location of the footprint of a targeting beam emitted from a laser source or light source based on one or more spectral characteristics, and to mark the footprint of the targeting beam using a visual identifier indicating the identified tissue type. The system according to claim 1.

7. The aforementioned video processor From the landmarks identified from one or more of the aforementioned endoscopic images, a subset of landmarks is selected based on whether laser energy is activated at each target site where the identified landmarks are located. The target map is generated by integrating the multiple endoscopic images based on the selected subset of landmarks. The system according to claim 1, configured to perform the following:

8. The plurality of endoscopic images include a first endoscopic image of a first target area and a second endoscopic image of a second target area, and the video processor includes images of various parts of the target, Identifying matching landmarks that include two or more landmarks in the first endoscopic image that match two or more corresponding landmarks in the second endoscopic image, Aligning the first and second endoscopic images with respect to the matching landmark in the coordinate system of the first endoscopic image, The target map is generated using at least the aligned first and second images. The system according to claim 1, configured to perform the following:

9. The aforementioned video processor Transforming the second image, which involves one or more of scaling, translation, or rotation of the second image, Aligning the converted second image and the first image with respect to the matching landmark. The system according to claim 8, configured to perform the following:

10. The system according to claim 9, wherein the video processor is configured to transform the second image in order to compensate for a change in the orientation of the endoscope between the first and second images, and the orientation of the endoscope indicates the inclination of the tip of the endoscope relative to the target site.

11. The spatial relationship between the landmarks is, The distance between two landmarks represented in the coordinate system of an endoscopic image, or Gradient between two landmarks represented in the coordinate system of the endoscopic image. The system according to claim 1, comprising at least one of the above.

12. A non-temporary machine-readable storage medium, which, when executed by one or more processors of a machine, includes instructions causing the machine to perform an operation, wherein the operation is Aiming the targeting beam at the target and To generate an endoscopic image of the aforementioned target, Identifying multiple landmarks from the generated endoscopic image, The endoscopic image of the target determines the spatial relationships between the multiple landmarks, A target map is generated by integrating multiple endoscopic images based on landmarks identified from the spatial relationships between multiple endoscopic images and the determined landmarks, wherein the multiple endoscopic images include images of various parts of the target. Equipped with, At least one non-transient machine-readable storage medium.

13. The command is given to the machine. To capture real-time images of the target during endoscopic procedures, Identifying a matching landmark in the target map that matches a corresponding landmark in the real-time image of the target, Registering the real-time image to the target map using the matching landmark, Based on the registration of the real-time images, the position of the tip of the endoscope is tracked, To perform an action that further includes the following: The at least one non-transient machine-readable storage medium according to claim 12.

14. The operation for identifying matching landmarks is based on one or more ratios of distances between landmarks in the real-time image and one or more ratios of distances between landmarks in the target map, according to claim 13, for at least one non-temporary machine-readable storage medium.

15. The instruction causes the machine to perform an operation which further includes generating a display of changes in tissue type at a target site, according to claim 14, at least one non-temporary machine-readable storage medium.

16. The aforementioned instruction is given to the machine, The measurement of one or more spectral characteristics of the illumination light signal reflected from the target, Based on the one or more spectral characteristics, identify the tissue type at the location of the targeting beam footprint, Marking the footprint of the aiming beam using a visual identifier indicating the identified tissue type and A non-transient machine-readable storage medium according to claim 12, which enables the execution of an operation further comprising the above.

17. The aforementioned instruction is given to the machine, From the landmarks identified from one or more of the aforementioned endoscopic images, a subset of landmarks is selected based on whether laser energy is activated at each target site where the identified landmarks are located. The target map is generated by integrating the multiple endoscopic images based on the selected subset of landmarks. A non-transient machine-readable storage medium according to claim 12, which enables the execution of an operation further comprising the above.

18. The images of various parts of the target include a first endoscopic image of a first target area and a second endoscopic image of a second target area, and the command is given to the machine, Identifying matching landmarks that include two or more landmarks in the first endoscopic image that match two or more corresponding landmarks in the second endoscopic image, Aligning the first and second endoscopic images with respect to the matching landmark in the coordinate system of the first endoscopic image, At least the first and second images aligned to generate the target map, To perform an action that further enhances the functionality The at least one non-transient machine-readable storage medium according to claim 12.

19. The operation of aligning the first and second endoscopic images is, Transforming the second image, which involves one or more of scaling, translation, or rotation of the second image, Aligning the converted second image and the first image with respect to the matching landmark. The at least one non-temporary machine-readable storage medium according to claim 18, including

20. The spatial relationship between the landmarks is, The distance between two landmarks represented in the coordinate system of an endoscopic image, or Gradient between two landmarks represented in the coordinate system of the endoscopic image. It includes at least one of the following: The operation for transforming the second image includes correcting for changes in the orientation of the endoscope, such that the orientation of the endoscope indicates the inclination of the tip of the endoscope relative to the target site. The at least one non-transient machine-readable storage medium according to claim 19.

Images