System for endoscopic mapping of target and non-transitory machine-readable storage medium

Abstract

The invention relates to a system for endoscopic mapping of a target and a non-transitory machine-readable storage medium. The system comprises: an imaging system configured to generate an endoscopic image of the target; and a video processor configured to identify one or more landmarks from each of the endoscopic images and determine respective positions of the identified one or more landmarks relative to an optical reference, the optical reference is generated by a light source and displayed on a corresponding one of the endoscopic images of the target; and generating a target map by integrating the endoscopic images based on one or more landmarks identified from each of the endoscopic images, where the endoscopic images include images of respective portions of the target.

Description

[0001] Cross-references to related applications

[0002] This application is a divisional application of patent application No. 202180064896.8 entitled "Image Reconstruction and Endoscopic Tracking", filed on July 16, 2021, with international application number PCT / US2021 / 042053 and entered the Chinese national phase on March 22, 2023. Technical Field

[0003] This article generally relates to endoscopy, and more specifically to systems and methods for endoscopic mapping of targets and tracking of the position of the endoscope during surgery. Background Technology

[0004] Endoscopes are typically used to access a subject's internal location, providing doctors with visual access. An endoscope is usually inserted into the patient's body, directing light towards the target being examined (e.g., a target anatomical structure or object) and collecting the light reflected from that object. The reflected light carries information about the object being examined and can be used to create an image of the object. Some endoscopes include a working channel through which the operator can perform suction, or pass instruments such as brushes, biopsy needles, or forceps, or perform minimally invasive surgery to remove unwanted tissue or foreign bodies from the patient's body.

[0005] Some endoscopes include, or can be used with, laser or plasma systems to deliver surgical laser energy to target anatomical structures or objects, such as soft or hard tissue. Examples of laser treatments include ablation, coagulation, vaporization, and fragmentation. In lithotripsy applications, lasers are used to break down stone structures in areas where stones form, such as the kidneys, gallbladder, and ureters, or to ablate large stones into smaller fragments.

[0006] Video systems have been used to assist physicians or technicians in visualizing the surgical site and guiding the endoscope during endoscopic procedures. Image-guided endoscopy typically requires locating the endoscope in a coordinate system of the target area and tracking its movement. Accurate mapping of the target area and effective endoscope positioning and tracking can improve the precision of endoscopic manipulation during endoscopic procedures, enhance the interventional capabilities of physicians or technicians, and improve the effectiveness of treatments such as laser therapy. Summary of the Invention

[0007] One aspect of the present invention provides a system for endoscopic mapping of a target, the system comprising: an imaging system configured to generate an endoscopic image of the target; and

[0008] A video processor configured to: identify one or more landmarks from each of the endoscopic images and determine the respective positions of the identified one or more landmarks relative to an optical reference generated by a light source and displayed on a corresponding endoscopic image of the target; and generate a target map by integrating the endoscopic images based on one or more landmarks identified from each of the endoscopic images, wherein the endoscopic images include images of various parts of the target.

[0009] One aspect of the present invention provides at least one non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including: directing a light signal generated by a light source toward a target; generating an endoscopic image of the target; identifying one or more landmarks from each of the endoscopic images and determining the respective positions of the identified one or more landmarks relative to an optical reference generated by the optical signal and displayed on a corresponding endoscopic image of the target; and generating a target map by integrating the endoscopic images based on the one or more landmarks identified from each of the endoscopic images, the endoscopic images including images of various portions of the target.

[0010] This document describes systems, apparatus, and methods for endoscopic mapping of a target and for tracking the position of an endoscope within a subject during surgery. Exemplary systems include: an imaging system configured to capture endoscopic images of a target, the endoscopic images including a targeting beam footprint pointing towards the target; and a video processor configured to: identify one or more landmarks from the captured endoscopic images and determine their respective positions relative to the targeting beam footprint; and generate a target map by integrating multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images. The target map can be used to track the position of the endoscope during endoscopic surgery.

[0011] Example 1 is a system for endoscopic mapping of a target. The system includes: an imaging system configured to capture endoscopic images of the target, the endoscopic images including a targeting beam footprint pointing towards the target; and a video processor configured to: identify one or more landmarks from the captured endoscopic images and determine their respective positions relative to the targeting beam footprint; and generate a target map by integrating multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images.

[0012] In Example 2, the subject of Example 1 may optionally include a video processor that can be configured to identify the tissue type at the location of the aiming beam and to mark the aiming beam footprint using a visual identifier that indicates the identified tissue type.

[0013] In Example 3, any one or more of the subjects in Examples 1-2 may optionally include a spectrometer communicatively coupled to a video processor, the spectrometer being configured to measure one or more spectral characteristics of an illumination light signal reflected from a target; wherein the video processor is configured to identify the tissue type at the location of the aiming beam based on one or more spectral characteristics, and to mark the aiming beam footprint using a visual identifier indicating the identified tissue type.

[0014] In Example 4, any one or more of the topics in Examples 2-3 may optionally include a video processor that can be configured to identify an organization type as a normal organization or an abnormal organization.

[0015] In Example 5, any one or more of the subjects in Examples 2-4 may optionally include a video processor that can be configured to aim the beam footprint with different color markers to indicate different tissue types.

[0016] In Example 6, any one or more of the subjects in Examples 1-5 may optionally include a video processor that can be configured to identify one or more landmarks from an endoscope image based on changes in the brightness of pixels in the endoscope image.

[0017] In Example 7, the subject of Example 6 may optionally include one or more landmarks represented as line segments or intersecting line segments in the endoscopic image.

[0018] In Example 8, any one or more of the subjects in Examples 1-7 may optionally include a video processor that can be configured to: select a subset of landmarks identified from one or more of a plurality of endoscopic images based on whether laser energy is activated at each target location where the identified landmarks are located; and generate a target map by integrating the plurality of endoscopic images based on the selected subset of landmarks.

[0019] In Example 9, the subject of any one or more of Examples 1-8 may optionally include multiple endoscopic images, which may include images of various parts of the target, including a first endoscopic image of a first target part captured from a first endoscopic position and a second endoscopic image of a second target part captured from a second endoscopic position, wherein the video processor is configured to: identify matching landmarks, which include two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; align the first and second endoscopic images with respect to the matching landmarks in the coordinate system of the first images; and generate a target map using at least the aligned first and second images.

[0020] In Example 10, the subject of Example 9 may optionally include a video processor that can be configured to: transform a second image, including scaling, translating, or rotating the second image; and align the transformed second image and the first image with respect to a matching landmark.

[0021] In Example 11, the subject of Example 10 may optionally include a transformation of the second image, which may include matrix multiplication performed by the transformation matrix.

[0022] In Example 12, any one or more of the topics in Examples 10-11 may optionally include a video processor that can be configured to scale the second image using a scaling factor based on the ratio of the distance between two matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0023] In Example 13, any one or more of the subjects in Examples 10-12 may optionally include a video processor that can be configured to scale the second image by a scaling factor based on the ratio of the geometry of the aiming beam footprint in the first image to the geometry of the aiming beam footprint in the second image.

[0024] In Example 14, any one or more of the subjects in Examples 10-13 may optionally include a video processor that can be configured to transform a second image to correct for a change in endoscope orientation between the first and second images, the endoscope orientation indicating the tilt of the endoscope tip relative to the target site.

[0025] In Example 15, the subject of Example 14 may optionally include a video processor that can be configured to detect changes in endoscope orientation using a first slope between two matching landmarks in a first image and a second slope between corresponding two landmarks in a second image.

[0026] In Example 16, any one or more of the subjects in Examples 14-15 may optionally include a video processor that can be configured to detect changes in endoscope orientation using a first geometric feature of the aiming beam footprint in a first image and a second geometric feature of the aiming beam footprint in a second image.

[0027] In Example 17, the subject of Example 16 may optionally include at least one of a first aiming beam footprint or a second aiming beam footprint, which may have an elliptical shape having a major axis and a minor axis, and at least one of the first geometric features or the second geometric feature may include the ratio of the length of the major axis to the length of the minor axis.

[0028] In Example 18, any one or more of the subjects in Examples 1-17 may optionally include an endoscope tracking system configured to: identify matching landmarks from real-time images of the surgical site of a target captured by an imaging system from an unknown endoscopic location during endoscopic surgery, the matching landmarks including two or more corresponding landmarks in the target image that match two or more landmarks in the real-time image; register the real-time image to the target image using the matching landmarks; and track the endoscope tip position based on the registration of the real-time image.

[0029] In Example 19, the subject of Example 18 may optionally include an endoscope tracking system that can be configured to identify matching landmarks based on one or more ratios of the distances between landmarks in a real-time image and one or more ratios of the distances between landmarks in a target image.

[0030] In Example 20, any one or more of the subjects in Examples 18-19 may optionally include an endoscopic tracking system that can be configured to generate an indication of changes in tissue type at a target site.

[0031] Example 21 is a method for endoscopic mapping of a target. The method includes: pointing a targeting beam at the target; capturing an endoscopic image of the target via an imaging system, the endoscopic image including the footprint of the targeting beam; identifying one or more landmarks from the captured endoscopic image via a video processor, and determining the corresponding positions of the one or more landmarks relative to the footprint of the targeting beam; and generating a target map via the video processor by integrating the multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images.

[0032] In Example 22, the subject matter of Example 21 may optionally include: identifying the tissue type at the location of the aiming beam using an illumination light signal reflected from the target; and marking the aiming beam footprint using a visual identifier indicating the identified tissue type.

[0033] In Example 23, any one or more of the topics in Examples 21-22 may optionally include: identifying one or more landmarks from an endoscope image based on changes in the brightness of pixels in the endoscope image.

[0034] In Example 24, the subject matter of any one or more of Examples 21-23 may optionally include: wherein the plurality of endoscopic images include images of various parts of a target, including a first endoscopic image of a first target part captured at a first endoscopic position and a second endoscopic image of a second target part captured at a second endoscopic position, the method comprising: identifying matching landmarks, the matching landmarks including two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; aligning the first endoscopic image and the second endoscopic image with respect to the matching landmarks in a coordinate system of the first image; and generating a target map using at least the aligned first image and the second image.

[0035] In Example 25, the subject matter of Example 24 may optionally include: aligning a first endoscopic image and a second endoscopic image by: transforming the second image, including scaling, translating, or rotating the second image; and aligning the transformed second image and the first image with respect to a matching landmark.

[0036] In Example 26, the subject of Example 25 may optionally include: transforming the second image by scaling the second image by a scaling factor based on the ratio of the distance between two matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0037] In Example 27, any one or more of the subjects in Examples 25-26 may optionally include: transforming the second image by scaling the second image by a scaling factor based on the ratio of the geometry of the aiming beam footprint in the first image to the geometry of the aiming beam footprint in the second image.

[0038] In Example 28, the subject matter of any one or more of Examples 25-27 may optionally include: transforming the second image includes: correcting for changes in endoscope orientation between the first image and the second image, the endoscope orientation indicating the tilt of the endoscope tip relative to the target site.

[0039] In Example 29, the subject of Example 28 may optionally include: using a first slope between two matching landmarks in a first image and a second slope between corresponding two landmarks in a second image to detect changes in endoscope orientation.

[0040] In Example 30, the subject matter of any one or more of Examples 28-29 may optionally include: using a first geometric feature of the aiming beam footprint in the first image and a second geometric feature of the aiming beam footprint in the second image to detect changes in endoscope orientation.

[0041] In Example 31, the subject matter of any one or more of Examples 21-30 may optionally include: during endoscopic surgery, using an imaging system to capture real-time images of the target surgical site from an unknown endoscopic location; identifying matching landmarks, which include two or more corresponding landmarks in the target image that match two or more landmarks in the real-time image; registering the real-time image to the target image using the matching landmarks; and tracking the endoscope tip position based on the registration of the real-time image.

[0042] In Example 32, the subject of Example 31 may optionally include: identifying matching landmarks based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

[0043] Example 33 is at least one non-transitory machine-readable storage medium including instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including: pointing a targeting beam at a target; capturing an endoscopic image of the target, the endoscopic image including a target beam footprint; identifying one or more landmarks from the captured endoscopic image and determining the corresponding position of one or more landmarks relative to the target beam footprint; and generating a target map by integrating multiple endoscopic images based on landmarks identified from one or more of the multiple endoscopic images.

[0044] In Example 34, the subject matter of Example 33 may optionally include: wherein the instructions cause the machine to perform operations that also include: identifying the tissue type at the location of the aiming beam; and marking the aiming beam footprint using a visual identifier that indicates the identified tissue type.

[0045] In Example 35, the subject matter of any one or more of Examples 33-34 may optionally include: wherein the instructions cause the machine to perform an operation that also includes: identifying one or more landmarks from an endoscope image based on changes in the brightness of pixels in the endoscope image.

[0046] In Example 36, the subject matter of any one or more of Examples 33-35 may optionally include: wherein the plurality of endoscopic images include images of various parts of the target, including a first endoscopic image of a first target part captured at a first endoscopic position and a second endoscopic image of a second target part captured at a second endoscopic position, and wherein the instructions cause the machine to perform operations including: identifying matching landmarks, the matching landmarks including two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; aligning the first endoscopic image and the second endoscopic image with respect to the matching landmarks in the coordinate system of the first image; and generating a target map using at least the aligned first image and the second image.

[0047] In Example 37, the subject matter of Example 36 may optionally include: wherein the operation of aligning the first endoscopic image and the second endoscopic image includes: transforming the second image, including scaling, translating, or rotating the second image, or rotating it; and aligning the transformed second image and the first image with respect to a matching landmark.

[0048] In Example 38, the subject of Example 37 may optionally include: wherein the operation of transforming the second image includes: scaling the second image by a scaling factor based on the ratio of the distance between two matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0049] In Example 39, any one or more of the subjects in Examples 37-38 may optionally include: wherein the operation of transforming the second image includes: scaling the second image by a scaling factor based on the ratio of the geometric features of the aiming beam footprint in the first image to the geometric features of the aiming beam footprint in the second image.

[0050] In Example 40, the subject matter of any one or more of Examples 37-39 may optionally include: wherein the operation of transforming the second image includes: correcting for a change in endoscope orientation between the first image and the second image, the endoscope orientation indicating the tilt of the endoscope tip relative to the target site.

[0051] In Example 41, the subject of Example 40 may optionally include: wherein the instructions cause the machine to perform an operation that also includes: detecting a change in endoscope orientation using a first slope between two matching landmarks in a first image and a second slope between corresponding two landmarks in a second image.

[0052] In Example 42, the subject matter of any one or more of Examples 40-41 may optionally include: wherein the instructions cause the machine to perform an operation that also includes: using a first geometric feature of the aiming beam footprint in the first image and a second geometric feature of the aiming beam footprint in the second image to detect a change in endoscope orientation.

[0053] In Example 43, the subject matter of any one or more of Examples 33-42 may optionally include: wherein the instructions cause the machine to perform operations that also include: during endoscopic surgery, using an imaging system to capture a real-time image of a target surgical site from an unknown endoscopic location; identifying matching landmarks, which include two or more corresponding landmarks in the target image that match two or more landmarks in the real-time image; registering the real-time image to the target image using the matching landmarks; and tracking the endoscope tip position based on the registration of the real-time image.

[0054] In Example 44, any one or more of the topics in Examples 33-43 may optionally include: wherein the operation of identifying matching landmarks is based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

[0055] This invention disclosure is an overview of some teachings of this application and is not intended to be exclusive or exhaustive of the subject matter. Further details regarding the subject matter can be found in the detailed description and the appended claims. Other aspects of this disclosure will be apparent to those skilled in the art after reading and understanding the following detailed description and viewing the accompanying drawings, which form a part thereof, and each of these should not be construed in a limiting sense. The scope of this disclosure is defined by the appended claims and their legal equivalents. Attached Figure Description

[0056] Various embodiments are illustrated by way of example in the accompanying drawings. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the subject matter.

[0057] Figure 1 This is a diagram illustrating an example of a medical system used for endoscopic surgery.

[0058] Figure 2 yes Figure 1 A schematic diagram of a portion of the system shown.

[0059] Figure 3 This shows the control used for, such as Figure 1 A block diagram illustrating an example of the endoscope controller for each part of the system shown.

[0060] Figures 4A to 4FThe diagram shows a sequence of endoscopic images or video frames captured at different target sites and landmarks, along with examples of the target beam footprints detected therefrom.

[0061] Figure 5 An example of a target map reconstructed from multiple endoscopic images or video frames captured at different target sites is shown.

[0062] Figures 6A to 6F It is a diagram illustrating the effect of endoscope orientation on the characteristics of endoscopic images and the correction for changes in endoscope orientation from one endoscopic image to another.

[0063] Figure 7 An example is shown of identifying matching landmarks between a real-time image and a reconstructed target map, and of registering the real-time image onto the target map with respect to the matching landmarks.

[0064] Figure 8 This is a flowchart illustrating a method for endoscopic mapping of targets within a subject's body during surgery.

[0065] Figure 9 This is a flowchart illustrating an example of a method for endoscopic tracking using a reconstructed target map.

[0066] Figure 10 This is a block diagram illustrating an example machine to which any or more of the techniques (e.g., methods) discussed herein can be performed. Detailed Implementation

[0067] Minimally invasive endoscopic surgery is a surgical procedure in which a rigid or flexible endoscope is introduced into a target area of ​​the subject's body through a natural opening or a small incision in the skin. Additional surgical instruments, such as laser fibers, can be introduced into the subject's body through similar ports. The endoscope is used to provide the surgeon with visual feedback on the surgical site and instruments.

[0068] Endoscopic surgery can include a preoperative phase and an intraoperative phase. The preoperative phase involves acquiring images or video frames of the target anatomical structure or object using an imaging system (e.g., a video camera device) and reconstructing a map using the images or video frames. This map can be used for diagnostic evaluation or for endoscopic surgical planning. During the intraoperative phase, the endoscope can be introduced into the target surgical site. The operator can move and rotate the distal end of the endoscope and acquire real-time images of the surgical site via an imaging system (such as one located at the distal end of the endoscope). The position and orientation of surgical instruments (e.g., laser fibers) at the distal end of the endoscope can be monitored and tracked throughout the procedure.

[0069] A conventional intraoperative tracking method involves a free-hand technique, whereby the surgeon views the surgical area on a monitor displaying real-time images or video of the area without an automated tracking or navigation system. This approach fails to establish a relationship between images that facilitates tracking the position and orientation of the endoscopic surgical instrument relative to the target. Another approach involves navigation-based tracking systems (e.g., optical or electromagnetic tracking systems) that track the position and orientation of the endoscopic surgical instrument. Image registration procedures can be performed to align the real-time images with the target image. Reference markers visible on the real-time images are used as a reference to guide the surgeon through real-time feedback on the position and orientation of the endoscopic surgical instrument. Reference markers can be external objects attached to the patient or internal anatomical references. External references may lack positional consistency and increase system complexity. Using internal references often restricts the physical movement of the endoscope, such as requiring the endoscope to contact anatomical reference markers during surgery, which can prolong the procedure. Conventional navigation-based endoscopic tracking systems can also suffer from degraded tracking performance in the presence of image distortion or deformation, such as changes in camera position, viewing direction, and orientation relative to the target surface (e.g., skewness or tilt). At least for the reasons described above, the inventors have recognized an unmet need for an improved endoscopic mapping and tracking system that is more robust to image distortion or deformation when used during endoscopic procedures.

[0070] This document describes systems, apparatus, and methods for endoscopic mapping of a target and for tracking the position of an endoscope within a subject during surgery. An exemplary system includes an imaging system and a video processor. The imaging system is configured to capture an endoscopic image of the target, including the footprint of an aiming beam directed at the target. The video processor is configured to identify one or more landmarks from the captured endoscopic image and determine their corresponding positions relative to the aiming beam footprint, and to integrate multiple endoscopic images based on the landmarks identified from one or more of the multiple endoscopic images to generate a target map. This target map can be used to track the position of the endoscope during surgery.

[0071] The systems, apparatus, and methods according to the various embodiments discussed herein can provide improved endoscopic mapping of targets and tracking of the endoscope position during endoscopic surgery. According to various embodiments of this disclosure, various image features can be generated from endoscopic images, including, for example, landmarks and their positions relative to the aiming beam footprint, spatial relationships between landmarks, the shape and geometry of the aiming beam footprint, etc. Image registration, target map reconstruction, and endoscopic tracking during endoscopic surgery based on these image features described herein are more flexible to image rotation, magnification, reduction, changes in camera device position, viewing direction, or endoscope orientation. Improved navigation and endoscopic tracking enhance operator intervention and the precision of endoscopic manipulation, reduce surgical time, and improve overall surgical outcomes, patient safety, and system reliability.

[0072] The topics discussed herein can be applied to a variety of endoscopic applications, including but not limited to arthroscopy, bronchoscopy, colonoscopy, laparoscopy, neuroscopy, and endoscopic cardiac surgery. Examples of endoscopic cardiac surgery include, but are not limited to, endoscopic coronary artery bypass grafting, endoscopic mitral and aortic valve repair and replacement. In this document, "endoscopic" is broadly defined as the characteristic of images acquired by any type of endoscope capable of imaging from within the body. For the purposes of this invention, examples of endoscopes include, but are not limited to, any type of flexible or rigid endoscope (e.g., endoscope, arthroscopy, bronchoscope, cholangioscope, colonoscope, cystoscope, duodenoscope, gastroscopy, hysteroscopy, laparoscope, laryngoscope, neuroscopy, otoscope, colonoscope, nasal laryngoscope, sigmoidoscope, sinusoscope, thoracoscope, etc.) and any device similar to an endoscope equipped with an imaging system (e.g., a nested cannula capable of imaging). Imaging is localized, and surface images can be obtained optically via fiber optics, lenses, or miniaturized (e.g., CCD-based) imaging systems. For the purposes of this invention, examples of fluorescence microscopes include, but are not limited to, X-ray imaging systems.

[0073] Figure 1 This is a diagram illustrating an example of a medical system 100 for endoscopic surgery. System 100 includes an endoscope 102, an endoscope controller 103, a light source 104, a laser device 106, and a display 108. Figure 2 A schematic diagram of a portion of system 100 is shown. Endoscope 102 may include an insertion portion 110 at its distal end and an operating unit 107 at its proximal end. Insertion portion 110 can be configured to be inserted into a target site on a subject, capture an image of the target 101, and optionally perform surgery therein. Insertion portion 110 can be formed using illumination fiber (optical guide), cable, optical fiber, etc. Figure 1In the example shown, the insertion portion 110 includes a distal portion 110a, a bendable portion 110b, and a flexible tube portion 110c disposed on one side of the proximal portion of the bendable portion 110b. The distal portion 110a contains an imaging unit, the bendable portion 110b includes a plurality of bends, and the flexible tube portion 110c is flexible.

[0074] Reference Figure 2 The distal portion 110a may be provided with a light guide 120 configured to couple to a light source 104 and project illumination light 230 onto the target 101 via an illumination lens 122. The distal portion 110a may include an observation unit such as an imaging system 115 configured to image the target 101. The imaging system 115 may include an image sensor 116 and an associated lens system 118. Examples of the image sensor 116 may include a CCD or CMOS imaging device sensitive to ultraviolet (UV), visible (VIS), or infrared (IR) wavelengths. The endoscope 102 may include an insertion port 107b coupled to a treatment tool channel 102a located inside the endoscope 102 and extending along the insertion portion 110. An optical path 112 disposed within a channel 102a via an insertion port 107b has a proximal end operatively connected to a laser device 106 and extends distally from a distal opening 102b of the channel 102a. Laser energy (e.g., a treatment beam 240a or a targeting beam 240b) can be transmitted through the optical path 112 and emitted from its distal end 112a and directed toward a target 101. The endoscope 102 may optionally include an air / water supply nozzle (not shown) at its distal portion 110a.

[0075] The operating unit 107 can be configured to be held by an operator. The operating unit 107 can be located at the proximal portion of the endoscope 102 and is configured to communicate with the endoscope controller 103 and the light source 104 via a flexible universal cable 114 extending from the operating unit 107. Figure 1 As shown, the operating unit 107 includes: a bending knob 107a for bending the bending portion 110b in the vertical and horizontal directions; a treatment tool insertion port 107b through which treatment tools such as medical forceps or optical path 112 are inserted into the body cavity of the subject; and multiple switches 107c for operating peripheral devices such as endoscope controller 103, light source 104, air supply device, water supply device, or gas supply device. Treatment tools such as optical path 112 can be inserted from the treatment tool insertion port 107b and through channel 102a, such that their distal end is at the distal end of the insertion portion 110 from the opening 102b of channel 102a (see...). Figure 2 (Exposed)

[0076] Endoscope controller 103 can control the operation of one or more elements of system 100 (e.g., light source 104, laser device 106, or display 108, which displays an image of target 101 based on an image or video signal sensed by image sensor 116). The distal end of endoscope 102 can be positioned and oriented such that aiming beam 240b is directed at the target location within the field of view (FOV) of imaging system 115; and the endoscopic image includes the footprint of aiming beam 240b. Although aiming beam 240b is shown as a laser beam emitted from a laser energy source, other light sources can also be used to generate an aiming beam propagating along an optical fiber. Endoscope controller 103 can apply image processing to the endoscopic images and reconstruct a map of the target by integrating multiple endoscopic images. In some examples, endoscope controller 103 can use the reconstructed target map to locate and track the endoscope tip during endoscopic surgery. Reference will be made below, for example. Figure 3 Examples of endoscope controller 103 are discussed, including endoscopic mapping of the target and tracking of the endoscope position.

[0077] The universal cable 114 includes optical fibers, electrical cables, etc. The universal cable 114 may branch at its proximal end. One end of the branch is a connector 114a, and the other proximal end of the branch is a connector 114b. Connector 114a can be attached to / removed from the connector of the endoscope controller 103. Connector 114b can be attached to / removed from the light source 104. The universal cable 114 transmits illumination light from the light source 104 to the distal end 110a via connector 114b and light guide 120. Furthermore, the universal cable 114 can transmit signal line 124 (see...) within the cable. Figure 2 The image or video signal captured by the imaging system 115 is transmitted to the endoscope controller 103 via connector 114a. The endoscope controller 103 performs image processing on the image or video signal output from connector 114a and controls at least some of the components constituting the system 100.

[0078] Light source 104 can generate illumination light when endoscope 102 is used in surgery. Light source 104 may include, for example, a xenon lamp, a light-emitting diode (LED), a laser diode (LD), or any combination thereof. In this example, light source 104 may include two or more light sources emitting light with different illumination characteristics (referred to as illumination modes). Under the control of endoscope controller 103, light source 104 emits light, supplying it to endoscope 102 connected via connector 114b of universal cable 114 and a light guide, as illumination light for the interior of the subject. The illumination mode may be a white light illumination mode or a specific light illumination mode such as a narrow-band imaging mode, an autofluorescence imaging mode, or an infrared imaging mode. Specific light illumination can focus and intensify light of specific wavelengths, for example, to better visualize superficial microvessels and mucosal surface structures, enhancing the subtle contrasts of mucosal irregularities.

[0079] Display 108 includes, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Display 108 can display information including endoscopic images of the target, which are processed via video cable 108a by endoscope controller 103. In some examples, one or more endoscopic images may all include the footprint of the aiming beam 240b. An operator can observe and track the behavior of the endoscope within the subject by simultaneously manipulating the endoscope 102 while viewing the images displayed on display 108.

[0080] Laser device 106 is used in, for example, the optical path 112 of a laser fiber. (See reference...) Figure 2 The laser device 106 may include one or more energy sources (e.g., a first energy source 202 and a second energy source 204) for generating laser energy coupled to the proximal end of the optical path 112. In this example, a user can, for example, access the laser device 106 via a button 106a (see [link to example]). Figure 1 The energy source can be selected via a foot switch (not shown), a user interface on software or display 108, or other manual or automatic inputs known in the art.

[0081] The first energy source 202 may be optically coupled to the optical path 112 and configured to deliver a therapeutic beam 240a to the target 101 through the optical path 112. By way of example and not limitation, the first energy source 202 may include a thulium laser for generating laser light to deliver to the target tissue through the optical path 112, to operate in different treatment modes, such as cutting (ablation) mode and coagulation (hemostasis) mode. Other energy sources or any other treatment modes known in the art for treating such tissues may also be used in the first energy source 202, such as Ho:YAG, Nd:YAG, and CO2, as well as others known in the art.

[0082] The second energy source 204 may be optically coupled to the optical path 112 and configured to direct the aiming beam 240b toward the target 101 via the optical path 112. Although the aiming beam 240b is shown as a laser beam emitted from a laser source, other light sources may also be used to generate the aiming beam 240b traveling along the optical fiber. The aiming beam 240b may be emitted while the target is illuminated by the illumination light 230. In some examples, the second energy source 204 may emit at least two different aiming beams, wherein the first aiming beam has at least one characteristic different from the second aiming beam. Such different characteristics may include wavelength, power level, and / or emission pattern. For example, the first aiming beam may have a wavelength in the range of 500 nm to 550 nm, while the second aiming beam may have a wavelength in the range of 635 nm to 690 nm. The characteristics of the different aiming beams may be selected based on the visibility of the aiming beam in an image displayed on the display 108 in certain illumination modes provided by the light source 104 and processed by the endoscope controller 103.

[0083] The laser device 106 may include a controller 206, which includes hardware such as a microprocessor that controls the operation of the first energy source 202 and the second energy source 204. Figure 2 In the example shown, in response to illumination light 230, light reflected from target 101 can enter optical path 112 from distal end 112a. Optical path 112, configured to transmit a laser beam, can also serve as a pathway to transmit reflected light back to laser device 106. Beam splitter 205 can collect the reflected light, separating it from the laser beam delivered to target 101 via the same optical path 112. Laser device 106 may include spectrometer 208 operatively coupled to beam splitter 205 and configured to detect reflected light exiting the beam splitter. Alternatively, the reflected light can be guided via an optical path (e.g., an optical fiber) separate from optical path 112. Spectrometer 208 can be operatively coupled to a dedicated optical path and detect light reflected therefrom.

[0084] Spectrometer 208 can measure one or more spectral properties from the sensed reflectance signal. Examples of spectrometer 208 may include Fourier transform infrared (FTIR) spectrometers, Raman spectrometers, UV-VIS spectrometers, UV-VIS-IR spectrometers, or fluorescence spectrometers. Spectral properties may include properties such as reflectance, reflectance spectrum, and absorption index. Spectral properties may indicate structural category (e.g., anatomical tissue or stone) or a specific structural type indicating the chemical composition of a target.

[0085] Figure 3This is a block diagram illustrating an example of an endoscope controller 103 used in system 100. The endoscope controller 103 includes hardware such as a microprocessor for performing operations according to various examples described herein. The endoscope controller 103 may include a device controller 310, a video processor 320, an endoscope tracker 330, and a memory 340. The device controller 310 can control the operation of one or more components of system 100, such as the endoscope 102, display 108, light source 104, or laser device 106.

[0086] The video processor 320 can receive image or video signals from the imaging system 115 via signal line 124 and process the image or video signals to generate image or video frames that can be displayed on the display 108. In some examples, multiple endoscopic images (e.g., video frames) can be generated and displayed on the display 108. Multiple endoscopic images can be captured at the same location on the target 101 while the endoscope tip and imaging system 115 remain stationary, while the distal end 112a of the optical path 112 (e.g., a laser fiber) can move and guide the laser beam to different locations on the target. In some examples, multiple endoscopic images of the same target location can be captured at different times. Images captured at later times can be registered with previously captured images of the same target location, for example, through image transformation and / or image alignment processes. The registered images can be used to determine changes in tissue state at the target location. Additionally or alternatively, multiple endoscopic images of different locations on the target 101 can be captured, for example, as the distal endoscope tip translates over the target 101. During endoscope translation, the distal end of the endoscope can be moved and positioned at different endoscope positions {L1, L2, ..., LN} (i.e., various positions of the distal end of the endoscope), which can be done manually by the operator or automatically by the endoscope actuator. Under the control of the endoscope controller 103, the imaging system 115 can capture a series of images (or video frames) {G1, G2, ..., GN} at various target sites {S1, S2, ..., GN}, which collectively cover the basic surface area of ​​the target 101. For example, when the lens system 118 is positioned and oriented at the endoscope position Li, an endoscopic image Gi of the target site Si falling within the FOV of the imaging system 115 can be captured. When the distal end of the endoscope moves to a different endoscope position Lj, another endoscopic image Gj of a different target site Sj falling within the FOV of the imaging system 115 can be captured. For example, see reference below. Figures 4A to 4F and Figure 5 In various further examples, the video processor 320 can integrate the obtained images {G1, G2, ..., GN} to create a graph of target 101.

[0087] As described above, the endoscope tip can be positioned and oriented such that the aiming beam 240b falls within the field of view (FOV) of the imaging system 115, and the aiming beam footprint can be captured in the endoscopic image (e.g., Gi). In this example, the video processor 320 can color the aiming beam footprint with a color different from the background of the endoscopic image. The video processor 320 can identify the current illumination position of the aiming beam 240b by matching the color of the aiming beam 240b with the colors of the pixels in the endoscopic image.

[0088] The video processor 320 may include a landmark detector 321 configured to detect one or more landmarks from an endoscopic image. The landmarks may be manually created by an operator or automatically identified using an image processing algorithm. In one example, the landmark detector 321 may detect landmarks based on changes in pixel brightness in the endoscopic image. In another example, the landmark detector 321 may detect landmarks using edge detection constrained by a minimum contrast threshold and a number of pixels between similar positive and negative contrast slopes. Detected landmarks may indicate blood vessels. Edge detection may involve detecting a transition from bright to dark brightness at the beginning of a segment indicating a blood vessel, and a subsequent transition from dark to bright brightness at the end of a segment indicating a blood vessel. Other criteria may be applied to confirm the detection of blood vessels. For example, if the subsequent transition occurs with at least a user-defined number of dark pixels and is limited above by a threshold number of bright pixels on both sides, then the edge defined by the transition between bright and dark pixels can be used as a landmark, provided that the points along one or both edges are at least greater than the length of another threshold. In another example, if the line generated by linear regression on the pixels of the edge has an R-squared or other fit greater than the target threshold (e.g., 0.8 in this example), the detected edge is identified as a blood vessel.

[0089] In endoscopic images, landmarks can have different forms. In one example, a landmark can be represented as a line segment in an endoscopic image. In another example, a landmark can be represented as two or more line segments intersecting at a point in an endoscopic image; this is called an intersecting landmark, or a point landmark. In some examples, for two non-intersecting and non-parallel line segments that are close to each other (e.g., within a specific distance range), the landmark detector 321 can use an algorithm to extend one line segment until it intersects the other line segment to create a point landmark.

[0090] Landmark detector 321 can locate landmarks relative to the aiming beam footprint in the coordinate system of the endoscopic image. For example, the position of a point landmark in the endoscopic image can be represented by a vector between the point landmark and the aiming beam footprint, or by distance along the X and Y axes in the coordinate system. In some examples, landmark detector 321 can determine the spatial relationships between landmarks (e.g., distances and slopes between landmarks) in the coordinate system of the endoscopic image. Information about the landmarks and their positions, the aiming beam footprint, and the spatial relationships between landmarks can be stored in memory 340 and used, according to the various examples discussed herein, for endoscopic image registration, target image reconstruction, or endoscopic tracking during endoscopic procedures.

[0091] Landmark detector 321 can select a subset of detected landmarks to store in memory 340 or for applications such as image registration, target map reconstruction, or endoscopic tracking. In one example, the subset of landmarks can be selected based on their location (e.g., the spatial distribution of landmarks in an endoscopic image). For example, landmarks scattered throughout an endoscopic image may be more easily selected than clusters of closely spaced landmarks in the endoscopic image. In another example, the subset of landmarks can be selected based on whether laser energy is activated at the target site where the landmark is located. Because laser energy can affect the accuracy and consistency of landmark detection, in this example, a landmark not activated by laser energy may be more advantageously selected than another landmark activated by laser energy.

[0092] In some examples, the endoscope controller 103 can control the light source 104 or the illumination lens 122 to produce specific illumination conditions for the target 101 to improve landmark detection and localization. For example, the light source 104 can provide blue or green illumination to increase the contrast of the target 101 in the endoscopic image and more clearly define vessels that are unlikely to move or change over time. This allows for more consistent landmark detection and localization under slightly different illumination conditions. In the examples, the endoscope controller 103 can temporarily change the illumination, such as turning on a green or blue light source for optimal landmark identification, and return to the normal illumination mode after landmark identification.

[0093] The video processor 320 may include a target recognizer 322 configured to identify the type of target at the aiming beam position of target 101. In an example, the target type identification may be based on one or more spectral characteristics of the illumination light reflected from target 101. The spectral characteristics may be measured using spectrometer 208. The identified target type may include anatomical tissue type or stone type. Examples of stone types may include stones or stone fragments in different stone formation sites such as the urinary system, gallbladder, nasal cavity, gastrointestinal tract, stomach, or tonsils. Examples of anatomical tissue types may include soft tissue (e.g., muscle, tendon, ligament, blood vessel, fascia, skin, fat, and fibrous tissue), hard tissue such as bone, connective tissue such as cartilage, etc. In an example, the target recognizer 323 may identify the tissue type at the aiming beam position of target 101 as normal and abnormal tissue, or mucosal or muscular tissue, based on the characteristics of the reflected illumination signal.

[0094] The video processor 320 can use visual identifiers indicating the identified tissue type to mark the aiming beam footprint in an endoscopic image (as displayed on the monitor 108). In an example, the visual identifiers may include color codes, allowing the aiming beam footprint to be colored differently to indicate different tissue types. For example, if the target site is identified as normal tissue, the aiming beam footprint may be colored green, or if the target site is identified as abnormal tissue (e.g., cancer), the aiming beam footprint may be colored red. In another example, the video processor 320 can use visual identifiers indicating a change in tissue type at the target site over time (e.g., from normal to abnormal, or vice versa), for example, by using a different color than that representing normal or abnormal tissue. In yet another example, the video processor 320 can use visual identifiers indicating the treatment status at the target site to mark the aiming beam footprint. For example, if the target site has been treated (e.g., with laser therapy), the aiming beam footprint may be represented by dots of a different color than those representing normal or abnormal tissue.

[0095] Figures 4A to 4FA sequence {G1, G2, ..., GN} of endoscopic images (e.g., video frames) taken at target 101 (e.g., the interior of the kidney, bladder, urethra, or ureter, and other anatomical structures of interest) is shown by way of example, not limitation. The endoscopic images can be displayed on display 108. As described above, endoscopic images of the same target site can be taken at different times, or endoscopic images of different target sites {S1, S2, ... SN} can be taken as the endoscope tip translates over target 101. Target recognizer 322 can identify the target type at the aiming beam position corresponding to the target site {S1, S2, ... SN}. Video processor 320 can mark the aiming beam footprint in the corresponding endoscopic images using appropriate visual identifiers (e.g., color) that identify the corresponding target type, detect and locate landmarks from the endoscopic images, and integrate the endoscopic images into a target map of target 101 based on the landmarks identified from the endoscopic images.

[0096] like Figures 4A to 4F The endoscopic images 410-460 shown are generated as the distal end of the endoscope is translated over the target 101, during which time the distal end of the endoscope is manually or automatically moved and positioned at different endoscopic locations. Figure 4A An endoscopic image 410 is shown, which includes a graphical representation of the illuminated target region S1 falling within the field of view (FOV) of the imaging system 115, and a circular aiming beam footprint 412. The aiming beam footprint 412 is tinted green to indicate that the tissue at the aiming beam location is normal tissue. In this example, image 410 also shows an image 413 of the distal end 112a of the optical path 112 (e.g., a laser fiber) and an image 414 of the distal portion of the endoscope 102.

[0097] Image 410 also includes, for example, landmarks 415A-415C detected by target recognizer 322. In this example, landmarks 415B and 415C are each represented by two line segments that intersect to form a point landmark, and landmark 415A is represented by two line segments that intersect by an algorithm (e.g., by projecting one line segment onto another) to form a point landmark. The positions of landmarks 415A-415C can be determined by target recognizer 322 (as described above). Figure 3 (As discussed). Images 410, including information about the aiming beam footprint 412 and landmarks 415A-415C, can be stored in memory 340.

[0098] As the distal endoscope is moved manually or automatically to a new endoscope position, images can be generated such as... Figure 4BAnother endoscopic image 420 is shown. The new endoscopic image 420 includes a graphical representation of the new illuminated target area S2 corresponding to the new endoscopic position and a new aiming beam footprint 422. Since the tissue at the current aiming beam location is identified as normal tissue, the aiming beam footprint 422 is stained green. If a new landmark is detected from the current endoscopic image, the new landmark can be included in the image. In this example, no new landmark is detected from the endoscopic image 420. If the previous aiming beam footprint 412 and the previously generated landmarks 415A-415C are located within the FOV of the imaging system at the current endoscopic position, the previous aiming beam footprint 412 and the previously generated landmarks 415A-415C can be retained in the current image 420.

[0099] If the distal end is moved in small steps, the illuminated target areas S1 and S2 may overlap, such that both endoscopic images 410 and 420 can cover the common area of ​​target 101 (e.g., Figure 4A and Figure 4B (As shown in the diagram). One or more matching landmarks can be identified from endoscopic images 410 and 420. Based on various examples discussed below, such matching landmarks can be used to align images 410 and 420 to reconstruct a map of the target.

[0100] The endoscope translation process can continue, and additional endoscopic images can be generated. Figure 4C Image 430 is shown, which includes a graphical representation of the newly illuminated target area S3 and a new aiming beam footprint 432, which is stained red to indicate the identification of abnormal tissue at the current aiming beam position. New landmarks 435A-435B can be detected from the current endoscopic image. The previous aiming beam footprint and previously generated landmarks (e.g., 415A-415C) remain within the FOV of the imaging system at the current endoscopic position and can be preserved in image 430.

[0101] Figure 4DImage 440 is shown, which includes a graphical representation of the new illuminated target area S4 and a new aiming beam footprint 442, the new aiming beam footprint 442 being tinted green to indicate the identification of normal tissue at the current aiming beam position. New landmarks 445A-445C can be detected from the current endoscopic image. The new aiming beam footprints (including their positions and colors representing tissue types) and the new landmarks (including their positions relative to the aiming beam footprints), as well as previous aiming beam footprints and previously generated landmarks, can be stored in memory 340. Previous aiming beam footprints and previously generated landmarks (e.g., 435B) falling within the FOV of the imaging system at the current endoscopic position can be saved in image 440.

[0102] The distal endoscope can be moved manually or automatically along a specific path or following a specific pattern, such that the endoscopic images generated during the translation process can collectively provide panoramic coverage of the basic surface area of ​​target 101. As a non-limiting example, Figures 4A to 4F The rectangular path indicated by the aiming beam footprint in the corresponding endoscopic image is shown. Moving horizontally to the left (as...) Figures 4A to 4D (As shown) After that, the distal end of the endoscope moves vertically upward, during which time endoscopic images of each target area can be captured. Figure 4E Image 450 is shown, which includes a graphical representation of the newly illuminated target area S5 and a new aiming beam footprint 452, which is stained green to indicate the identification of normal tissue at the current aiming beam position. New landmarks 455A-455B can be detected from the current endoscopic image. Previous aiming beam footprints and previously generated landmarks falling within the FOV of the imaging system at the current endoscopic position are preserved in image 450.

[0103] After the upward vertical movement, the distal end of the endoscope moves horizontally to the right, during which endoscopic images of various target areas can be captured. Figure 4F Image 460 is shown, which includes a graphical representation of the illuminated target area S6 (which includes a portion of the previously visited illuminated area captured in image 410) and a new aiming beam footprint 462, which is stained green to indicate the identification of normal tissue at the current aiming beam location. New landmarks 465A-465B can be detected from the current endoscopic image. Previous aiming beam footprints and previously generated landmarks falling within the FOV of the imaging system at the current endoscopic location are preserved in image 460. This includes a previously generated landmark 435A that previously fell outside endoscopic images 440 and 450.

[0104] Return to Figure 3The video processor 320 may include a target image generator 323, which is configured to reconstruct a target image by integrating multiple endoscopic images (or video frames) {G1, G2, ..., GN} of various target parts of the target 101 stored in the memory 340. (Refer to the above) Figures 4A to 4F The stored endoscopic image Gi discussed may include a graphical representation of the illuminated target site, a targeting beam footprint (including its location and color representing tissue type), and one or more landmarks (including their positions relative to the targeting beam footprint and the spatial relationships between the landmarks). The target image generator 323 may perform image registration to align the stored endoscopic image {G1, G2, ..., GN} based on landmarks with relative positions. Image registration may include identifying matching landmarks, including two or more landmarks identified from a first endoscopic image (e.g., image Gi taken at a first target site Si) that match two or more landmarks identified from a second endoscopic image (e.g., image Gj taken at a different second target site Sj), and the target image generator 323 aligns the second image with the first image relative to the identified matching landmarks. For example, it can be used in… Figure 4B Image 420 and Figure 4C Matching landmarks 415A-415C, present in both images 430, align image 430 with image 420. The aligned images can then be stitched together with respect to the matching landmarks to reconstruct the target image. In some examples, landmark detector 321 can adjust the landmark detection algorithm (e.g., reduce the edge detection threshold) to allow more landmarks to be identified from the endoscopic images. Multiple landmarks can increase the probability of identifying matching landmarks between images and improve the accuracy of image alignment.

[0105] Figure 5 An example of a target map 500 for target 101 (e.g., a solid region of the bladder) is shown. In addition to the stitched images, the reconstructed map may additionally include one or more of the following: a set of landmarks identified from multiple endoscopic images, a targeting beam footprint, a target type identifier (e.g., a color code for the targeting beam footprint), landmark positions relative to the targeting beam footprint, or spatial relationships between landmarks. Target map 500 can be used to assist in medical diagnosis or treatment planning, such as locating and tracking the endoscope position during endoscopic procedures.

[0106] It may be used to reconstruct the target image from the endoscopic image (e.g., for reconstruction). Figure 5 Target Figure 500 Figures 4A to 4FEndoscopic images (410-460) introduce geometric distortions or deformations, resulting in mismatches in common areas between images. For example, moving the distal end of the endoscope closer to or away from the target, or body movement (such as breathing), can cause the image to be magnified or reduced. Changes in the viewing direction (from the imaging system 115 at the distal end of the endoscope toward the target) can cause the image to rotate. In some cases, geometric distortions or deformations may be caused by changes in endoscope orientation. In this document, endoscope orientation refers to the tilt or skewing of the endoscope tip relative to the target surface. Changes in endoscope orientation from one image to another can cause distortions in length, shape, and other geometric properties. To correct such distortions or deformations, in some examples, the target image generator 323 can transform the image before aligning it with another image. Examples of image transformations can include one or more of scaling, translation, rotation, or shearing transformations in a coordinate system, as well as other rigid transformations, similarity-based transformations, or affine transformations. In the example, the image can be scaled using a scaling factor based on the distance between the landmarks measured from the two images respectively, or a scaling factor based on the geometric features measured from the aiming beam footprints in the two images respectively (e.g., as discussed below). Figure 7 (As described). In the example, changes in endoscope orientation can be corrected based on the slope between landmarks measured from the two images respectively, or based on geometric features measured from the aiming beam footprint in the two images respectively (e.g., as described below regarding...). Figures 6A to 6F (as described).

[0107] The transformation can be implemented as a transformation matrix multiplied by image data (e.g., a data array). The transformed image and another image can be aligned relative to matching landmarks between the transformed image and the other image. In some examples, the alignment can be based on the slopes of multiple landmarks relative to each other. Such alignment can be insensitive to the distance between the landmarks (related to differences in magnification or the distance between the endoscope and the target). Image transformations according to the various examples described herein can improve the robustness of target image reconstruction to differences in rotation, magnification, or reduction of the endoscopic image.

[0108] Many landmarks in the transformed endoscopic images are stored in memory 340 in their transformed state (as if on the two-dimensional projection surface of the target). Therefore, the landmarks in the transformed endoscopic images are invariant to surface inhomogeneities, tilts, rotations, scaling, and other distortions or deformations. The stored transformed endoscopic images can be integrated to form an integrated target map. The stored landmarks can be used as a basis for comparison with new images, or to transform new images to a two-dimensional projection surface and register the new images onto the stored target map (as referred to below). Figure 7 (as described).

[0109] Figures 6A to 6F This diagram illustrates the effect of endoscope orientation on the characteristics of endoscopic images and methods for correcting differences in endoscope orientation between two endoscopic images. The endoscopic orientation correction methods discussed in this paper can be applied to image registration applications, such as registering real-time intraoperative endoscopic images to a target image (e.g., target image 500), which will be referred to below. Figure 7 Discussion. Endoscopic orientation refers to the tilt or skew angle θ of the lens system 118 relative to the target surface. For two endoscopic images Gi and Gj taken with different endoscopic orientations, image properties such as landmark positions (e.g., distances from the aiming beam footprint) and spatial relationships between landmarks (e.g., distances between landmarks) are measured in the respective coordinate systems of the two images. If Gi and Gj are endoscopic images of the same target site, the image properties of endoscopic images Gi and Gj are measured in the same coordinate system by correcting for this difference in endoscopic orientation. Assessing anatomical differences or similarities based on image properties (e.g., distances between landmarks) between the two images is more robust to different imaging conditions. If Gi and Gj are endoscopic images of different target sites (e.g., ... Figures 4A to 4F By correcting for such differences in endoscope orientation (two images during endoscope translation), inconsistencies between endoscopic images Gi and Gj can be reduced, and the integration of Gi and Gj (as part of target image 500) can provide a more reliable representation of the extended surface region of target 101.

[0110] Figure 6A The first endoscope orientation θ1 is shown, wherein the end of the endoscope 102 is perpendicular to the surface of the target site 611 (i.e., θ1 = 90 degrees), and the lens system 118 is parallel to the target site 611. Figure 6C The image shown is an endoscopic image 615 taken at the endoscopic orientation θ1. Figure 6B The second endoscope orientation θ2 is shown, wherein the end of the endoscope 102 is tilted relative to the target site 621 (i.e., θ2 is an acute angle), and the lens system 118 is not parallel to the target site 621. Figure 6D Endoscopic image 625 taken at endoscope orientation θ2 is shown. Matching landmarks {M1, M2, M3} (e.g., in the form of intersecting line segments) can be identified from endoscope images 615 and 625 by landmark detector 321.

[0111] The target image generator 323 can use features generated from images 615 and 625 respectively to detect changes in endoscope orientation from θ1 to θ2, transform endoscope image 625 to correct for changes in endoscope orientation, and align the transformed image 625 with image 615 relative to matching landmarks {M1, M2, M3}. Figure 6C and Figure 6DAs shown, due to different endoscope orientations, the spatial relationships (e.g., distances and relative positions) between landmarks {M1, M2, M3} in image 615 may differ from the relative positions between landmarks {M1, M2, M3} in image 625. In the example, the spatial relationships between landmarks can be represented by the slope between two landmarks in a coordinate system (e.g., the slope k13 between landmarks M1 and M3), which can be calculated as the ratio of the distance y13 on the y-axis between M1 and M3 to the distance x13 on the x-axis between M1 and M3, i.e., k13 = y13 / x13. To determine the change in endoscope orientation, the target image generator 323 can compare the first slope (e.g., k13 = y13 / x13) between two landmarks in image 615 with the second slope (e.g., k13' = y13' / x13') between the same two landmarks in image 625. In the example shown, the relative slope (e.g., the ratio between k13 and k13') can indicate a change in endoscope orientation.

[0112] Alternatively, in some examples, the target map generator 323 may use geometric features generated from the aiming beam footprints in images 615 and 625, respectively, to detect changes in endoscope orientation. Figure 6A and Figure 6B The distal end 112a of the laser fiber (an example of optical path 112) is shown directing the aiming beam toward its respective target site. As shown in the respective endoscopic images 615 and 625, the resulting aiming beam footprints 612 and 622 have different geometric characteristics due to the different endoscope orientations. Corresponding to an endoscope orientation θ1 = 90°, Figure 6E The circular aiming beam footprint 612 with diameter d is shown. This corresponds to an endoscope orientation θ1 < 90°. Figure 6F An elliptical aiming beam footprint 622 is shown, with a major axis 623 of length "a" and a minor axis 624 of length "b". In this example, the target image generator 323 can use the elliptical axis length ratio Re = a / b to determine the endoscope orientation. For the elliptical footprint 622, the elliptical axis length ratio Re > 1. A larger elliptical axis length ratio indicates a more tilted endoscope orientation. For a circular footprint 612 with diameter d, the major and minor axes a = b = d, and the elliptical axis length ratio Re = 1. The target image generator 323 can determine changes in endoscope orientation based on a comparison between the elliptical axis length ratios calculated from the aiming beam footprints 612 and 622, respectively, transform the endoscope image 625 to correct for changes in endoscope orientation, and align the transformed image 625 with image 615 relative to an identified matching landmark.

[0113] Return to reference Figure 3 The endoscope tracker 330 can use a pre-generated target map (e.g., as shown in the image) during endoscopic surgery. Figure 5 The target (Figure 500) shown is located and tracked at the endoscope tip. Endoscopic tracking can begin by capturing real-time images or video signals from the surgical site of the target 101 using imaging system 115, and generating real-time images or video frames using video processor 320, as described above regarding the generation of endoscopic images (e.g., Figures 4A to 4F The discussion of reconstructing target map 500 is similar to that of one of the images shown. Imaging system 115 can be positioned at an unknown endoscope location. Landmark detector 321 can identify one or more landmarks from the real-time images. Endoscope tracker 330 can register the real-time image of target 101 with a pre-generated target map (e.g., target map 500) and locate the captured area in the real-time image based on the target map.

[0114] Endoscopic tracker 330 can determine changes in tissue state at the target site (e.g., a change from normal tissue to abnormal tissue, or vice versa). Endoscopic tracker 330 can, for example, locate and track the endoscope tip during surgery based on landmarks identified from real-time images and stored landmarks associated with the target image. In an example, endoscopic tracker 330 can identify two or more matching landmarks between the landmarks of the target image and the landmarks of the real-time image, register the real-time image to the target image using the identified matching landmarks, and locate and track the endoscope tip based on the registration of the real-time image.

[0115] Figure 7 An example is shown of identifying matching landmarks between real-time image 710 and reconstructed target image 500, and of registering real-time image 710 to target image 500 relative to the matching landmarks. In the example, matching landmarks can be identified based on the distance ratio (r) between the landmarks. (Refer to above) Figure 5 The target map 500 discussed contains pairs of landmark distances {d1, d2, d3, ..., dK}, where K represents the number of landmark pairs identified from the target map 500. The distance ratio {r1, r2, ..., rM} between any two of the K landmark distances {d1, d2, d3, ..., dK} can be calculated, where M represents the number of distance ratios.

[0116] According to one example, to identify matching landmarks, the endoscope tracker 330 can identify the intersecting landmark 711 that is closest to the current aiming beam footprint 701 in a set of intersecting landmarks (i.e., intersecting line segments) with their respective intersection point locations in the real-time image. The distances from landmark 711 to other landmarks in the image 710 can be measured: distance D1 to Pc, distance D2 to Pd, distance D3 to Pb, distance D4 to Pa, and so on. The endoscope tracker 330 can then calculate the distance ratios (R) between distances originating from the same landmark (such as landmark 711 in this example): R1 = D1 / D2, R2 = D1 / D3, R3 = D1 / D4, and so on. The distance ratios {R1, R2, R3} can be compared with the distance ratios {r1, r2, ..., rM} associated with the same target image 500. When the distance ratios {R1, R2, R3} (corresponding to the initial landmark 711 in the real-time image 710) match the distance ratios {rx, ry, rz} (corresponding to the initial landmark Pk in the target image 500), such that R1=rx, R2=ry, R3=rz, then the distances {D1, D2, D3, D4} have a high probability of matching the distances {d1, d2, d3, d4}; and the landmarks {Pa, Pb, Pc, Pd} in the real-time image 710 have a high probability of matching the landmarks {p1, p2, p3, p4} in the target image 500. The more matching distances there are, the higher the probability of landmark matching between the real-time image 710 and the image 500.

[0117] The endoscope tracker 330 can then determine the correspondence between distances {D1, D2, D3, D4} and distances {d1, d2, d3, d4} based on the ratio of the distances between the landmarks. For example, if D1 / D2 = d1 / d2, then D1 = d1 and D2 = d2. By checking various combinations until they all match, the endoscope tracker 330 can identify which distance in {D1, D2, D3, D4} corresponds to which distance in {d1, d2, d3, d4}. Since all matches are made relative to D1, if D1 is identified as d1, the remaining distances D2, D3, and D4 can be matched to d2, d3, and d4 respectively. The correspondence between {Pa, Pb, Pc, Pd} and {p1, p2, p3, p4} can also be determined through the correspondences established between D1 and d1, D2 and d2, D3 and d3, and D4 and d4.

[0118] Endoscope tracker 330 can register real-time image 710 to target image 500 using identified matching landmarks {Pa, Pb, Pc, Pd} that match landmarks {p1, p2, p3, p4} in target image 500. To correct for geometric distortions or deformations in the image, such as those caused by image magnification, reduction, rotation, or changes in endoscope orientation, endoscope tracker 330 can transform real-time image 710 or target image 500 in a manner similar to that discussed above, i.e., transforming a first endoscope image, aligning the first endoscope image with a second endoscope image, and reconstructing the panoramic target image using at least the transformed first and second images (as described above). Figures 4A to 4F (As discussed). Transformations can include one or more operations such as scaling, translation, or rotation. A transformation can be implemented as a transformation matrix multiplied by the image data (e.g., a data array) of the target image 500 in the coordinate system of the real-time image 710. Alternatively, the transformation can be applied to the real-time image 710.

[0119] like Figure 7 As shown, the transformation can include scaling factors. Figure 500 is scaled to correct for differences in image magnification or reduction between real-time image 710 and Figure 500. The scaled Figure 720 also includes elements determined by the scaling factor. The scaled landmarks and their positions (e.g., relative distances to the aiming beam footprint) and the distances between landmarks. In the example, the scaling factor. The distance ratio r can be determined using the ratio of the distance between two matching landmarks (e.g., P1 and P2) in real-time image 710 to the distance between the corresponding two landmarks (e.g., Pa and Pb) in Figure 500. In the example, the largest inter-landmark distance among the matching landmarks in real-time image 710 can be selected for calculating the distance ratio r. For example, if D4 is the largest distance among {D1, D2, D3, D4}, then the scaling factor... =D4 / d4.

[0120] scaling factor The scaling factor can be determined alternatively or additionally by comparing the shape of the aiming beam footprint in real-time image 710 with the shape of the aiming beam footprint in Figure 500. In the example, the scaling factor can be determined using the ratio of the geometric features of the aiming beam footprint in real-time image 710 to the corresponding geometric features of the aiming beam footprint in Figure 500. For example, if real-time image 710 has a circular footprint with a diameter of dR (such as...), the scaling factor can be determined using the ratio of the geometric features of the aiming beam footprint in real-time image 710 to the geometric features of the aiming beam footprint in Figure 500. Figure 6E As shown), while Figure 500 has a circular footprint with a diameter of dM, the scaling factor is... =dR / dM. In the example, if the real-time image 710 has an elliptical footprint (such as... Figure 6FAs shown in Figure 500, the major axis length is aR and the minor axis length is bR. Figure 500 has an elliptical footprint with a major axis length of aM and a minor axis length of bM. Therefore, the scaling factor... =aR / aM, or =bR / bM.

[0121] Scaling factor calculated above Assume the surface onto which the aiming beam is projected is flat. In some cases, the surface onto which the aiming beam is projected may not be perfectly flat, but rather have a three-dimensional shape. This may affect the calculation of the scaling factor. Changes are introduced. The system can adapt to an ideal scaling factor. The variation is shown. In the example, multiple aiming beam footprints can be captured as the aiming beam is pointed at different locations on a target portion with a corresponding projection surface. The superposition of multiple aiming beam footprints can reveal variations in the shape of the aiming beam footprint. Geometric features (e.g., the diameter of a circular footprint or the length of the major or minor axis of an elliptical footprint) can be measured separately from the multiple aiming beam footprints, and multiple corresponding scaling factors can be calculated. The scaling factors can be averaged or weighted to obtain the scaling factor. The expected value.

[0122] Scaling factor as described above The determination is based on the assumption that the endoscope orientation remains substantially unchanged between real-time image 710 and Figure 500 (e.g., the tilt of the endoscope tip relative to the target surface is substantially the same). In cases where significantly different endoscope orientations exist, measures such as landmark position, inter-landmark distance, shape of the aiming beam footprint, and its geometric properties (e.g., the lengths of the major and minor axes) may be affected by the endoscope orientation. Real-time image 710 or Figure 500 can be transformed to correct for variations in endoscope orientation (e.g., according to information regarding...). Figures 6A to 6F (Description). Then the scaling factor can be determined based on the transformed image. .

[0123] The scaled image 720 (including the markers therein) can be aligned with the target image 500 relative to the identified matching markers (e.g., P1 and Pa as shown in Figure 730). The real-time image 710 can be translated toward the scaled image 720 such that Pa is at the same coordinate as P1 of the scaled image 720 (denoted as P1(Pa)). The scaled image 720 is then rotated 730 clockwise by an angle α or ∠Pb-P1(Pa)-P2. After rotation, the matching marker Pb is at the same coordinate as P2 of the scaled image 720 (denoted as P2(Pb)), as shown in the registered image 740. Since the scaling and rotation operations preserve the relative positions (e.g., angles) between the markers, other matching markers P3 and P4 also overlap with the markers Pc and Pd on the scaled image 720, denoted as P3(Pc) and P4(Pd) in the registered image 740. Therefore, the real-time image 710 is registered to the target image 500 with respect to the matching landmarks P1-P4 (corresponding to Pa-Pd in ​​image 710).

[0124] As described above, the registration of real-time images captured during endoscopic surgery with target maps can be used in various applications to improve the accuracy and efficiency of endoscopic procedures. In this example, image registration can assist the operator in identifying surgical sites with improved accuracy in real time from a pre-generated target map. Because the target map stores information about the position of the endoscope tip relative to multiple stored landmarks, the image registration discussed herein can assist in the real-time localization and tracking of the endoscope tip throughout the procedure. For target maps storing information about the target type (e.g., normal or abnormal tissue) at various aiming beam positions, the endoscope tracker 330 can detect and track changes in tissue type at various target sites over time, or provide an assessment of the effectiveness of treatment delivered at the target site.

[0125] Figure 8 This is a flowchart illustrating a method 800 for endoscopic mapping of targets within a subject's body during surgery. Method 800 may be implemented and performed by a medical system for endoscopic surgery (e.g., system 100) or a variant thereof. Although the procedures of method 800 are depicted in a flowchart, these procedures are not required to be performed in a specific order. In various examples, some procedures may be performed in a different order than shown herein.

[0126] At 810, a targeting beam can be emitted from the endoscope tip and directed at a part of the target (e.g., a portion of target 101). This targeting beam can be generated by a laser source, such as a second laser energy source 204. Alternatively, the targeting beam can be generated by other light sources and transmitted, for example, via optical fiber.

[0127] At 820, an image of the target area can be captured by an imaging system, such as imaging system 115. This image can be captured when the lens system 118 of imaging system 115 is positioned at the endoscope location. The image can be captured when the target is illuminated by electromagnetic radiation (also known as illumination light) in the optical range from UV to IR. The illumination light can be generated by a light source, such as light source 104, and transmitted to the target area via light guide 120. In this example, light source 104 may include two or more light sources emitting light with different illumination characteristics.

[0128] The aiming beam directed at the target area can fall within the field of view (FOV) of the imaging system, so that the image captured at the target area can include not only a graphic representation of the irradiated target (e.g., the surface of the target's anatomical structure) but also the trail of the aiming beam. This image can be displayed to the user, for example, on display 108 (e.g., on a monitor). Figures 4A to 4F (As shown in any of them). The aiming beam footprint can be colored differently from the background of the endoscopic image. In the example, the location of the aiming beam footprint can be identified from the endoscopic image by matching the color of the aiming beam with the color of pixels in the endoscopic image.

[0129] The signal reflected from the target in response to the illumination light can be analyzed, for example, using a spectrometer 208 to identify the target type at the aiming beam position. In examples, based on the spectral characteristics of the reflected signal, the tissue at the target site can be identified as normal and abnormal tissue, or mucosal or muscle tissue, as well as other anatomical structure types. In some examples, the target site can be identified as one of several stone types with their respective components.

[0130] The aiming beam footprint in endoscopic images can be marked with visual identifiers that indicate the tissue type at the location of the aiming beam. In examples, the aiming beam footprint can be colored differently to indicate different tissue types. For instance, if the target site is identified as normal tissue, the aiming beam footprint could be colored green, or if the target site is identified as abnormal tissue (e.g., cancer), the aiming beam footprint could be colored red. In some examples, the aiming beam footprint can be marked with identifiers to indicate changes in tissue type over time or to indicate treatment status.

[0131] At 830, for example using video processor 320, one or more landmarks can be identified from the captured image of the target site, and the landmark positions relative to the aiming beam footprint can be determined. In the example, the landmarks represent anatomical structures (e.g., blood vessels). Landmarks can be detected based on variations in pixel brightness in the endoscopic image. In the example, landmarks can be detected using edge detection constrained by a contrast threshold, and similarly, the number of pixels between positive and negative contrast slopes.

[0132] The location (e.g., X and Y distance) of one or more landmarks in the coordinate system of an endoscopic image can be determined relative to the aiming beam footprint in the same endoscopic image. In another example, landmark localization involves determining the distances between landmarks in the coordinate system of the endoscopic image. In some examples, a subset of detected landmarks can be selected based on the spatial distribution of landmarks in the endoscopic image. For example, the selected subset may include landmarks distributed throughout the endoscopic image (rather than a tightly spaced cluster of landmarks in one region of the image). In another example, landmarks can be selected based on whether laser energy activates them, as laser energy may distort such landmarks. For example, a landmark not activated by laser energy may be more advantageously selected than another landmark activated by laser energy.

[0133] In some examples, the target can be illuminated under specific lighting conditions to improve landmark detection and localization. For instance, the target can be illuminated with blue or green light to increase the contrast of the endoscopic image of the target and to more clearly delineate blood vessels that are unlikely to move or change over time. This allows for more consistent landmark detection and localization under slightly different illumination conditions.

[0134] At 840, for example, a target map can be reconstructed by integrating multiple images based on landmarks identified from multiple images of various parts of the target. For example, when the endoscope tip is manually or automatically translated over the target by an endoscope actuator, the distal endoscope tip is moved and positioned at different endoscope positions {L1, L2, ..., LN}, and the imaging system can capture a series of endoscope images {G1, G2, ..., GN} at corresponding multiple target parts {S1, S2, ..., SN} of the target within the FOV of the imaging system at the corresponding endoscope positions. Figures 4A to 4F An example of a sequence of images (or video frames) captured at different parts of a target is shown.

[0135] A target image can be reconstructed by integrating multiple endoscopic images {G1, G2, ..., GN}. The endoscopic images {G1, G2, ..., GN} can be aligned with respect to landmarks identified from the images. In the example, between two endoscopic images Gi and Gj, matching landmarks can be identified, comprising two or more landmarks identified from image Gi that match two or more landmarks identified from image Gj. Images Gi and Gj can then be aligned with respect to the identified matching landmarks.

[0136] In some examples, an endoscopic image (e.g., Gi) may be transformed before being aligned with another endoscopic image (e.g., Gj). This transformation can correct geometric image distortions or deformations, such as image magnification or reduction caused by the distal endoscope moving too close or too far relative to the target, or by body movement (e.g., breathing); image rotation caused by changes in the viewing direction from the imaging system toward the target; or distortions in length, shape, and other image properties caused by changes in endoscope orientation. Examples of image transformations may include one or more of scaling, translation, rotation, or shearing transformations of the image in a coordinate system, as well as other rigid, similarity-based, or affine transformations.

[0137] In the example, before aligning the first endoscopic image Gi with the second endoscopic image Gj, image Gj can be scaled using a scaling factor based on the ratio of the distance between two matching landmarks in image Gi to the distance between corresponding two landmarks in image Gj. Scaling is applied. In another example, the scaling factor... The ratio of the geometric features derived from the aiming beam footprint in image Gi to those derived from the aiming beam footprint in image Gj can be used to determine the geometric features. Examples of geometric features could include the diameter of a circular footprint or the length of the major (or minor) axis of an elliptical footprint (as referenced above). Figure 7 (Discussed).

[0138] In the example, before aligning the first endoscopic image Gi with the second endoscopic image Gj, changes in endoscopic orientation between images Gi and Gj can be detected and corrected. In the example, changes in endoscopic orientation can be determined based on a comparison of a first slope between two matching landmarks in image Gi and a second slope between two matching landmarks in image Gj. In another example, changes in endoscopic orientation can be determined based on a comparison of a first geometric feature of the aiming beam footprint in image Gi and a second geometric feature of the aiming beam footprint in image Gj. In the example, at least one of the first or second aiming beam footprint has an elliptical shape having a major axis and a minor axis, and at least one of the first or second geometric features can include the ratio of the major axis length to the minor axis length of the elliptical aiming beam footprint (as referenced above). Figures 6A to 6F As discussed.

[0139] For example Figure 5As shown, the transformed image can be aligned and integrated into the reconstructed target map. The reconstructed map may include one or more of the following: a set of landmarks identified from multiple endoscopic images, a targeting beam footprint generated during tissue mapping, a target type identifier (e.g., a color-coded targeting beam footprint), the position of the landmarks relative to the targeting beam footprint, or the relative position between landmarks. The target map (including information about the landmarks and the targeting beam footprint) can be stored in memory 340. At 850, the target map can be used to locate and track the endoscope tip during endoscopic surgery (see reference below). Figure 9 Additionally or alternatively, the target map can be used to determine changes in tissue state at the target site (e.g., changes from normal to abnormal, or vice versa).

[0140] The systems and methods for image reconstruction and endoscopic tracking according to the various embodiments discussed herein can be adapted to tolerances around their measurements. Comparisons or equations described herein or inferred from the description herein are not limited to perfect equality. In examples, the systems and methods described herein may initially compare perfectly equal values, but then progressively expand the tolerance around each calculation to identify overlapping portions, which are then considered equal. For example, when comparing anatomical images of the same patient from two different times (potentially weeks, months, or years apart), the algorithm may progressively expand the tolerance window from an ideal value to a limit value, or may create a larger area around each landmark, then check for overlap using standard statistical methods such as t-tests or Mann-Whitney median comparisons, with a target confidence level of, for example, 5 to 25%. In examples, the tolerance window or confidence interval may be between 0% and X% of the distance between each pair of compared landmarks, where X% could be 20% in one example or 25% in another. Alternative locations, tolerance windows, or confidence intervals can be gradually increased until a proper match can be obtained among a considerable number of landmarks, for example, approximately 70-100% of the landmarks are found to be matched.

[0141] Figure 9 This is a flowchart illustrating an example of a method 900 for endoscopic tracking using a reconstructed target map (e.g., a target map generated using method 800). At 910, during endoscopic surgery, a real-time image of the surgical site at the target location can be captured, for example via an imaging system positioned at an unknown endoscopic location. At 920, matching landmarks can be identified, comprising two or more landmarks in the target map that respectively match two or more landmarks in the real-time image. Matching landmarks can be identified based on the distance ratio between the landmarks (as referenced above). Figure 7(As discussed above). At 930, the real-time image can be registered to the target image using identified matching landmarks. This registration can include image transformations (e.g., translation, scaling, rotation, etc.) and image alignment (as referred above). Figure 7 (As described above). At 940, the endoscope tip can be located and tracked based on real-time image registration. Because the target map stores positional information of the endoscope tip relative to multiple landmarks, the endoscope tracker 330 can locate and track the endoscope tip in real time based on landmarks on the registered image throughout the procedure. For target maps storing information about tissue types (e.g., normal or abnormal tissue at different target sites indicated by the aiming beam footprint), the endoscope tracker 330 can detect and track changes in tissue type at different sites of the target over time or the effectiveness of the treatment delivered thereto.

[0142] Figure 10 A block diagram of an example machine 1000 is shown in general, to which any or more of the techniques (e.g., methods) discussed herein can be performed. Parts of this specification can be applied to the computational framework of various parts of system 100, such as endoscope controller 103.

[0143] In alternative implementations, machine 1000 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 1000 may operate as a server machine, a client machine, or both in a server-client network environment. In the example, machine 1000 may operate as a peer-to-peer (P2P) (or other distributed) network environment. Machine 1000 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web application, network router, switch, or bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by that machine. Furthermore, although only a single machine is shown, the term "machine" may also be considered to include any collection of machines that individually or jointly execute a set (or more) of instructions to perform any one or more of the methods discussed herein, such as cloud computing, Software as a Service (SaaS), or other computer cluster configurations.

[0144] As described herein, examples may include logic or multiple components or mechanisms, or may be operated by logic or multiple components or mechanisms. A circuit group is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic, etc.). The members of a circuit group may vary over time and with changes in the underlying hardware. A circuit group includes members that can perform a specified operation individually or in combination during operation. In the example, the hardware of the circuit group may be invariably designed to perform the specified operation (e.g., hardwired). In the example, the hardware of the circuit group may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) to encode instructions for the specified operation, and the variably connected physical components may include computer-readable media that are physically modified (e.g., magnetically grounded, electrically grounded, movable placement of massless particles, etc.). When the physical components are connected, the underlying electrical properties of the hardware components are changed, for example, from an insulator to a conductor or from a conductor to an insulator. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create portions of the circuit group in the hardware via variable connections to perform the specified operation during operation. Therefore, when the device is in operation, the computer-readable medium is communicatively coupled to other components of the circuit group members. In the example, any component of the physical components can be used in more than one member of more than one circuit group. For example, in operation, an execution unit can be used at one point in time in a first circuit of a first circuit group, and can be reused at different times by a second circuit in the first circuit group or by a third circuit in the second circuit group.

[0145] Machine (e.g., computer system) 1000 may include a hardware processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 1004, and static memory 1006, some or all of which may communicate with each other via an interconnect link (e.g., a bus) 1008. Machine 1000 may also include a display unit 1010 (e.g., a raster display, a vector display, a holographic display, etc.), an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In the example, display unit 1010, input device 1012, and UI navigation device 1014 may be a touch screen display. Machine 1000 may additionally include a storage device (e.g., a drive unit) 1016, a signal generation device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1021, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 1000 may include output controller 1028, for example, serial (e.g., Universal Serial Bus (USB), parallel or other wired or wireless (e.g., infrared (IR), near field communication (NFC) etc.) connection, to communicate or control one or more peripheral devices (e.g., printer, card reader, etc.).

[0146] Storage device 1016 may include a machine-readable medium 1022 on which one or more sets of data structures or instructions 1024 (e.g., software) are stored, said set of one or more sets of data structures or instructions 1024 implementing or being used by any one or more of the techniques or functions described herein. Instructions 1024 may also reside wholly or at least partially within main memory 1004, static memory 1006, or hardware processor 1002 during execution by machine 1000. In this example, one or any combination of hardware processor 1002, main memory 1004, static memory 1006, or storage device 1016 may constitute a machine-readable medium.

[0147] Although machine-readable medium 1022 is shown as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to store one or more instructions 1024.

[0148] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions for use by machine 1000 and to cause machine 1000 to perform any one or more of the techniques of this disclosure, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media can include solid-state memory, as well as optical and magnetic media. In examples, mass-capacity machine-readable media includes machine-readable media having a plurality of particles having invariant (e.g., rest) mass. Therefore, mass-capacity machine-readable media are not transiently propagating signals. Specific examples of mass-capacity machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPSOM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0149] Commands 1024 can also be sent or received via the communication network 1026 using a transmission medium via the network interface device 1020, utilizing any of several transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., the IEEE 802.11 family of standards known as WiFi®, the IEEE 802.16 family of standards known as WiMax®), the IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, etc. In this example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet jacks, coaxial jacks, or telephone jacks) or one or more antennas for connection to the communication network 1026. In the example, network interface device 1020 may include multiple antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions for execution by machine 1000, and the term "transmission medium" should be considered to include digital communication signals or analog communication signals or other intangible media to facilitate communication of such software.

[0150] Additional notes

[0151] The above detailed description includes reference to the accompanying drawings, which form part of the detailed description. For illustration, the drawings show specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Furthermore, the inventors contemplate examples using any combination or substitution of these elements (or one or more aspects of these elements) shown or described with respect to a particular example (or one or more aspects of a particular example) or with respect to other examples shown or described herein (or one or more aspects of other examples).

[0152] In this document, as is common in patent documents, terms without quantifiers are used to include one or more, unrelated to any other instance or use of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to indicate a non-exclusive "or", such that "A or B" includes "A but not B", "B but not A", and "A and B". In this document, the terms "comprising" and "in..." are used as concise English equivalents to the corresponding terms "including" and "wherein". Furthermore, in the appended claims, the terms "comprising" and "including" are open-ended, meaning that a system, apparatus, article, composition, formulation, or process that includes elements other than those listed after this term in a claim is still considered to fall within the scope of that claim. Additionally, in the following claims, the terms "first", "second", and "third", etc., are used merely as designations and are not intended to impose numerical requirements on their objects.

[0153] The above description is intended to be illustrative and not restrictive. For example, the examples (or one or more aspects of the examples) described above can be used in combination with each other. Other embodiments can be used by those skilled in the art after consulting the above description. An abstract is provided to enable the reader to quickly determine the nature of the technical disclosure. The abstract is submitted under the understanding that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the specific embodiments described above, various features may be combined to organize the present disclosure. This should not be construed as meaning that all unclaimed disclosed features are necessary for any claim. Rather, the subject matter of the invention may lie in fewer than all features of a particular disclosed embodiment. Therefore, the appended claims are thus incorporated into the specific embodiments as examples or embodiments, wherein each claim is an independent, separate embodiment, and such embodiments are contemplated to be combined with each other in various combinations or substitutions. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

[0154] This technology can also be configured as follows:

[0155] (1) A system for endoscopic mapping of a target, the system comprising:

[0156] An imaging system configured to capture endoscopic images of the target, the endoscopic images including the footprint of a targeting beam directed at the target; and

[0157] Video processor, the video processor being configured to:

[0158] Identify one or more landmarks from the captured endoscopic images and determine the corresponding positions of said one or more landmarks relative to the aiming beam footprint; and

[0159] A target map is generated by integrating the multiple endoscopic images based on landmarks identified from one or more of the multiple endoscopic images.

[0160] (2) The system according to (1), wherein the video processor is configured to identify the tissue type at the location of the aiming beam and to mark the aiming beam footprint using a visual identifier indicating the identified tissue type.

[0161] (3) The system according to any one of (1)-(2) includes a spectrometer communicatively coupled to the video processor, the spectrometer being configured to measure one or more spectral characteristics of the illumination light signal reflected from the target;

[0162] The video processor is configured to identify the tissue type at the location of the aiming beam based on one or more spectral characteristics, and to mark the aiming beam footprint using a visual identifier indicating the identified tissue type.

[0163] (4) The system according to any one of (2)-(3), wherein the video processor is configured to identify the tissue type as normal tissue or abnormal tissue.

[0164] (5) The system according to any one of (2)-(3), wherein the video processor is configured to use different colors to mark the aiming beam footprint to indicate different tissue types.

[0165] (6) The system according to any one of (1)-(5), wherein the video processor is configured to identify the one or more landmarks from the endoscope image based on the change in brightness of the pixels of the endoscope image.

[0166] (7) The system according to (6), wherein the one or more landmarks are represented as line segments or intersecting line segments in the endoscopic image.

[0167] (8) The system according to any one of (1)-(7), wherein the video processor is configured to:

[0168] Based on whether laser energy is activated at each target location where the identified landmarks are located, a subset of landmarks is selected from the landmarks identified in one or more of the plurality of endoscopic images; and

[0169] The target image is generated by integrating the multiple endoscopic images based on a selected subset of landmarks.

[0170] (9) The system according to any one of (1)-(8), wherein the plurality of endoscopic images include images of various parts of the target, including a first endoscopic image of a first target part captured from a first endoscopic position and a second endoscopic image of a second target part captured from a second endoscopic position, wherein the video processor is configured to:

[0171] Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image;

[0172] Align the first endoscopic image and the second endoscopic image with respect to the matching landmark in the coordinate system of the first image; and

[0173] The target image is generated using at least an aligned first image and a second image.

[0174] (10) The system according to (9), wherein the video processor is configured to:

[0175] Transforming the second image includes scaling, translating, or rotating the second image, or performing one or more of these operations; and

[0176] Align the transformed second image and the first image with respect to the matching landmark.

[0177] (11) According to the system of (10), wherein the transformation of the second image includes matrix multiplication by a transformation matrix.

[0178] (12) The system according to any one of (10)-(11), wherein the video processor is configured to scale the second image using a scaling factor based on the ratio of the distance between two matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0179] (13) The system according to any one of (10)-(11), wherein the video processor is configured to scale the second image by a scaling factor based on the ratio of the geometry of the aiming beam footprint in the first image to the geometry of the aiming beam footprint in the second image.

[0180] (14). The system according to any one of (10)-(13), wherein the video processor is configured to transform the second image to correct for a change in endoscope orientation between the first image and the second image, the endoscope orientation indicating the tilt of the endoscope tip relative to the target site.

[0181] (15) The system according to (14), wherein the video processor is configured to detect changes in the endoscope orientation using a first slope between two markers in the matching landmarks in the first image and a second slope between two corresponding markers in the second image.

[0182] (16) The system according to (14), wherein the video processor is configured to detect changes in the endoscope orientation using a first geometric feature of the aiming beam footprint in the first image and a second geometric feature of the aiming beam footprint in the second image.

[0183] (17) The system according to (16), wherein,

[0184] At least one of the first or second aiming beam footprints has an elliptical shape, the ellipse having a major axis and a minor axis; and

[0185] At least one of the first geometric feature or the second geometric feature includes the ratio of the length of the major axis to the length of the minor axis.

[0186] (18) The system according to any one of (1)-(17) includes an endoscope tracking system, the endoscope tracking system being configured to:

[0187] Identify matching landmarks from real-time images of the surgical site of the target captured from an unknown endoscopic location by the imaging system during endoscopic surgery, the matching landmarks including two or more corresponding landmarks in the target image that match two or more landmarks in the real-time image;

[0188] The real-time image is registered to the target image using the matching delimiters; and

[0189] Based on the registration of the real-time images, the position of the endoscope tip is tracked.

[0190] (19) The system according to (18), wherein the endoscope tracking system is configured to identify the matching landmarks based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

[0191] (20) The system according to any one of (18)-(19)1, wherein the endoscopic tracking system is configured to generate an indication of changes in tissue type at the target site.

[0192] (21) A method for endoscopic mapping of a target, the method comprising:

[0193] Point the aiming beam at the target;

[0194] An endoscopic image of the target is captured via an imaging system, the endoscopic image including the footprint of the aiming beam;

[0195] Identify one or more landmarks from the captured endoscopic images via a video processor, and determine the corresponding positions of the one or more landmarks relative to the aiming beam footprint; and

[0196] The target image is generated by integrating the plurality of endoscopic images via the video processor based on landmarks identified from one or more of the plurality of endoscopic images.

[0197] (22) The method according to (21) includes:

[0198] Using the illumination light signal reflected from the target, the tissue type at the location of the aiming beam is identified; and

[0199] The target beam footprint is marked using a visual identifier that indicates the type of tissue being identified.

[0200] (23) The method according to any one of (21)-(22), wherein the identification of the one or more landmarks from the endoscope image is based on the variation of the brightness of the pixels of the endoscope image.

[0201] (24) The method according to any one of (21)-(23), wherein the plurality of endoscopic images include images of various parts of the target, including a first endoscopic image of a first target part captured at a first endoscopic position and a second endoscopic image of a second target part captured at a second endoscopic position, the method comprising:

[0202] Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image;

[0203] Align the first endoscopic image and the second endoscopic image with respect to the matching landmark in the coordinate system of the first image; and

[0204] The target image is generated using at least the aligned first and second images.

[0205] (25) The method according to (24), wherein aligning the first endoscopic image and the second endoscopic image includes:

[0206] Transforming the second image includes scaling, translating, or rotating the second image, or performing one or more of these operations; and

[0207] Align the transformed second image and the first image with respect to the matching landmark.

[0208] (26) According to the method of (25), wherein transforming the second image includes: scaling the second image by a scaling factor based on the ratio of the distance between two matching landmarks in the first image to the distance between two corresponding landmarks in the second image.

[0209] (27) According to the method of (25), wherein transforming the second image includes scaling the second image by a scaling factor based on the ratio of the geometric features of the aiming beam footprint in the first image to the geometric features of the aiming beam footprint in the second image.

[0210] (28) According to the method of (25), wherein transforming the second image includes: correcting for a change in endoscope orientation between the first image and the second image, the endoscope orientation indicating the tilt of the endoscope tip relative to the target site.

[0211] (29) The method according to any one of (21)-(28) further comprises:

[0212] During endoscopic surgery, the imaging system is used to capture real-time images of the target surgical site from an unknown endoscopic location;

[0213] Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the target image that match two or more landmarks in the real-time image;

[0214] The real-time image is registered to the target image using the matching delimiters; and

[0215] Based on the registration of the real-time images, the position of the endoscope tip is tracked.

[0216] (30) According to the method of (29), wherein the identification of matching landmarks is based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

Claims

1. A system for endoscopic mapping of a target, the system comprising: An imaging system configured to generate endoscopic images of the target; as well as Video processor, the video processor being configured to: Identify one or more landmarks from each of the endoscopic images and determine the respective positions of the identified one or more landmarks relative to an optical reference generated by a light source and displayed on the corresponding endoscopic image of the target; as well as A target map is generated by integrating the endoscopic images based on one or more landmarks identified from each of the endoscopic images, wherein the endoscopic images include images of various parts of the target.

2. The system of claim 1, further comprising an endoscope tracking system, the endoscope tracking system being configured to: Identify matching landmarks in the target map that match the corresponding landmarks in the real-time image of the target generated by the imaging system during endoscopic surgery; The real-time image is registered to the target image using the matching delimiters; and Based on the registration of the real-time images, the position of the endoscope tip is tracked.

3. The system according to claim 2, wherein, The endoscopic tracking system is configured to identify the matching landmarks based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

4. The system according to claim 2, wherein, The endoscopic tracking system is configured to generate indications of changes in tissue type at the target site.

5. The system according to claim 1, wherein, The optical reference includes the footprint of a targeting beam emitted from a laser or light source, directed at the target, and captured by the imaging system.

6. The system of claim 1, further comprising a spectrometer communicatively coupled to the video processor, the spectrometer being configured to measure one or more spectral characteristics of an illumination light signal reflected from the target; in, The video processor is configured to: identify the tissue type at the location of the optical reference based on one or more spectral characteristics, and to label the optical reference using a visual identifier indicating the identified tissue type.

7. The system according to claim 1, wherein, The video processor is configured to: In each of the endoscopic images, a subset of landmarks is selected from the identified one or more landmarks based on whether the laser energy is activated at the corresponding target location where one or more landmarks are identified; as well as The target image is generated by integrating the endoscopic images based on a selected subset of landmarks.

8. The system according to claim 1, wherein, The endoscopic images include images of various parts of the target, including a first endoscopic image of a first target part and a second endoscopic image of a second target part, wherein the video processor is configured to: Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; Align the first endoscopic image and the second endoscopic image with respect to the matching landmark in the coordinate system of the first image; and The target image is generated using at least an aligned first image and a second image.

9. The system according to claim 8, wherein, The video processor is configured to: Transforming the second image includes scaling, translating, or rotating the second image, or performing one or more of these operations; and Align the transformed second image and the first image with respect to the matching landmark.

10. The system according to claim 9, wherein, The video processor is configured to transform the second image to correct for a change in endoscope orientation between the first and second images, the endoscope orientation indicating the tilt of the endoscope tip relative to the target.

11. The system according to claim 10, wherein, The video processor is configured to detect changes in the endoscope orientation using a first geometric feature of a first optical reference displayed on the first image and a second geometric feature of a second optical reference displayed on the second image.

12. At least one non-transitory machine-readable storage medium, the at least one non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of the machine, cause the machine to perform operations, the operations including: Direct the light signal generated by the light source toward the target; Generate an endoscopic image of the target; Identify one or more landmarks from each of the endoscopic images and determine the respective position of the identified one or more landmarks relative to an optical reference generated by the optical signal and displayed on the corresponding endoscopic image of the target in the endoscopic image; as well as A target map is generated by integrating the endoscopic images, which include images of various parts of the target, based on one or more landmarks identified from each of the endoscopic images.

13. The at least one non-transitory machine-readable storage medium according to claim 12, wherein, The instruction causes the machine to perform an operation, the operation further including: Capture real-time images of the target during endoscopic surgery; Identify matching landmarks in the target image that match the corresponding landmarks in the real-time image of the target; The real-time image is registered to the target image using the matching delimiters; and Based on the registration of the real-time images, the position of the endoscope tip is tracked.

14. The at least one non-transitory machine-readable storage medium according to claim 13, wherein, The operation of identifying matching landmarks is based on one or more ratios of the distances between landmarks in the real-time image and one or more ratios of the distances between landmarks in the target image.

15. The at least one non-transitory machine-readable storage medium according to claim 14, wherein, The instructions cause the machine to perform an operation, which further includes: generating an indication of a change in the tissue type at the target location.

16. The at least one non-transitory machine-readable storage medium according to claim 12, wherein, The instruction causes the machine to perform an operation, the operation further including: Measure one or more spectral characteristics of the illumination light signal reflected from the target; Identify the tissue type at the location of the optical signal based on one or more of the spectral characteristics; and The optical reference is labeled using a visual identifier that indicates the type of tissue being identified.

17. The at least one non-transitory machine-readable storage medium according to claim 12, wherein, The instruction causes the machine to perform an operation, the operation further including: In each of the endoscopic images, a subset of landmarks is selected from the identified one or more landmarks based on whether laser energy is activated at the corresponding target location where one or more identified landmarks are located; and The target image is generated by integrating the endoscopic images based on a selected subset of landmarks.

18. The at least one non-transitory machine-readable storage medium according to claim 12, wherein, The images of various parts of the target include a first endoscopic image of a first target part and a second endoscopic image of a second target part, and wherein the instructions cause the machine to perform operations, the operations further including: Identify matching landmarks, wherein the matching landmarks include two or more corresponding landmarks in the first endoscopic image that match two or more landmarks in the second endoscopic image; Align the first image and the second image with respect to the matching marker in the coordinate system of the first image; and The target image is generated using at least the aligned first and second images.

19. The at least one non-transitory machine-readable storage medium according to claim 18, wherein, The operation of aligning the first endoscopic image and the second endoscopic image includes: Transforming the second image includes scaling, translating, or rotating the second image, or performing one or more of these operations; and Align the transformed second image and the first image with respect to the matching landmark.

20. The at least one non-transitory machine-readable storage medium according to claim 19, wherein, The instructions cause the machine to perform an operation, which further includes: using a first geometric feature of a first optical reference displayed on the first image and a second geometric feature of a second optical reference displayed on the second image to detect a change in endoscope orientation between the first image and the second image. The operation of transforming the second image includes: correcting for changes in endoscope orientation, wherein the endoscope orientation indicates the tilt of the endoscope tip relative to the target.

Images