Viewpoint correction for enhancing scene depth estimation

Abstract

An imaging system (100) and method for determining the size of an object viewed through a monocular endoscope (10) is presented. The system includes an imaging device (40) mounted on the endoscope, at least one light source disposed at a fixed position relative to the imaging device, and a processor (102) configured to accept input from the imaging device and determine the size of the viewed object by analyzing at least one of the shadowing effect from the light source, structured illumination provided by the light source, multi-point illumination including the light source, the time of flight associated with a light pulse from the light source, and lateral chromatic aberration from the light source.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 155,988, filed March 3, 2021, the disclosure of which is incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE The present disclosure relates to medical devices and, more particularly, to enhancements to depth determination in endoscopy procedures and methods of using such medical devices. [Background technology]

[0003] With any imaging system, the user must know the actual physical size of the displayed object in order to accurately interpret the image. In optical imaging systems that image a 2D scene at a fixed point in space, this is typically accomplished by calibrating the system's optical parameters, such as focal length and distortion, and using this information to calculate pixel size (often displayed using a scale bar). This is not possible in monocular optical imaging systems that image 3D scenes with significant depth. In these systems, the image sensor pixel size is fixed, but the physical size of the displayed object depends on the object's distance from the focusing optics. Two objects of the same size will appear different in the image, with objects farther from the optics appearing smaller.

[0004] This problem is common to all endoscopy systems. Solving this problem is especially important in ureteroscopic procedures. Knowing the physical size of the kidney stone (and / or residual stone fragments) can directly impact procedural decision-making and overall procedural efficiency. Optical calibration alone is not sufficient for ureteroscopic applications as anatomical features and / or stone fragments will have a significant range of distances from the main objective.

[0005] In existing practice, field size is estimated by intuitive comparison to objects of known diameter (e.g., comparing the size of a stone to an adjacent laser fiber). It takes a significant amount of time for surgeons to develop this intuition. For endoscopic imaging systems, even this intuition remains limited. Endoscopes are inherently spatial, resulting in designs that (1) use a single objective lens and (2) rely on fixed focal length optics. These constraints result in imaging configurations where there is variable magnification of the object across the scene, and any pixel detected by the sensor may represent a different physical size for the object. This fundamental challenge calls for more sophisticated designs that accurately measure size. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Provisional Patent Application No. 63 / 155,939 Summary of the Invention [Problem to be solved by the invention]

[0007] To increase procedural efficiency, facilitate standardization of care, and improve patient safety, methods and devices are needed that provide sizing guidance during this critical sizing step. [Means for solving the problem]

[0008] The present disclosure provides design, material, manufacturing, and use alternatives suitable for medical devices. An exemplary imaging system for determining the size of an observed object includes a monocular endoscope, an imaging device mounted on the endoscope, at least one light source positioned at a fixed position relative to the imaging device, and a processor configured to accept input from the imaging device and analyze at least one of shadowing effects from the light source, structured illumination provided by the light source, multi-point illumination including the light source, time of flight associated with light pulses from the light source, and lateral chromatic aberration from the light source, wherein the analysis provides a depth determination for the observed object from which the size of the object is inferred.

[0009] Alternatively or additionally to the above embodiments, the system further comprises a tool positionable relative to the imaging device and the light source.

[0010] Alternatively or additionally to any of the above described embodiments, the light source is spaced apart from the imaging device, and the processor is configured to perform a shadow effect analysis of shadows cast from the tool.

[0011] Alternatively or additionally to any of the above embodiments, a reference point on the tool is positioned at a first distance from the light source, and the shadow effect analysis includes determining a ratio of the first distance to a second distance from the light source to a surface on which the shadow of the tool is visible.

[0012] Alternatively or additionally to any of the above embodiments, a third distance from the light source to the object is determined, and a fourth distance from the light source to a surface on which a shadow of the object is visible is also determined, and the processor is further configured to determine the size of the object based on this ratio.

[0013] Alternatively or additionally to any of the above embodiments, at least one light source projects a predetermined light pattern onto the observed object, and the processor is configured to perform structured lighting analysis by comparing a size and curvature of the pattern measured on the object with the predetermined light pattern, and to determine a size of the object based on the comparison.

[0014] Alternatively or additionally to any of the above embodiments, the at least one light source that projects the light pattern includes an optical fiber positioned between the endoscope and the object.

[0015] Alternatively or additionally to any of the above embodiments, the optical fiber includes a first optical fiber that provides uniform illumination and a second optical fiber that provides a predetermined light pattern.

[0016] Alternatively or additionally to any of the above embodiments, the at least one light source comprises at least first and second light sources that are separately controlled.

[0017] Alternatively or additionally to any of the above embodiments, the first and second light sources provide light of different colors.

[0018] Alternatively or additionally to any of the above embodiments, the system further comprises a third light source, wherein the first, second and third light sources are separately controlled red, green and blue LEDs.

[0019] Alternatively or additionally to any of the above embodiments, the system further comprises a time-of-flight sensor, and the at least one light source is configured to emit light pulses.

[0020] Alternatively or additionally to any of the above described embodiments, the sensor and light source are positioned through the endoscope and extend distally to the distal end of the endoscope.

[0021] Alternatively or additionally to any of the above embodiments, the light source includes lateral chromatic aberration over the wavelength range being detected by the imaging device, and the processor is configured to measure the wavelength shift and correlate it with the distance to the object.

[0022] An exemplary method for determining the size of an object viewed through a monocular endoscope includes capturing an image of the object using an imaging device mounted on the endoscope and using at least one light source positioned at a fixed position relative to the imaging device; analyzing the image using a processor configured to perform one or more of a shadow effect analysis, a structured light analysis, a multi-point illumination analysis, a time-of-flight analysis, and a lateral chromatic aberration analysis; and determining the size of the object based on the one or more analyses.

[0023] Alternatively or additionally to any of the above embodiments, the analysis is shadowing, and the method further comprises positioning a tool of known size at a first distance through the endoscope; illuminating the tool with at least one light source to generate a shadow of the tool; and measuring a second distance from the at least one light source to a surface on which the shadow of the tool is visible, wherein the shadowing analysis includes determining a ratio of the first distance to the second distance; and using this ratio to determine the size of the object based on a measured third distance to the object and a fourth distance to the shadow of the object.

[0024] Alternatively or additionally to any of the above embodiments, the analysis is a structured light analysis, and the method further comprises projecting a predetermined light pattern onto an imaging field of view including the object using at least one light source, and measuring the resulting pattern on the object, wherein the structured lighting analysis comprises comparing a size and curvature of the pattern measured on the object with the predetermined light pattern, and determining a size of the object based on the comparison.

[0025] Alternatively or additionally to any of the above embodiments, the analysis is a multi-point lighting analysis, and the at least one light source includes at least first and second light sources that are separately controlled, and the method further comprises separately illuminating the object using the first and second light sources.

[0026] Alternatively or additionally to any of the above embodiments, the analysis comprises time-of-flight analysis, the at least one light source is configured to emit discrete light pulses, and the endoscope further comprises a time-of-flight sensor.

[0027] Alternatively or additionally to any of the above embodiments, the analysis includes a leveraged optical aberration analysis, wherein the at least one light source includes lateral chromatic aberration over a wavelength range detected by the imaging device, and analyzing the image includes measuring a wavelength shift and correlating it with a distance to the object.

[0028] The above summary of some embodiments, aspects, and / or examples is not intended to describe each embodiment or every implementation of the present disclosure. The figures and the detailed description that follow more particularly exemplify these embodiments.

[0029] The present disclosure can be more fully understood from the following detailed description of various embodiments in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a perspective view of an exemplary endoscope assembly. [Figure 2] 1 is a schematic diagram of a system for determining the size of an object according to an example of the present disclosure; [Figure 3] 1 is a flow diagram of a method for determining the size of an object viewed through an endoscope using shadow analysis. [Figure 4] 1 is a flow diagram of a method for determining the size of an object viewed through an endoscope using structured lighting analysis.

[0031] While aspects of the present disclosure are susceptible to various modifications and alternative forms, specifics thereof are shown by way of example in the drawings and will be described in detail below. It should be understood, however, that the intention is not to limit aspects of the present disclosure to the particular embodiments illustrated. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0032] Unless a different definition is provided for the following defined terms in the claims or elsewhere in this specification, the following definitions shall apply.

[0033] All numerical values ​​herein are assumed to be modified by the term "about," whether explicitly stated or not. The term "about" in the context of numerical values ​​generally refers to a range of numbers that one of ordinary skill in the art would consider equivalent to the recited value (e.g., having the same function or result). In many instances, the term "about" may include numbers that are rounded to the nearest significant figure. Other uses of the term "about" (e.g., in contexts other than numerical values) can be assumed to have the ordinary and customary definition of the term that is understood in the context of this specification and is not inconsistent therewith, unless otherwise specified.

[0034] The recitation of numerical ranges by endpoints includes all numbers within that range, inclusive of the endpoints (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, and 5). While some suitable dimensions, ranges, and / or values ​​for various components, features, and / or specifications are disclosed, one of ordinary skill in the art, inspired by the present disclosure, will understand that desirable dimensions, ranges, and / or values ​​may deviate from those explicitly disclosed.

[0035] As used in this specification and claims, the singular forms "a," "an," and "the" include singular and plural referents unless the context clearly dictates otherwise. As used in this specification and claims, the term "or" is generally used to include "and / or" unless the context clearly dictates otherwise. For ease of understanding, certain features of the present disclosure may be described in the singular; however, it should be noted that these features may be plural or repeated within embodiments of the present disclosure. Each instance of these features may include and / or be encompassed by the singular disclosure unless expressly stated to the contrary. For purposes of brevity and clarity, not every element of the present disclosure is shown in each figure or discussed in detail below. However, it will be understood that when more than one element is present, the following discussion may apply equally to any and / or all of these elements, unless expressly stated to the contrary. Additionally, for purposes of clarity, not every instance of some element or feature may be shown in each figure.

[0036] It should be noted that references herein to "embodiments," "some embodiments," "other embodiments," etc., indicate that the described embodiments may include particular features, structures, or characteristics, but that not all embodiments necessarily include these particular features, structures, or characteristics. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when particular features, structures, or characteristics are described with respect to an embodiment, it is believed that those skilled in the art will recognize that these particular features, structures, or characteristics can also be implemented with respect to other embodiments, unless expressly stated to the contrary. That is, as those skilled in the art will understand, the various individual elements described below, even if not specified in specific combinations, are still believed to be combinable or configurable with each other to form other or additional embodiments or to complement and / or extend the described embodiments.

[0037] For purposes of clarity, certain distinguishing numerical nomenclature (e.g., first, second, third, fourth, etc.) may be used throughout the specification and / or claims to name and / or distinguish among various features described and / or claimed. It should be understood that this numerical nomenclature is not intended to be limiting, but is merely exemplary. In some embodiments, variations and departures from previously used numerical nomenclature may be made for purposes of brevity and clarity. That is, a feature designated as a "first" element may later be designated a "second," "third," etc., or may be omitted entirely, and / or a different feature may be designated the "first" element. The meaning and / or designation in each instance will be apparent to one of ordinary skill in the art.

[0038] The detailed description is intended to illustrate, rather than limit, the present disclosure. Those skilled in the art will recognize that the various elements described can be arranged in various combinations and configurations without departing from the scope of the present disclosure. The detailed description shows exemplary embodiments of the present disclosure.

[0039] Current ureteroscope systems are monocular and therefore lack direct data regarding the depth of objects within the scene. Some systems estimate object size by overlaying a simple grid on the image displayed on the screen and using the size of this grid to estimate the size of the object. However, such analysis does not consider the depth of the object from the light source and imaging device. Even when a size grid is provided using objects of known size within the field of view, depth remains an unknown variable that can significantly change the size of the object as viewed relative to the grid. For example, a stone located 5 mm distal to the light source and imaging device may appear significantly smaller than a similarly sized stone located 1 mm from the light source and imaging device. The grid added to the image on the screen may cause two stones to appear different sizes when, in fact, they are the same physical size.

[0040] An exemplary endoscope assembly 10 is illustrated in FIG. 1. The endoscope 10 may be any of several types of endoscopes or related medical devices, typically identified by the particular anatomical structure desired to be reached. For example, the endoscope 10 may be a bronchoscope, colonoscope, duodenoscope, esophagoscope, guide tube, introducer (with or without visual or visualization capabilities), or any other type of endoscope or related medical device. The endoscope 10 may include a handpiece 12 and an elongated shaft 14 extending distally therefrom to a distal tip 18. The shaft 14 may include a lumen defining a working channel 16 extending therethrough from a distal end 19 near the distal tip 18 to an access port 20, which may be disposed within the handpiece 12 or another portion of the endoscope 10. While FIG. 1 depicts the endoscope 10 with a single working channel 16, it will be appreciated that in other embodiments, the endoscope 10 may include multiple working channels as desired. The working channel can be configured to accommodate the passage of various surgical instruments, including, but not limited to, imaging devices and tools for irrigation, vacuum aspiration, biopsy, and drug delivery.

[0041] The handpiece 12 may include one or more controls 22, such as rotation knobs, that can be used to control movement of the distal tip 18 of the shaft 14 during surgery. For example, a first rotation knob 22a may control up-and-down movement or deflection of the distal tip 18 of the shaft 14, while a second rotation knob 22b may control left-and-right movement or deflection of the distal tip 18 of the shaft 14. The handpiece 12 may further include one or more buttons 24 that can be used to activate aspiration or delivery of fluids, such as air, saline, and / or water, through the lumen of the endoscope 10 or to perform other functions as desired. Additionally, the handpiece 12 may include an optical cable 26 connected to an external light source (not shown).

[0042] FIG. 2 illustrates a system 100 for determining the size of an object according to one example of the present disclosure. The system may include a processor 102 operably coupled to a display 104. The processor 102 may be coupled to an energy source 106 and the endoscope 10. The processor 102 may generally be configured to accept information from the system and system components and process it according to various algorithms. The energy source 108 may include an optical energy source, such as a holmium (Ho) laser source, a carbon dioxide (CO) laser source, an argon laser source, a diode laser source, or another suitable laser source. In some examples, one or more high-power LEDs may be used in place of the laser source. In some examples, a high-intensity pulsed light source may be used in place of the laser source. The optical energy source may be mounted at the distal end 19 of the endoscope 10 or on a catheter or other suitable elongated member.

[0043] In some examples, one or more optical fibers 20 may extend from the distal tip 19 of the endoscope 10 through the lumen 17 and be configured to deliver optical energy from the energy source 106 to the treatment location. In other examples, the optical fibers 20 may be fixed to the distal tip 19. An imaging device 40 may also be disposed at the distal tip 19 of the endoscope 10. The imaging device 40 may include any suitable device configured to provide images to the processor 102 for display on the display 104, such as a device including a CMOS imaging sensor or other solid-state device or camera and one or more glass or polymer lenses that generates electronic image signals representing an image of tissue or other objects in front of the imaging device 40. The imaging device 40 may be a low-light-sensitive, low-noise video VGA, CMOS, color imager, or a higher resolution sensor such as SVGA, SXGA, or XGA. In some examples, the imaging device 40 and / or the light source may be fixed relative to one another.

[0044] Depth data is available in stereoscopic systems, which generate multiple views of the same scene from different viewpoints separated by a known distance. However, monocular endoscopes cannot automatically generate scene depth. Five techniques can be used to enhance monocular depth estimation to generate distance data similar to stereoscopic systems: shadow analysis, structured illumination, multi-point illumination, time-of-flight sensors, and exploiting optical aberrations.

[0045] Shading analysis Shadow analysis recognizes that the scene illumination source is very close to, but not exactly coincident with, the imaging device, as shown in Figure 2. When the illumination source remains in a fixed position relative to the imaging device, the distance to an object of known size and its shadow can be used in the analysis. In particular, the distance D1 from the illumination source to an object of known size and / or depth in the field of view can be measured, and the distance D2 from the illumination source to the surface on which the object's shadow is visible can be measured. The shadowing effect can be expected to be proportional to the ratio D1 / D2. The depth of the shadow location can then be calculated directly. Since the tool dimensions are known, the portion of the tool that casts the shadow (the reference point) can be inferred, and the depth of this portion in the field of view can be measured to infer its size.

[0046] A reference point or known portion of the tool is placed at a first distance D1 from the light source, and then D2 is measured to the surface on which the shadow of the tool is seen. A ratio of D1 / D2 is calculated. This ratio of D1 / D2 provides a depth estimate that can be used to calculate the size of an unknown object in the field of view. Distances D1 and D2 are measured for the unknown object, and a ratio is calculated and used to estimate the size of the unknown object. A third distance from the light source to the unknown object and a fourth distance from the light source to the surface on which the shadow of the unknown object is visible are determined, and the system's processor determines the size of the unknown object based on this ratio.

[0047] One method 200 for determining the size of an object viewed through a monocular endoscope is illustrated in FIG. 3. An object of known size, such as a tool, can be placed in the endoscope, e.g., the working channel, at a location at a known depth relative to the edge of the tool. The tool is identified in an image generated by an imaging device (step 202). Next, a user searches for and detects the shadow edge projected from the tool onto the tissue surface from the tool edge (step 204). Corresponding points on the tool and the shadow are found (step 206), and the shadow is measured (step 208). The width of the shadow is used to assign depth to points on the shadow edge (step 210). If a relative depth map is available (step 212), the depth, size, and other characteristics of the image are inferred (step 214). If a relative depth map is not available, the dimensions of surface features at the shadow edge are inferred (step 216).

[0048] To convert information about the depth of a tool's shadow into the depth (and therefore size) of an object such as a kidney stone, relative depth within a scene can be established by means such as epipolar measurements using successive aligned images and other lighting cues. These techniques can produce a depth map for the scene, but do not have any absolute scale; thus, while a first reference point may be determined to be twice as far from the light source as a second reference point, they cannot determine in an accurate manner whether these points are 4 mm and 2 mm, 8 mm and 4 mm, or any such combination. By reliably measuring the depth of a single point that is both in the scene and in the relative depth map, scale can be determined and absolute distance and size can be inferred relative to all other features in the depth map.

[0049] In some examples, the tool can have markings, such as piecewise measurement markings or geometric patterns, that can be used to determine D1. See, for example, U.S. Provisional Patent Application No. 63 / 155,939, filed March 3, 2021, entitled "Measurement Markings in Distal Tip Imaging Field of View," the entire contents of which are incorporated by reference.

[0050] The use of tool shadows also maximizes the ratio between depths depending on the tool's proximity to the light source, thereby maximizing the signal-to-noise ratio (SNR) of the depth estimation. Some shadows may be barely detectable and can benefit from estimation at sub-pixel resolution, and therefore image processing techniques can be employed to enhance light-dark boundary estimation, including multi-frame sampling with alignment.

[0051] structured lighting Structured illumination techniques are known in optical metrology and often involve the projection of a known pattern or grating into the imaging field of view, where the pattern is projected from a position at a known offset from the imaging device recording the scene, thereby providing a stereoscopic baseline. Distance and topography can be calculated by comparing the size and curvature of the measurement pattern to the known pattern.

[0052] One method 300 for determining the dimensions of a target object is illustrated in FIG. 4. In step 302, an illumination pattern is projected onto the target object from a known position. In step 304, projected features are detected in the camera image. In step 306, the depth to each feature (epipolar) is calculated. In step 308, it is determined whether the target object has a known depth. If yes, the dimensions of the target object are estimated (step 310), and then the camera image is annotated with size information (step 312). If the target object did not have a known depth (step 308), the next question is whether a relative depth map is available (step 314). If yes, the method proceeds to step 310. If no, the position and / or orientation of the pattern is adjusted (step 316), and the method returns to step 302.

[0053] This approach presents unique challenges in ureteroscopy, where both the illumination and imaging systems are severely constrained and very close. The optics required to project a discernible pattern may be impractical to integrate directly into the ureteroscope. Therefore, an additional tool, potentially interposed between the scope and the scene through the scope's working channel, can serve to generate a discernible illumination pattern that indicates depth, similar to the shadow analysis described above. For example, optical fibers can be used as an additional tool to project a predetermined discernible illumination pattern onto the field of view. In this case, the system processor performs structured illumination analysis by comparing the size and curvature of the pattern measured on the unknown object with a predetermined light pattern, and determines the size of the unknown object based on this comparison. In some examples, the predetermined light pattern can be generated using first and second optical fibers.

[0054] In another example, lights with two known wavelengths positioned at a known separation distance from the imaging device can act as a structured light source. Two or more light-emitting diodes (LEDs) of the same or different wavelengths, LEDs using collimated laser beams, or pilot bursts of a laser can be used. Yet another example involves the use of two optical fibers, one with uniform illumination for standard visualization and the second with structured light for physical depth measurement, where the illumination is generated within the scope itself. When multiple LEDs or optical fibers are used, they can be controlled separately and individually. Particularly when using LEDs, they can be turned on and off separately, allowing the user to vary the color of the light to achieve a desired color by mixing light from different colored LEDs.

[0055] Multi-point lighting In conjunction with either or both of the above techniques, a second, separately controllable light source can be incorporated onto the scope with maximum practical light source separation, thereby providing additional depth information. The shading effects described above can be determined for each light source. Distinguishing between shading from various light sources can be achieved by timing (potentially by toggling the light source on and off at intervals synchronized with the camera frame rate) or color; for example, using complementary red-blue light sources could produce detectable color fringes in the image, the measurement of which could enable depth estimation. Similarly, individual red, green, and blue (RGB) LEDs, e.g., micro-LEDs, on the tip of the scope could produce balanced illumination with subtle depth cues that a computer could interpret without distracting the user. For example, white light could be achieved by combining the outputs of red, green, and blue LEDs.

[0056] Time-of-Flight Sensor The time-of-flight principle is a method for measuring the distance between a sensor and an object and is based on the time difference between the emission of a light signal and its return to the sensor after being reflected by the object. A time-of-flight sensor requires a light source capable of emitting a light pulse, which can be provided by a laser or an LED. The system can either modify the LED to generate the light pulse or utilize a lithotripsy system. Instead of mounting the sensor on the scope, the sensor and light source can be interposed between the scope and the scene. In this way, the sensor and light source can extend distally beyond the distal end of the scope.

[0057] Leverage optical aberrations Endoscopy imaging sensors can use RGB patterns in a tiled (typically Bayer) layout. These adjacent portions of the sensor are reconfigured into a single pixel that represents a single collocated location in (RGB) color. This configuration works very well, especially if the optics are designed to minimize lateral chromatic aberration, which can result in ray shifts within the image plane. Alternatively, the optics can be designed to intentionally include lateral chromatic aberration across the wavelength range being detected. The ray shifts at individual red, green, and blue sensors in the raw image can be measurable and correlated to physical distance. Chromatic aberration can be based on the material used in the lens. In some cases, flint glass can be used. Flint glass contains some lead and has particularly high chromatic aberration (represented by an Abbe number less than 50) and a high refractive index (usually greater than 1.55). Some types of flint glass are F, LF, SF, KF, BaF, and BaLF. Lead-free flint glass typically has an "N" before the glass name. For example, N-SF8. These shifts may appear in the raw pixels if they are significant compared to the pixel size. When combined with glass grades that have different absorption in this portion of the visible spectrum (e.g., N-SF57) and / or when combined with collimated multispectral illumination, it is conceivable that individual R, G, and B images could be generated, aligned, and used to estimate any differences between distances.

[0058] One or more of the above techniques can be used in an imaging system to provide a more accurate estimation of the physical size of an object observed through the monocular endoscope. The imaging system can include a monocular endoscope, an imaging device mounted on the endoscope, at least one light source positioned at a fixed position relative to the imaging device, and a processor configured to accept input from the imaging device and analyze one or more of shadowing effects from the light source, structured illumination provided by the light source, multi-point illumination including the light source, time of flight associated with light pulses from the light source, and lateral chromatic aberration from the light source. The analysis provides a depth determination for the observed object.

[0059] It should be understood that the present disclosure is in many respects merely illustrative. Changes may be made in details, particularly in matters of shape, size, and arrangement of steps, without exceeding the scope of the present disclosure. This may include, where appropriate, the use in other embodiments of any of the features of one illustrative embodiment. The scope of the present disclosure is, of course, defined by the language in which the claims are expressed. [Explanation of symbols]

[0060] 200 One way to determine the size of an object 202 Identifying tools in an image 204 Searching and detecting shadow edges from tool edges 206 Finding corresponding points on the tool and on the shadow 208 Shadow measurement stage

Claims

1. 1. An imaging system for determining the size of an observed object, comprising: A monocular endoscope, an imaging device mounted on the endoscope; at least one light source positioned at a fixed position relative to the imaging device; a processor configured to accept input from the imaging device having a field of view and to analyze at least two of shadowing effects from the light source, structured illumination provided by the light source, multi-point illumination including the light source, time of flight associated with light pulses from the light source, and lateral chromatic aberration from the light source; Equipped with the analysis provides a depth determination for the observed object within the field of view, from which a size of the object is inferred; The imaging system, wherein the processor is configured to combine results from at least two of the analyses to generate a composite depth map.

2. The system of claim 1 further comprising a tool positionable relative to the imaging device and light source.

3. The system of claim 2 , wherein the light source is spaced apart from the imaging device, and the processor is configured to perform a shadow effect analysis of a shadow cast from the tool.

4. a reference point on the tool is positioned at a first distance from the light source; The system of claim 3 , wherein the shadow effect analysis includes determining a ratio of the first distance to a second distance from the light source to a surface on which the tool shadow is visible.

5. a third distance from the light source to the object is determined, and a fourth distance from the light source to a surface on which a shadow of the object is visible is also determined; 5. The system of claim 4, wherein the processor is further configured to determine the size of the object based on the third distance and the fourth distance using a ratio of the first distance to the second distance.

6. 6. The system of claim 1, wherein the at least one light source projects a predetermined light pattern onto the observed object, and the processor is configured to perform structured lighting analysis by comparing a size and curvature of a pattern measured on the object with the predetermined light pattern, and to determine the size of the object based on the comparison.

7. The system of claim 6 , wherein the at least one light source that projects a light pattern includes an optical fiber positioned between the endoscope and the object.

8. 8. The system of claim 7, wherein the optical fibers include a first optical fiber that provides uniform illumination and a second optical fiber that provides the predetermined light pattern.

9. The system of any one of claims 1 to 8, wherein the at least one light source comprises at least first and second light sources that are separately controlled.

10. The system of claim 9 , wherein the first and second light sources provide light of different colors.

11. further comprising a third light source; 11. The system of claim 10, wherein the first, second, and third light sources are separately controlled red, green, and blue LEDs.

12. further comprising a time-of-flight sensor; The system of any one of claims 1 to 11, wherein the at least one light source is configured to emit pulses of light.

13. The system of claim 12 , wherein the sensor and light source are disposed through the endoscope and extend distally of the distal end of the endoscope.

14. the light source includes lateral chromatic aberration over the wavelength range being detected by the imaging device; A system according to any preceding claim, wherein the processor is configured to measure a wavelength shift and correlate it to a distance to the object.

Images