Offset illumination of scenes using multiple emitters in hyperspectral, fluorescence, and laser mapping imaging systems

Abstract

Offset illumination using multiple emitters in a fluorescence imaging system is described. The system includes an emitter to emit pulses of electromagnetic radiation and an image sensor including an array of pixels to sense reflected electromagnetic radiation. The emitter includes a first emitter and a second emitter to emit electromagnetic radiation of different wavelengths. The system causes at least a portion of the pulses of electromagnetic radiation emitted by the emitter to include one or more of a hyperspectral emission, a fluorescence emission, or a laser mapping mode.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to digital imaging, and in particular to hyperspectral imaging, fluorescence imaging, and / or laser mapping imaging in light deficient environments. BACKGROUND

[0002] Advances in technology have provided advances in medical imaging capabilities. Endoscopes can be used to view inside the body and examine the interior of organs or cavities of the body. Endoscopes are used to investigate a patient's symptoms, confirm a diagnosis, or provide medical treatment. Medical endoscopes can be used to view a variety of body systems and portions, such as the gastrointestinal tract, respiratory tract, urinary tract, abdominal cavity, etc. Endoscopes can also be used for surgical procedures, such as orthopedic surgery, procedures performed on joints or bones, procedures performed on the nervous system, procedures performed within the abdominal cavity, etc.

[0003] In some cases of endoscopic imaging, it can be advantageous or necessary to view a color space. A digital color image includes at least three layers or "color channels" that cumulatively form an image with a range of hues. Each of the color channels measures the intensity and hue of a band of the light spectrum. Typically, a digital color image includes color channels for the red, green, and blue bands of the spectrum (this can be referred to as a red-green-blue or RGB image). Each of the red, green, and blue channels includes luminance information for the red, green, or blue band of the spectrum. The luminance information for the individual red, green, and blue layers is combined to generate a color image. Because a color image is composed of individual layers, conventional digital camera image sensors include an array of color filters that allow red, green, and blue visible wavelengths of light to hit selected pixel sensors. Each individual pixel sensor element is sensitive to either red, green, or blue wavelengths, and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image. The at least three different types of pixel sensors consume a large amount of physical space, such that a complete array of pixels cannot fit in the small distal tip of an endoscope.

[0004] Because conventional image sensors cannot fit in the distal tip of an endoscope, the image sensor is traditionally located in the handpiece unit of the endoscope, which is held by the endoscope operator and is not placed within the body cavity. In such endoscopes, light is transmitted along the length of the endoscope from the handpiece unit to the distal tip of the endoscope. This configuration has significant limitations. Endoscopes with this configuration are delicate, and can easily become misaligned or damaged when they are bumped or impacted during routine use. This can significantly degrade the quality of the images, and the endoscope needs to be frequently repaired or replaced.

[0005] Conventional endoscopes with image sensors placed in the handpiece unit are also limited to capturing only color images. However, in some implementations, it can be desirable to capture images with fluorescence data, hyperspectral data, and / or laser mapping data in addition to color image data. Fluorescence imaging captures light emitted by substances that have absorbed electromagnetic radiation and “glow” when they relax at their emission wavelengths. Hyperspectral imaging can be used to identify different materials, biological processes, and chemical processes by partitioning the emitted electromagnetic radiation and evaluating the spectral response of the material. Laser mapping imaging can capture the surface shape of objects and landscapes and measure distances between objects within a scene. Laser mapping imaging can also encompass tool tracking, where distances and / or dimensions of tools within a scene can be tracked relative to each other, relative to the imaging device, and / or relative to structures within the scene. In some implementations, it can be desirable to use one or more of fluorescence imaging, hyperspectral imaging, and / or laser mapping imaging in combination when imaging a scene.

[0006] However, the application of fluorescence imaging techniques, hyperspectral imaging techniques, and laser mapping techniques known in the art generally require highly specialized equipment that can not be available for a variety of applications. Moreover, such techniques provide limited views of the environment and must generally be used in conjunction with multiple separate systems and multiple separate image sensors made sensitive to particular bands of electromagnetic radiation. It is therefore desirable to develop an imaging system that can be used in a space-constrained environment to generate fluorescence imaging data, hyperspectral imaging data, and / or laser mapping imaging data.

[0007] In view of the foregoing, described herein are systems, methods, and devices for fluorescence imaging, hyperspectral imaging, and laser mapping imaging in light deficient environments. Such systems, methods, and devices can provide multiple data sets for identifying key structures within the body and providing accurate and valuable information about the body cavity. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present disclosure is described in reference to the following drawings, wherein like numerals are used to refer to like elements throughout. The advantages of the present disclosure will become better understood in view of the following description taken in conjunction with the accompanying drawings wherein:

[0009] FIG. 1 is a schematic of a system for digital imaging in light deficient environments with paired emitters and pixel arrays;

[0010] FIG. 2 is a system for providing illumination to a light deficient environment for endoscopic imaging;

[0011] FIG. 2A is a schematic of complementary system hardware;

[0012] FIGS. 3A-3D is a graphical representation of the operation cycle of the sensor used to construct the exposure frame;

[0013] FIG. 4A is a graphical representation of the operation of an embodiment of the electromagnetic emitter;

[0014] FIG. 4B is a graphical representation of varying the duration and magnitude of the emitted electromagnetic pulse to provide exposure control;

[0015] FIG. 5 is a combination of the operation cycle of the sensor of FIGS. 3A-4B the electromagnetic emitter and the emitted electromagnetic pulse, showing the imaging system during operation, an embodiment of the present disclosure;

[0016] FIG. 6A is a schematic of a method for recording a video with full-spectrum light over a time period of t(0) to t(l);

[0017] FIG. 6B is a schematic of a process for recording a video by pulsing partial-spectrum light over a time period of t(0) to t(l);

[0018] FIGS. 7A-7E is a schematic showing a method for recording video frames of both full-spectrum light and partitioned-spectrum light over a time interval;

[0019] FIG. 8 is a graphical display of the delay or jitter between a control signal and the emitter emitting electromagnetic radiation;

[0020] FIG. 9 is a cross-sectional view of a fiber bundle comprising a central fiber and a plurality of surrounding fibers;

[0021] FIG. 10 is a graphical display of a top-hat profile and a Gaussian profile for sending electromagnetic radiation to a fiber bundle;

[0022] FIG. 11 is a side view showing the output of electromagnetic radiation (light) from a fiber bundle compared to the field of view of a camera;

[0023] FIG. 12 is a side view showing the output of electromagnetic radiation from a fiber bundle, where the end of a single fiber is intended to provide a more uniform distribution of electromagnetic radiation;

[0024] FIG. 13 is a side view showing the output of electromagnetic radiation from a fiber bundle, where the fiber bundle comprises plastic fibers and glass fibers coupled near the output;

[0025] FIG. 14is a side view showing output of electromagnetic radiation from a fiber bundle including a diffuser located near the output;

[0026] FIG. 15 is a schematic flow diagram of a method for driving emitters to illuminate a scene according to a dithering specification;

[0027] FIG. 16 is a schematic flow diagram of a method for providing electromagnetic radiation to an imaged scene in a light deficient environment;

[0028] FIGS. 17A-17C shows a light source having multiple emitters;

[0029] FIG. 18 shows a single optical fiber outputting at an output via a diffuser to illuminate a scene in a light deficient environment;

[0030] FIG. 19 shows a portion of an electromagnetic spectrum divided into multiple different sub-spectra that can be emitted by emitters of a light source according to the principles and teachings of the present disclosure;

[0031] FIG. 20 is a schematic diagram showing emission and readout timing for generating an image frame comprising multiple exposure frames produced by different partitions of pulsed light;

[0032] FIG. 21 shows an imaging system including a single cut filter for filtering wavelengths of electromagnetic radiation;

[0033] FIG. 22 shows an imaging system including multiple cut filters for filtering wavelengths of electromagnetic radiation;

[0034] FIG. 23 shows an exemplary laser mapping pattern that can be pulsed by an imaging system;

[0035] FIG. 24A and FIG. 24B shows an implementation having multiple pixel arrays for producing a three-dimensional image according to the principles and teachings of the present disclosure;

[0036] FIG. 25A and FIG. 25B shows perspective and side views, respectively, of an implementation of an imaging sensor built on multiple substrates, with multiple pixel columns forming a pixel array located on a first substrate and multiple circuit columns located on a second substrate, with electrical connections and communications between a pixel column and its associated or corresponding circuit column shown; and

[0037] FIG. 26A and FIG. 26BPerspective and side views of a particular implementation of an imaging sensor having a plurality of pixel arrays for producing three-dimensional images are shown, respectively, wherein the plurality of pixel arrays and image sensor are built on a plurality of substrates. DETAILED DESCRIPTION

[0038] Disclosed herein are systems, methods, and apparatus for digital imaging that can be primarily applicable to medical applications such as medical endoscopic imaging. One embodiment of the disclosure is an endoscopic system for hyperspectral, fluorescent, laser mapping, and color imaging in light deficient environments. Such methods, systems, and computer-based products disclosed herein provide imaging or diagnostic capabilities for medical robotic applications such as the use of robots for performing imaging procedures, surgical procedures, and the like.

[0039] One embodiment of the disclosure is an endoscopic imaging system including an emitter for emitting a pulse of electromagnetic radiation to illuminate a scene. The emitter includes a plurality of laser beams (which can alternatively be referred to herein as independent“emitters” that make up the overall emitter) that can be operated independently of one another and emit different wavelengths of electromagnetic radiation. The plurality of laser beams making up the emitter can each be configured to pulse electromagnetic radiation at different partitions or wavelengths of the electromagnetic spectrum. The pulse of electromagnetic radiation can be pulsed to a fiber bundle, and the fiber bundle can then carry the pulsed electromagnetic radiation to a distal end of an endoscope to illuminate the scene. This implementation using a plurality of laser beams raises issues when the electromagnetic radiation reaches the fiber bundle. For example, a single fiber within the fiber bundle can receive different wavelengths of light, different power levels of light, or can receive more or less light than other single fibers. This can result in non-uniform illumination of the scene.

[0040] In accordance with the foregoing, embodiments of the present disclosure include an intervening optical element and / or one or more dichroic mirrors. The intervening optical element and the one or more dichroic mirrors can be used in combination to provide uniform light to a fiber bundle. The intervening optical element can be positioned between an emitter and a fiber bundle. The intervening optical element can include, for example, a diffuser, a mixing rod, a lens, or some other optical component for facilitating a uniform light mixture. In one embodiment, there is a dichroic mirror for each laser beam of an emitter. A first dichroic mirror for a first laser beam can be configured to reflect electromagnetic radiation pulsed by the first laser beam at a certain wavelength. A second dichroic mirror for a second laser beam can be configured to reflect electromagnetic radiation pulsed by the second laser beam at a certain wavelength, and so on. Thus, a dichroic mirror can be configured to reflect electromagnetic radiation pulsed by a certain laser beam and be transparent to other wavelengths of electromagnetic radiation that can be pulsed by other laser beams of an emitter. In one embodiment, the dichroic mirrors are angled and positioned within an endoscope system such that electromagnetic radiation pulsed by an emitter hits the dichroic mirror and then changes direction that is pulsed into a fiber bundle. The fiber bundle can then carry the light through the endoscope system to illuminate a scene. This system for offset illumination enables the use of multiple different lasers, laser beams, or emitters without the risk of non-uniform light illuminating a scene or being transmitted through a fiber bundle.

[0041] Conventional endoscopes are designed such that the image sensor is placed at the proximal end of the device within the handpiece unit. This configuration requires the incident light to travel the length of the endoscope through precisely coupled optical elements. Precise optical elements can not be easily aligned during normal use, and this can result in image distortion or image loss. Embodiments of the present disclosure place the image sensor within the spatially constrained environment in the distal end of the endoscope itself. This provides greater optical simplicity when compared to specific implementations known in the art. However, the acceptable solution to this approach is by no means simple and raises its own series of engineering challenges.

[0042] When the overall size of the image sensor is minimized such that the image sensor can fit within the distal tip of an endoscope, there can be a significant loss of image quality. The area of the pixel array of the image sensor can be reduced by reducing the number of pixels and / or reducing the sensing area of each individual pixel. Each of these modifications impacts the resolution, sensitivity, and dynamic range of the resulting image. Traditional endoscopic imaging systems aim to sense steady broadband illumination and provide color information through a segmented pixel array, such as a Bayer pattern array. In view of the deficiencies associated with segmented pixel arrays, alternative systems and methods are disclosed herein that use a monochrome (which can be referred to as “color agnostic”) pixel array that does not include individual pixel filters. In the embodiments disclosed herein, color information is provided by pulsing the emitter with different wavelengths of electromagnetic radiation. The pulsed imaging systems disclosed herein can generate color images that have hyperspectral, fluorescent, and / or laser mapping imaging data layered on top.

[0043] In one embodiment, color information is determined by capturing independent exposure frames in response to pulses of different wavelengths of electromagnetic radiation. The alternative pulses can include red, green, and blue wavelengths for generating an RGB image frame composed of a red exposure frame, a green exposure frame, and a blue exposure frame. In alternative implementations, the alternative pulses can include a luminance (“Y”) pulse, a red chrominance (“Cr”) pulse, and a blue chrominance (“Cb”) pulse of light for generating a YCbCr image frame composed of luminance data, red chrominance data, and blue chrominance data. The color image frame can also include data from a hyperspectral exposure frame, a fluorescent exposure frame, and / or a laser mapping exposure frame layered on top of the RGB or YCbCr image frame. The hyperspectral pulse can be an emission of electromagnetic radiation that elicits a spectral response from an object. The hyperspectral exposure frame can include an indication of the location of the object that emitted the spectral response. The fluorescent pulse can be electromagnetic radiation of a fluorescent excitation wavelength for causing a reagent to fluoresce. The fluorescent exposure frame can include an indication of fluorescent reagents within the scene. The laser mapping pulse can include one or more pulses for measuring distances or dimensions within the scene, tracking the presence and location of a tool in the scene, generating a three-dimensional topological map of the scene, etc. The alternating wavelengths of the pulsed electromagnetic radiation allow for the utilization of a full pixel array and avoids artifacts elicited by Bayer pattern pixel arrays.

[0044] In some cases, it is desirable to generate endoscopic imaging with multiple data types or multiple images layered on top of each other. For example, it can be desirable to generate a color (RGB or YCbCr) image that also includes hyperspectral imaging data, fluorescence imaging data, and / or laser mapping imaging data layered on top of the color image. A layered image of this nature can enable a medical practitioner or computer program to identify high-accuracy dimensions and three-dimensional topological structures of key body structures and further identify distances between tools and other structures within a light deficient environment based on laser mapping data. In the past, this would require the use of multiple sensor systems, including an image sensor for color imaging and one or more additional image sensors for hyperspectral imaging, fluorescence imaging, or laser mapping imaging. In such systems, the multiple image sensors would have multiple types of pixel sensors each sensitive to a different range of electromagnetic radiation. In systems known in the art, this includes three separate types of pixel sensors for generating color images, and additional sensors and systems for generating hyperspectral data, fluorescence data, and laser mapping data. These multiple different sensors occupy too much physical space and cannot be located at the distal tip of an endoscope. In systems known in the art, one or more cameras are not placed at the distal tip of the endoscope, but rather in the endoscope handpiece or robotic unit. This raises a number of drawbacks and results in an endoscope that is very fragile. When the fragile endoscope is bumped or impacted during use, the endoscope can be damaged and the image quality is reduced. In view of the foregoing, disclosed herein are systems, methods, and devices for endoscopic imaging in a light deficient environment. The systems, methods, and devices disclosed herein provide a way to employ multiple imaging techniques in a single imaging session while allowing one or more image sensors to be disposed in the distal tip of an endoscope.

[0045] The fluorescence imaging techniques discussed herein can be used in combination with one or more fluorescent agents or dyes. The location of the agent can be identified by an excitation wavelength that emits electromagnetic radiation that causes the agent to fluoresce. The relaxation wavelength emitted by the agent can be read by an image sensor to identify the location of the agent within the scene. Depending on the type of agent used, the location of the agent can also be indicative of the location of key structures, such as certain types of tissue, cancerous cells versus non-cancerous cells, etc.

[0046] The hyperspectral imaging techniques discussed herein can be used to "see through" tissue layers in the foreground of a scene to identify specific types of tissue and / or specific biological or chemical processes. Hyperspectral imaging can be used in a medical setting to quantitatively track the progression of a disease and determine tissue pathology. Additionally, hyperspectral imaging can be used to identify key structures, such as neural tissue, muscle tissue, cancerous cells, etc. In one embodiment, partitions of electromagnetic radiation are pulsed and, in response to the partitions of electromagnetic radiation, data is collected regarding the spectral response of different types of tissue. Data stores of the spectral responses can be generated and analyzed to evaluate a scene and predict what tissues are present within the scene based on the sensed spectral responses.

[0047] The laser mapping imaging techniques discussed herein can be evaluated to generate a three-dimensional landscape map of a scene and calculate distances between objects within the scene. Laser mapping data can be used in conjunction with fluorescence imaging and / or hyperspectral imaging to calculate precise locations and dimensions of key structures. For example, fluorescence and / or hyperspectral imaging can be utilized to identify the location and boundaries of key structures. Precise measurements of the location of the key structures, the dimensions of the key structures, and distances from the key structures to other objects can then be calculated based on the laser mapping data.

[0048] Hyperspectral imaging

[0049] In one embodiment, the systems, methods, and devices disclosed herein provide devices for generating hyperspectral imaging data in light deficient environments. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from conventional cameras that only capture light across three wavelengths based on the visible spectrum that is discernible by the human eye (including red, green, and blue wavelengths) to generate RGB images. Spectral imaging can use any band of wavelengths in the electromagnetic spectrum, including infrared wavelengths, the visible spectrum, the ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various bands of wavelengths.

[0050] Hyperspectral imaging was initially developed for use in mining and geology applications. Unlike normal camera images that provide limited information to the human eye, hyperspectral imaging can identify specific minerals based on the spectral signature of different minerals. Hyperspectral imaging is useful even when captured in aerial images and can provide information about, for example, oil or gas leaks from pipelines or natural wells and their impact on nearby vegetation. This information is collected based on the spectral signature of certain materials, objects, or processes that can be identified by hyperspectral imaging.

[0051] Hyperspectral imaging includes spectroscopy and digital photography. In one embodiment of hyperspectral imaging, the complete spectrum or some spectral information is collected at each pixel in the image plane. The goal of hyperspectral imaging can vary for different applications. In one application, the goal of hyperspectral imaging is to obtain the entire electromagnetic spectrum for each pixel in the image scene. This can enable the finding of certain objects that can not be identifiable under the visible wavelength band. This can enable the precise identification of certain substances or tissues that can not be identifiable under the visible wavelength band. In addition, this can enable the detection of certain processes by capturing an image across all wavelengths of the electromagnetic spectrum.

[0052] In one embodiment of the present disclosure, an endoscope system illuminates a source and pulses electromagnetic radiation for spectral or hyperspectral imaging. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from a conventional camera that only captures light across three wavelengths based on what is resolvable by the human eye (including red, green, and blue wavelengths) to generate an RGB image. Spectral imaging can use any band of wavelengths in the electromagnetic spectrum, including infrared wavelengths, the visible spectrum, the ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can overlay imaging generated based on non-visible bands (e.g., infrared) on top of imaging based on visible bands (e.g., standard RGB images) to provide additional information that can be easily resolvable by a human or computer algorithm.

[0053] Hyperspectral imaging has many advantages compared to conventional imaging. The information obtained by hyperspectral imaging enables a practitioner and / or a computer-implemented program to precisely identify certain tissues or conditions that can not be identifiable with RGB imaging. In addition, hyperspectral imaging can be used during a medical procedure to provide image-guided surgery, enabling a practitioner to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells in contrast to typical healthy cells, identify certain tissues or conditions, identify critical structures, etc. Hyperspectral imaging provides specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0054] In medical applications, hyperspectral imaging can provide certain advantages over conventional imaging. The information obtained by hyperspectral imaging enables a practitioner and / or a computer-implemented program to precisely identify certain tissues or conditions that can lead to a possible misdiagnosis or a less accurate diagnosis using conventional imaging, such as RGB imaging. In addition, hyperspectral imaging can be used during a medical procedure to provide image-guided surgery, enabling a practitioner to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells in contrast to typical healthy cells, identify certain tissues or conditions, identify critical structures, etc. Hyperspectral imaging can provide specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0055] In various applications and implementations of the present disclosure, endoscopic hyperspectral imaging can present advantages over conventional imaging. In medical implementations, endoscopic hyperspectral imaging can allow a medical practitioner or computer-implemented program to distinguish, for example, between neural tissue, muscle tissue, various blood vessels, blood flow direction, etc. Hyperspectral imaging can enable the precise differentiation of atypical cancerous tissue from typical healthy tissue, and thus can enable a medical practitioner or computer-implemented program to distinguish the boundaries of a cancerous tumor during a surgical or research imaging. Additionally, hyperspectral imaging in a light deficient environment as disclosed herein can be combined with the use of reagents or dyes to allow further differentiation of certain tissues or substances. In such implementations, a reagent or dye can fluoresce at a particular band of wavelengths in the electromagnetic spectrum, providing information specific to the purpose of that reagent or dye. The systems, methods, and devices disclosed herein can enable pulsing of any number of wavelength bands, such that one or more reagents or dyes can fluoresce at different times, and further that one or more sections of electromagnetic radiation can be pulsed for hyperspectral imaging in the same imaging session. In certain implementations, this enables the identification or study of multiple medical conditions during a single imaging procedure.

[0056] Fluorescence imaging

[0057] The systems, methods, and devices disclosed herein provide devices for generating fluorescence imaging data in a light deficient environment. The fluorescence imaging data can be used to identify certain substances, tissues, components, or processes within a body cavity or other light deficient environment. In certain implementations, fluorescence imaging is provided to a medical practitioner or computer-implemented program to enable the identification of certain structures or tissues within the body. Such fluorescence imaging data can be overlaid on black and white or RGB images to provide additional information and context.

[0058] Fluorescence is light emitted by a substance that has absorbed light or other electromagnetic radiation. Certain fluorescent substances can "glow" or emit a different color visible to the human eye when they are subjected to ultraviolet light or other wavelengths of electromagnetic radiation. Certain fluorescent materials will stop glowing almost immediately when the source of radiation stops.

[0059] Fluorescence occurs when an orbital electron of a molecule, atom, or nanostructure is excited by light or other electromagnetic radiation, and then relaxes to its ground state by emitting a photon from the excited state. The specific frequency of the excited orbital electron or the electromagnetic radiation emitted by the photon during relaxation depends on the specific atom, molecule, or nanostructure. In most cases, the light emitted by the substance has a longer wavelength and thus lower energy than the radiation absorbed by the substance. However, when the absorbed electromagnetic radiation is intense, one electron can absorb two photons. This two-photon absorption can result in the emission of radiation with a shorter wavelength and thus higher energy than the absorbed radiation. Additionally, the emitted radiation can also have the same wavelength as the absorbed radiation.

[0060] Fluorescence imaging has many practical applications, including mineralogy, geology, medicine, spectroscopy of chemical sensors, detecting biological processes or signals, etc. Fluorescence can be particularly useful in biochemistry and medicine as a non-destructive means for tracking or analyzing biological molecules. Biological molecules, including certain tissues or structures, can be tracked by analyzing the fluorescence emission of the biological molecules after excitation by electromagnetic radiation of certain wavelengths. However, relatively few cellular components are naturally fluorescent. In certain implementations, it can be advantageous to visualize certain tissues, structures, chemical processes, or biological processes that are not fluorescent in nature. In such implementations, a dye or agent can be administered to the body that can include molecules, proteins, or quantum dots with fluorescent properties. The agent or dye can then fluoresce after excitation by electromagnetic radiation of certain wavelengths. Different agents or dyes can include different molecules, proteins, and / or quantum dots that will fluoresce under electromagnetic radiation of a particular wavelength. Thus, it can be desirable to excite the agent or dye with electromagnetic radiation of a particular frequency band to achieve fluorescence and to identify the desired tissue, structure, or process in vivo.

[0061] Fluorescence imaging can provide valuable information that can be used for diagnostic purposes and / or can be visualized in real-time during medical procedures in the field of medicine. A particular agent or dye can be administered to the body to cause certain tissues, structures, chemical processes, or biological processes to fluoresce. The fluorescence of the agent or dye can highlight body structures, such as blood vessels, nerves, particular organs, etc. Additionally, the fluorescence of the agent or dye can highlight conditions or diseases, such as cancerous cells or cells undergoing certain biological or chemical processes that can be associated with a condition or disease. Fluorescence imaging can be used in real-time by a medical practitioner or computer program for distinguishing, for example, cancerous cells from non-cancerous cells during surgical tumor extraction. Fluorescence imaging can also be used as a non-destructive means for tracking and visualizing conditions in vivo that are otherwise invisible to the human eye or indistinguishable in RGB images over time.

[0062] Systems, methods, and apparatus for generating fluorescence imaging data can be used in conjunction with reagents or dyes. Some reagents or dyes are known to attach to certain types of tissue and fluoresce at specific wavelengths of the electromagnetic spectrum. In one implementation, a reagent or dye is administered to a patient that is configured to fluoresce when activated by light of certain wavelengths. The endoscopic imaging systems disclosed herein are used to excite the reagents or dyes and cause them to fluoresce. The fluorescence of the reagents or dyes is captured by the endoscopic imaging system to aid in the identification of tissue or structures in a body cavity. In one implementation, a plurality of reagents or dyes are administered to a patient, each configured to fluoresce at a different wavelength and / or provide an indication of a different structure, tissue, chemical reaction, biological process, etc. In such implementations, the endoscopic imaging system emits each of the applicable wavelengths to cause each of the applicable reagents or dyes to fluoresce. This can eliminate the need to perform individual imaging procedures for each of the plurality of reagents or dyes.

[0063] Imaging reagents can enhance imaging capabilities in the pharmaceutical, medical, biotechnology, diagnostic, and medical procedure industries. Many imaging technologies such as X-ray, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI), and nuclear medicine primarily analyze anatomy and morphology, and are not able to detect changes at the molecular level. Fluorescent reagents, dyes, and probes, including quantum dot nanoparticles and fluorescent proteins, aid medical imaging technologies by providing additional information about certain tissues, structures, chemical processes, and / or biological processes present within an imaged region. Imaging using fluorescent reagents enables cell tracking and / or tracking of certain molecular biomarkers. Fluorescent reagents can be used to image cancer, infections, inflammation, stem cell biology, etc. Many fluorescent reagents and dyes are being developed and applied to visualize and track biological processes in a non-destructive manner. Such fluorescent reagents can be excited by electromagnetic radiation of certain wavelengths or wavelength bands. Similarly, these fluorescent reagents can emit relaxation energy of certain wavelengths or wavelength bands when fluorescing, and the emitted relaxation energy can be read by a sensor to determine the location and / or boundaries of the reagent or dye.

[0064] In one embodiment of the present disclosure, an endoscopic imaging system pulses electromagnetic radiation to excite electrons in a fluorescent reagent or dye. The endoscopic imaging system can pulse electromagnetic radiation of multiple different wavelengths during a single imaging session to cause multiple different reagents or dyes to fluoresce. The endoscope includes an image sensor that is sensitive to the relaxation wavelengths of the one or more reagents or dyes. The imaging data generated by the image sensor can be used to identify the location and boundaries of the one or more reagents or dyes. The endoscopic system can also pulse electromagnetic radiation in the red, green, and blue bands of visible light so that the fluorescence imaging can be overlaid on an RGB video stream.

[0065] Laser mapping imaging

[0066] In one embodiment, the systems, methods, and devices disclosed herein provide devices for generating laser mapping data with an endoscopic imaging system. Laser mapping data can be used to determine precise measurements and topographical profiles of a scene. In one implementation, laser mapping data is used to determine precise measurements between, for example, structures or organs in a body cavity, devices or tools in a body cavity, and / or critical structures in a body cavity. As described herein, the term “laser mapping” can encompass technologies referred to as laser mapping, laser scanning, topographical scanning, three-dimensional scanning, laser tracking, tool tracking, and the like. Laser mapping exposure frames as discussed herein can include topographical data of a scene, dimensions between objects or structures within a scene, dimensions or distances of tools or objects within a scene, and the like.

[0067] Laser mapping generally includes the controlled deflection of a laser beam. Within the field of three-dimensional object scanning, laser mapping combines the controlled steering of a laser beam with a laser rangefinder. By making distance measurements in each direction, a laser rangefinder can quickly capture the surface shape of an object, tool, and panorama. The construction of a full three-dimensional topography can include combining multiple surface models obtained from different perspectives. Various measurement systems and methods exist in the art for applications in archaeology, geography, atmospheric physics, autonomous vehicles, and the like. One such system includes Light Detection and Ranging (LIDAR), which is a three-dimensional laser mapping system. LIDAR has been applied in navigation systems such as airplanes or satellites to determine the position and orientation of the sensor in combination with other systems and sensors. LIDAR uses an active sensor to illuminate an object and detect the energy that is reflected from the object and back to the sensor.

[0068] As described herein, the term “laser mapping” includes laser tracking. Laser tracking or tool tracking using lasers measures objects by determining the position of an optical target held relative to those objects. Laser trackers can be accurate to the order of 0.025 mm over distances of several meters. In one embodiment, an endoscopic imaging system pulses light for use in conjunction with a laser tracking system such that positions or tools within a scene can be tracked and measured. In such embodiments, the endoscopic imaging system can pulse a laser tracking mode on tools, objects, or other structures within a scene imaged by the endoscopic imaging system. A target can be placed on the tools, objects, or other structures within the scene. Measurements between the endoscopic imaging system and the target can be triggered and acquired at selected points such that the endoscopic imaging system can track the position of the target (and the tool, object, or other structure to which the target is attached).

[0069] Pulsed imaging

[0070] Some implementations of the present disclosure include various aspects of sensor and system combination designs that are capable of generating high definition images with reduced number of pixels in a limited lighting environment. This is achieved by pulsing a single color wavelength frame by frame and switching or alternating between single different color wavelengths per frame using a controlled light source in combination with a high frame capture rate and specially designed corresponding monochrome sensor. Additionally, electromagnetic radiation outside the visible spectrum can be pulsed to enable the generation of hyperspectral images. The pixels can be color agnostic such that each pixel generates data for each electromagnetic radiation pulse, including pulses of red, green, and blue visible light wavelengths and other wavelengths useful for hyperspectral imaging.

[0071] The system of the present disclosure is an endoscope system for use in light deficient environments. The system includes an endoscope including an image sensor, where the image sensor is configured to sense reflected electromagnetic radiation for generating a plurality of exposure frames that can be combined to generate an RGB image frame with hyperspectral data superimposed thereon. The system includes an emitter for emitting pulses of electromagnetic radiation. The system includes a controller (alternatively referred to as "control circuitry") in electrical communication with the image sensor and the emitter. The controller controls a duty cycle of the emitter in response to a signal corresponding to the duty cycle of the emitter. The image sensor includes a bidirectional pad that can send and receive information. The bidirectional pad of the image sensor operates in a frame period that is divided into three defined states, including a rolling readout state, a service line state, and a configuration state. The system includes an oscillator disposed in the controller and a frequency detector connected to the controller. The frequency detector controls a clock frequency of the image sensor in response to a signal from the controller corresponding to the oscillator frequency. The system causes clock signal data to be transmitted from the bidirectional pad of the image sensor to the controller during the service line phase and the configuration phase. The system causes the exposure frames to be synchronized without using an input clock or a data transfer clock.

[0072] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe these embodiments. It will, nevertheless, be understood that no limitation of the scope of the disclosure is intended by this specification. Alterations and further modifications of the inventive features illustrated herein, and additional applications of the principles of the present disclosure as illustrated herein, which would normally occur to one skilled in the relevant art and informed by the disclosure, are to be construed as being within the scope of the disclosure claimed.

[0073] Before the structures, systems and methods for producing images in light deficient environments are disclosed and described, it is to be understood that the disclosure is not limited to the particular structures, configurations, process steps, and materials disclosed herein as such structures, configurations, process steps, and materials can vary somewhat and still be within the scope and spirit of the disclosure. It is also to be understood that the terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting as the scope of the present disclosure will only be limited by the appended claims and equivalents thereof.

[0074] In describing and claiming the subject matter of the present disclosure, the following terminology will be used in accordance with the definitions set out below.

[0075] It must be noted that as used herein and in the appended claims, the singular forms "a", "an", and "the" include plural reference unless the context clearly dictates otherwise.

[0076] As used herein, the terms "comprises", "comprising", "characterized by" and grammatical equivalents thereof are open-ended transitional phrases and do not exclude additional, unrecited elements or method steps.

[0077] As used herein, the phrase "consisting of and grammatical equivalents thereof excludes any element or step not specified in the claim.

[0078] As used herein, the phrase "consisting essentially of and grammatical equivalents thereof limits the scope of a claim to the specified materials or steps and to materials or steps that do not materially affect the basic and novel characteristics of the claimed disclosure.

[0079] As used herein, the term "proximal" refers broadly to the concept of a portion that is close to a starting point.

[0080] As used herein, the term "distal" refers generally to the concept of a portion that is opposite to proximal, and thus depending on the context, refers to a portion that is further from or the farthest from a starting point.

[0081] As used herein, a color sensor or a multispectral sensor is one that is known to have an array of color filters (CFA) on it so as to filter the incident electromagnetic radiation into its individual components. In the visible range of the electromagnetic spectrum, such a CFA can be based on a Bayer pattern or a modified version thereof so as to separate the green, red and blue spectral components in the light.

[0082] As used herein, a monochrome sensor refers to an imaging sensor without a filtering function. Because the pixels are color agnostic, their effective spatial resolution is significantly higher than the pixel color (typically filtered in a Bayer pattern) counterparts in traditional single sensor cameras. Monochrome sensors can also have higher quantum efficiency because there are fewer incident photons wasted between individual pixels.

[0083] As used herein, an emitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of emitters can be configured to be capable of emitting pulses and have a very specific frequency or range of frequencies from within the entire electromagnetic spectrum. The pulses can include wavelengths in the visible and non-visible range. The emitters can cycle on and off to produce pulses or can produce pulses with a shutter mechanism. The emitters can have variable power output levels or can be controlled with auxiliary devices such as apertures or filters. The emitters can emit a broad spectrum or full spectrum of electromagnetic radiation that can be pulsed by color filtering or shuttering. The emitters can include multiple electromagnetic sources acting alone or in concert.

[0084] It should be noted that as used herein, the term "light" is both a particle and a wavelength and is intended to mean electromagnetic radiation that can be detected by the pixel array 122 and can include wavelengths from the visible and non-visible spectrum of electromagnetic radiation. The term "band" as used herein refers to a predetermined range of wavelengths of the electromagnetic spectrum that is less than the entire spectrum or in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that is controllable with respect to the portion of the electromagnetic spectrum that is emitted or the physical properties, emission intensity, or emission duration of the components that can be operated or all of the above. The emitter can emit light in any dithered, diffuse, or collimated emission and can be controlled digitally or by analog methods or systems. As used herein, an electromagnetic emitter is a source of bursts of electromagnetic energy and includes light sources such as lasers, LEDs, incandescent light, or any light source that can be digitally controlled.

[0085] Referring now to the drawings, FIG. 1 A schematic diagram of a system 100 for sequential pulsed imaging in a light deficient environment is shown. The system 100 can be deployed to generate an RGB image with specialized data layered thereon. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a band of electromagnetic radiation in a light deficient environment 112 and the pixel array 122 senses instances of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 work sequentially such that one or more pulses of the band of electromagnetic radiation produces image data sensed by the pixel array 122.

[0086] It should be noted that as used herein, the term "light" is both a particle and a wavelength, and is intended to mean electromagnetic radiation that can be detected by the pixel array 122, and can include wavelengths from the visible and non-visible spectrum of electromagnetic radiation. The term "bin" as used herein refers to a predetermined range of wavelengths of the electromagnetic spectrum that is less than the entire spectrum, or in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that is controllable with respect to the portion of the electromagnetic spectrum that is emitted, or the physical properties, emission intensity, or emission duration of the components that can be operated, or all of the above. The emitter can emit light in any of a dithered, diffuse, or collimated emission, and can be controlled digitally or by analog methods or systems. As used herein, an electromagnetic emitter is a source of bursts of electromagnetic energy, and includes light sources such as lasers, LEDs, incandescent light, or any light source that can be digitally controlled.

[0087] The pixel array 122 of the image sensor can be electronically paired with the emitter 102 such that the emitter 102 and the pixel array 122 are synchronized during operation for both receiving the emission and the adjustments made within the system. The emitter 102 can be tuned to emit electromagnetic radiation in the form of a laser that can be pulsed to illuminate the light deficient environment 112. The emitter 102 can pulse in correspondence to the operation and function of the pixel array 122. The emitter 102 can pulse light in a plurality of electromagnetic bins such that the pixel array receives electromagnetic energy and produces a data set that corresponds in time to each particular electromagnetic bin. For example, FIG. 1A specific implementation is shown in which the emitter 102 emits four different partitions of electromagnetic radiation, including a red 104 wavelength, a green 106 wavelength, a blue 108 wavelength, and a special 110 emission. The special 110 emission can include an excitation wavelength for causing a reagent to fluoresce, a hyperspectral partition of electromagnetic radiation, and / or a laser mapping mode. The special 110 emission can include multiple individual emissions that are separate and independent from each other. The special 110 emission can include a combination of an excitation wavelength for causing a reagent to fluoresce and a laser mapping mode, with the emissions being separate and independent from each other. Data resulting from the individual emissions can be analyzed sequentially to identify key structures within a scene based on fluorescence imaging data, and also to identify dimensions or positioning of the key structures based on a combination of laser mapping data and fluorescence imaging data. The special 110 emission can include a combination of a hyperspectral band of electromagnetic radiation and a laser mapping mode, with the emissions being separate and independent from each other. Data resulting from the individual emissions can be analyzed sequentially to identify key structures within a scene based on hyperspectral imaging data, and also to identify dimensions or positioning of the key structures based on a combination of laser mapping data and hyperspectral imaging data. In one embodiment, the special 110 emission includes any desired combination of emissions that can be combined with data resulting from the pulsed red 104, pulsed green 106, and pulsed blue 108 emissions. The special 110 emissions can be interspersed within the pulsed pattern such that pulses of different types of special 110 emissions are not as frequent as pulses of the pulsed red 104, pulsed green 106, and pulsed blue 108 emissions.

[0088] In FIG. 1 an alternative embodiment not shown in FIG. 1, the pulsed emission of light includes a luminance (“Y”) emission, a red chrominance (“Cr”) emission, and a blue chrominance (“Cb”) emission, rather than the pulsed red 104 emission, the pulsed green 106 emission, and the pulsed blue 108 emission. In one embodiment, the controller or the emitter 102 modulates the electromagnetic radiation pulses to provide luminance and / or chrominance information according to color transform coefficients that convert light energy from a red light energy space, a green light energy space, and a blue light energy space to a luminance light energy space, a red chrominance light energy space, and a blue chrominance light energy space. The pulsed emission of light can also include modulated blue chrominance (λ“Y + Cb”) pulses and / or modulated red chrominance (λ“Y + Cr”) pulses.

[0089] The light deficient environment 112 includes structures, tissues, and other elements that reflect a combination of red 114, green 116, and / or blue 118 light. Structures perceived as red 114 will reflect pulsed red 104 light. Reflection from the red structures results in the pixel array 122 sensing red 105 after the pulsed red 104 emission. The data sensed by the pixel array 122 produces a red exposure frame. Structures perceived as green 116 will reflect pulsed green 106 light. Reflection from the green structures results in the pixel array 122 sensing green 107 after the pulsed green 106 emission. The data sensed by the pixel array 122 produces a green exposure frame. Structures perceived as blue 118 will reflect pulsed blue 108 light. Reflection from the blue structures results in the pixel array 122 sensing blue 109 after the pulsed blue 108 emission. The data sensed by the pixel array 122 produces a blue exposure frame.

[0090] When the structures are a combination of colors, the structures will reflect a combination of the pulsed red 104 emission, the pulsed green 106 emission, and / or the pulsed blue 108 emission. For example, structures perceived as purple will reflect light from the pulsed red 104 emission and the pulsed blue 108 emission. The resulting data sensed by the pixel array 122 will indicate that light was reflected in the same area after the pulsed red 104 emission and the pulsed blue 108 emission. When the resulting red exposure frame and blue exposure frame are combined to form an RGB image frame, the RGB image frame will indicate that the structures are purple.

[0091] In embodiments where the light deficient environment 112 includes fluorescent reagents or fluorescent dyes or includes one or more fluorescent structures, tissues, or other elements, the pulsing scheme can include emission of certain fluorescent excitation wavelengths. Certain fluorescent excitation wavelengths can be selected to cause known fluorescent reagents, fluorescent dyes, or other structures to fluoresce. Fluorescent structures will be sensitive to the fluorescent excitation wavelengths and will emit a fluorescent relaxation wavelength. The fluorescent relaxation wavelength will be sensed by the pixel array 122 after the fluorescent excitation wavelength is emitted. The data sensed by the pixel array 122 produces a fluorescent exposure frame. The fluorescent exposure frame can be combined with a plurality of other exposure frames to form an image frame. The data in the fluorescent exposure frame can be overlaid on an RGB image frame that includes data from red exposure frames, green exposure frames, and blue exposure frames.

[0092] In embodiments in which the light deficient environment 112 includes structures, tissues, or other materials that emit a spectral response to certain partitions of the electromagnetic spectrum, the pulsing scheme can include emission of hyperspectral partitions of electromagnetic radiation for eliciting a spectral response from structures, tissues, or other materials present in the light deficient environment 112. The spectral response includes emission or reflection of electromagnetic radiation of certain wavelengths. The spectral response can be sensed by the pixel array 122 and produce a hyperspectral exposure frame. The hyperspectral exposure frame can be combined with a plurality of other exposure frames to form an image frame. Data in the hyperspectral exposure frame can be overlaid on an RGB image frame that includes data from red, green, and blue exposure frames.

[0093] In one embodiment, the pulsing scheme includes emission of a laser mapping mode or a tool tracking mode. After emitting the laser mapping mode or the tool tracking mode, the reflected electromagnetic radiation sensed by the pixel array 122 produces a laser mapping exposure frame. Data in the laser mapping exposure frame can be provided to a corresponding system to identify, for example, distances between tools present in the light deficient environment 112, a three-dimensional surface topology of a scene in the light deficient environment 112, distances, sizes, or locations of structures or objects within the scene, etc. This data can be overlaid on an RGB image frame or otherwise provided to a user of the system.

[0094] The emitter 102 can be a laser emitter capable of emitting pulsed red 104 light for generating sensed red 105 data to identify red 114 elements within the light deficient environment 112. The emitter 102 can also be capable of emitting pulsed green 106 light for generating sensed green 107 data to identify green 116 elements within the light deficient environment. The emitter 102 can also be capable of emitting pulsed blue 108 light for generating sensed blue 109 data to identify blue 118 elements within the light deficient environment. The emitter 102 can also be capable of emitting special 110 emissions for mapping a topology 120 of a scene within the light deficient environment 112. The emitter 102 can emit the pulsed red 104, pulsed green 106, pulsed blue 108, and pulsed special 110 emissions in any desired order.

[0095] The pixel array 122 senses the reflected electromagnetic radiation. Each of the sense red 105, sense green 107, sense blue 109, and sense special 111 data can be referred to as an "exposure frame." The sense special 111 can result in multiple separate exposure frames that are separate and independent from each other. For example, the sensed special 111 can produce a fluorescence exposure frame, a hyperspectral exposure frame, and / or a laser mapping exposure frame that includes laser mapping data. Each exposure frame is assigned a particular color or wavelength partition, where the assignment is based on the timing of the pulsed color or wavelength partitions from the emitter 102. The combination of an exposure frame and the assigned particular color or wavelength partition can be referred to as a data set. Any given data set can be assigned a color based on prior information about the emitter, even if the pixels 122 are not of a special color.

[0096] For example, during operation, after the pulsed red 104 light is pulsed in the light deficient environment 112, the pixel array 122 senses the reflected electromagnetic radiation. The reflected electromagnetic radiation produces an exposure frame, and the exposure frame is categorized as sense red 105 data because it corresponds in time to the pulsed red 104 light. The exposure frame along with an indication that it corresponds in time to the pulsed red 104 light is a "data set." This process is repeated for each partition of electromagnetic radiation emitted by the emitter 102. The data created by the pixel array 122 includes a sense red 105 exposure frame that identifies the red 114 component in the light deficient environment and corresponds in time to the pulsed red 104 light. The data also includes a sense green 107 exposure frame that identifies the green 116 component in the light deficient environment and corresponds in time to the pulsed green 106 light. The data also includes a sense blue 109 exposure frame that identifies the blue 118 component in the light deficient environment and corresponds in time to the pulsed blue 108 light. The data also includes a sense special 111 exposure frame that identifies the topology 120 and corresponds in time to the special 110 emission.

[0097] In one embodiment, the three data sets representing the red, green, and blue electromagnetic pulses are combined to form a single image frame. Thus, the information in the red, green, and blue exposure frames is combined to form a single RGB image frame. One or more additional data sets representing other wavelength partitions can be overlaid on the single RGB image frame. The one or more additional data sets can represent, for example, laser mapping data, fluorescence imaging data, and / or hyperspectral imaging data.

[0098] It should be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition, and that any color combination or any electromagnetic partition can be used in place of RED, GREEN, and BLUE, such as cyan, magenta, and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible wavelengths and non-visible wavelengths. In the figure, the light deficient environment 112 to be imaged includes a red 114 portion, a green 116 portion, and a blue 118 portion, and also includes a topology 120 that can be sensed and mapped into a three-dimensional rendering. As shown, the reflected light from the electromagnetic pulse contains data only for the portions of the object that have the particular color corresponding to the color partition of the pulse. These individual color (or color interval) data sets can then be used to reconstruct an image by combining the data sets at 126. The information in each of the multiple exposure frames (i.e., the multiple data sets) can be combined by a controller 124, control unit, camera control unit, image sensor, image signal processing pipeline, or some other computing resource that can be configured to process the multiple exposure frames and combine the data sets at 126. The data sets can be combined to generate a single image frame either within the endoscope unit itself or off-site by some other processing resource.

[0099] FIG. 2 is a system 200 for providing illumination to a light deficient environment such as for endoscopic imaging. The system 200 can be used in conjunction with any of the systems, methods, or devices disclosed herein. The system 200 includes an emitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an inner lumen waveguide 210, an inner lumen 212, and an image sensor 214 with accompanying optical components such as lenses. The emitter 202, which can generally be referred to as a “light source,” generates light that passes through the jumper waveguide 206 and the inner lumen waveguide 210 to illuminate a scene at a distal end of the inner lumen 212. The emitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared wavelengths, ultraviolet wavelengths, hyperspectral wavelengths, fluorescence excitation wavelengths, or other wavelengths. The inner lumen 212 can be inserted into a patient’s body for imaging, such as during a procedure or examination. The light is output as shown by the dashed line 216. The scene illuminated by the light can be captured using the image sensor 214 and displayed to a physician or some other medical personnel. The controller 204 can provide control signals to the emitter 202 to control when illumination is provided to a scene. In one embodiment, the emitter 202 and the controller 204 are located within a camera control unit (CCU) or external control console to which the endoscope is connected. If the image sensor 214 includes a CMOS sensor, the light can be provided to the scene periodically in a series of illumination pulses between readout cycles of the image sensor 214 during so-called blanking periods. Thus, the light can be pulsed in a controlled manner to avoid overlapping into the readout cycles of the image pixels in the pixel array of the image sensor 214.

[0100] In one embodiment, the lumen waveguide 210 includes one or more optical fibers. These optical fibers can be made of low cost materials, such as plastic, to allow for disposal of the lumen waveguide 210 and / or other portions of the endoscope. In one embodiment, the lumen waveguide 210 is a single glass fiber with a diameter of 500 microns. The jumper waveguide 206 can be permanently attached to the emitter 202. For example, the jumper waveguide 206 can receive light from an emitter within the emitter 202 and provide the light to the lumen waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 106 includes one or more glass optical fibers. The jumper waveguide can include any other type of waveguide for directing light to the lumen waveguide 210. The connector 208 can selectively couple the jumper waveguide 206 to the lumen waveguide 210 and allow light within the jumper waveguide 206 to pass through the lumen waveguide 210. In one embodiment, the lumen waveguide 210 is directly coupled to the light source without any intervening jumper waveguide 206.

[0101] The image sensor 214 includes an array of pixels. In one embodiment, the image sensor 214 includes two or more arrays of pixels for generating three-dimensional images. The image sensor 214 can constitute two additional image sensors each having an independent array of pixels and can operate independently of one another. The array of pixels of the image sensor 214 includes active pixels and optical black (“OB”) pixels or blind pixels. The active pixels can be transparent “color agnostic” pixels capable of sensing imaging data for any wavelength of electromagnetic radiation. The optical black pixels are read during a blanking period of the array of pixels when the array of pixels is “reset” or calibrated. In one embodiment, light is pulsed during the blanking period of the array of pixels when the optical black pixels are read. After the optical black pixels have been read, the active pixels are read during a readout period of the array of pixels. The active pixels can be charged by electromagnetic radiation pulsed during the blanking period so that the active pixels are ready to be read by the image sensor during the readout period of the array of pixels.

[0102] FIG. 2A complementary system hardware such as a special or general purpose computer. Implementations within the scope of the disclosure can also include physical and other non-transitory computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: computer storage media (devices) and transmission media.

[0103] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSDs") (e.g., based on RAM), Flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

[0104] "Network" refers to one or more data links that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. In one specific implementation, the sensors and camera control unit can be networked to communicate with each other, as well as with other components connected through the network to which they are connected. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and / or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0105] Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to computer storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received by way of network or data links can be buffered in RAM within a network interface module (e.g., a "NIC"), and then eventually transferred to computer system RAM and / or to less volatile computer storage media (devices) at a computer system. RAM can also include solid state drives (SSDs or PCIx-based real-time memory tiering storage devices such as FusionIO). Thus, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0106] Computer-executable instructions include, for example, instructions and data which, when executed by one or more processing devices, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions can be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0107] Those skilled in the art will appreciate that the present disclosure can be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, hand-held devices, game devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the above- described computing devices can be provided by or located within an entity. The present disclosure can also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0108] In addition, where appropriate, functions described herein can be performed in one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the following description and claims to refer to particular system components. As one skilled in the art will appreciate, components can be referred to by different names. This document does not intend to distinguish between components that differ in name but not function.

[0109] FIG. 2A FIG. 25 is a block diagram illustrating an example computing device 250. The computing device 250 can be used to execute various programs, such as those discussed herein. The computing device 250 can be used as a server, a client, or any other computing entity. The computing device 250 can perform various monitoring functions discussed herein, and can execute one or more applications, such as the applications described herein. The computing device 250 can be one of a variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a handheld computer, a camera control unit, a tablet computer, and the like.

[0110] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all of which are coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory device 254 and / or the mass storage device 258. The processor 252 can also include various types of computer-readable media, such as cache memory.

[0111] Memory devices 254 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 264) and / or nonvolatile memory (e.g., read-only memory (ROM) 266). Memory devices 254 can also include rewritable ROM, such as flash memory.

[0112] Mass storage devices 258 include various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., Flash memory), and so forth. As FIG. 2 indicated in FIG. 2, a particular mass storage device is a hard disk drive 274. Various drives can also be included in mass storage devices 258 to enable reading from and / or writing to various computer- readable media. Mass storage devices 258 include removable media 276 and / or non-removable media.

[0113] I / O devices 260 include various devices that enable input and output of data and / or other information to and from computing device 250. Exemplary I / O devices 260 include digital imaging devices, electromagnetic sensors and transmitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.

[0114] Display devices 280 include any type of device capable of displaying information to one or more users of computing device 250. Examples of display devices 280 include monitors, display terminals, video projection devices, and the like.

[0115] Interfaces 256 include various interfaces that enable computing device 250 to interact with other systems, devices, or computing environments. Exemplary interfaces 256 can include any number of different network interfaces 270, such as interfaces connecting local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include a user interface 268 and a peripheral device interface 272. Interfaces 256 can also include one or more user interface elements 268. Interfaces 256 can also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mice, trackpads, etc.), keyboards, and the like.

[0116] Bus 262 enables the processing system 252, the memory devices 254, the interface 256, the mass storage devices 258, and the I / O devices 260 to communicate with one another and with other devices or components coupled to bus 262. The bus 262 represents what in

[0117] For illustrative purposes, the programs and other executable program devices shown herein are discrete blocks, although it is understood that such programs and devices can reside at various times in different memory devices of computing device 250, and are executed by processor 252. Alternatively, the systems and procedures described herein can be implemented in hardware, or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to carry out one or more of the systems and procedures described herein.

[0118] FIG. 3A An operational cycle of a sensor used in a rolling readout mode or during sensor readout 300 is shown. Frame readout can begin at and can be represented by the vertical line 310. The readout period is represented by the diagonal or slanted line 302. The active pixels of the pixel array of the image sensor can be read out row by row, with the top of the downward sloping edge being the top sensor row 312 and the bottom of the downward sloping edge being the bottom sensor row 314. The time between the last row readout and the next readout cycle can be referred to as the blanking period 316. It should be noted that some of the sensor pixel rows can be covered with a light shield (e.g., a metallic coating or any other substantially black layer of another material type). These covered pixel rows can be referred to as optical black rows 318 and 320. The optical black rows 318 and 320 can be used as inputs to a correction algorithm. As shown, these optical black rows 318 and 320 can be located on the top of the pixel array or the bottom of the pixel array or both the top and bottom of the pixel array. FIG. 3A

[0119] FIG. 3B A method of controlling the amount of electromagnetic radiation (e.g., light) that is exposed to a pixel so as to be integrated or accumulated by the pixel is shown. It should be understood that a photon is the basic particle of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted to an electrical charge or current. An electronic shutter or rolling shutter (shown by dashed line 322) can be used to start the integration time by resetting the pixel. Light will then be integrated until the next readout phase. The position of the electronic shutter 322 can be moved between two readout cycles 302 in order to control the pixel saturation of a given amount of light. It should be noted that this technique allows the integration time to be constant between two different rows, but introduces a delay as one moves from the top row to the bottom row.

[0120] FIG. 3C A situation is shown in which the electronic shutter 322 has been removed. In this configuration, the integration of incident light can begin during readout 302 and can end at the next readout cycle 302, which also defines the beginning of the next integration.

[0121] FIG. 3D ​A configuration is shown without electronic shutter 322 but with a controlled and pulsed light 330 during the blanking period 316. This ensures that all rows see the same light emanating from the same light pulse 330. In other words, each row will start its integration in a dark environment, which can be located at the optical black back row 320 of the readout frame (m) for maximum light pulse width, and will receive the light pass and will end its integration in a dark environment, which can be located at the optical black front row 318 of the next subsequent readout frame (m+1) for maximum light pulse width. In, for example FIG. 3D The image generated by the light pulse will be available only during the frame (m+1) readout without interfering with frame (m) and frame (m+2). It should be noted that the condition for having the light pulse read out in only one frame and not interfering with adjacent frames is to fire the given light pulse during the blanking period 316. Because the optical black rows 318, 320 are not sensitive to light, the optical black back row 320 time of frame (m) and the optical black front row 318 time of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 330.

[0122] As shown in FIG. 3A The sensor can cycle multiple times to receive data for each pulsed color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum) as shown in FIG. 4. Each cycle can be timed. In one embodiment, the cycles can be timed to operate within a 16.67 ms interval. In another embodiment, the cycles can be timed to operate within an 8.3 ms interval. It should be understood that other timing intervals are contemplated by the present disclosure and are intended to fall within the scope of the present disclosure.

[0123] FIG. 4A The operation of an embodiment of an electromagnetic emitter is shown graphically. The emitter can be timed to correspond to the cycles of the sensor such that electromagnetic radiation is emitted during the sensor operation cycle and / or a portion of the sensor operation cycle. FIG. 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the emitter can pulse during the readout period 302 of the sensor operation cycle. In one embodiment, the emitter can pulse during the blanking portion 316 of the sensor operation cycle. In one embodiment, the emitter can pulse for a duration that is during portions of two or more sensor operation cycles. In one embodiment, the emitter can start the pulse during the blanking portion 316 or during the optical black portion 320 of the readout period 302 and end the pulse during the readout period 302 or the optical black portion 318 of the readout period 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of the present disclosure so long as the pulse of the emitter and the cycle of the sensor correspond.

[0124] FIG. 4B The duration and magnitude of the emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, pulse 3 at 416) are represented graphically to control the exposure. An emitter with a fixed output magnitude can be pulsed over a longer time interval to provide more electromagnetic energy to the pixels, or the emitter can be pulsed over a shorter time interval to provide less electromagnetic energy. Whether a longer or shorter time interval is needed depends on the operating conditions. FIG. 3D and FIG. 4A The pulses are of a certain interval during any of the cycles described to provide the required electromagnetic energy to the pixel array. An emitter with a fixed output magnitude can be pulsed over a longer time interval to provide more electromagnetic energy to the pixels, or the emitter can be pulsed over a shorter time interval to provide less electromagnetic energy. Whether a longer or shorter time interval is needed depends on the operating conditions.

[0125] The magnitude of the emission itself can be increased compared to adjusting the time interval of the emitter pulse fixed output magnitude to provide more electromagnetic energy to the pixels. Similarly, decreasing the magnitude of the pulse can provide less electromagnetic energy to the pixels. It should be noted that an embodiment of the system can have the ability to adjust both the magnitude and the duration if needed. Additionally, the sensor can be adjusted to increase its sensitivity and duration as needed for optimal image quality. FIG. 4B The magnitude and duration of the pulses are shown to be varied. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. Additionally, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416 such that the electromagnetic energy provided by the pulse is shown by the area under the pulse shown in the illustration. In the illustration, pulse 2 at 414 has a relatively low magnitude or intensity and a longer duration when compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration when compared to pulse 1 at 412 and pulse 2 at 414.

[0126] FIG. 5 for operation cycles, electromagnetic emitters, and emitted electromagnetic pulses in accordance with the principles and teachings of the present disclosure FIGS. 3A-3D and FIG. 4A The present embodiments of the disclosure show a graphical representation of the imaging system during operation cycles, electromagnetic emitters, and emitted electromagnetic pulses in accordance with the principles and teachings of the present disclosure. As can be seen in the figure, the electromagnetic emitters are pulsed primarily during the blanking period 316 of the image sensor such that the pixels will be charged and ready to be read during the readout period 302 of the image sensor cycle. FIG. 5 The dashed lines in the figure represent the electromagnetic radiation pulses (from FIG. 4A ) from the electromagnetic emitters. The electromagnetic radiation pulses are emitted primarily during the blanking period 316 of the image sensor, but can be overlaid with the readout period 302 of the image sensor.

[0127] An exposure frame includes data read by the pixel array of the image sensor during a readout period 302. The exposure frame can be combined with an indication of the type of pulse emitted by the emitter prior to the readout period 302. The combination of the exposure frame and the indication of the type of pulse can be referred to as a data set. Multiple exposure frames can be combined to generate a black and white or RGB color image. Additionally, hyperspectral, fluorescence, and / or laser mapping imaging data can be overlaid on the black and white or RGB image.

[0128] In one embodiment, an RGB image frame is generated based on three exposure frames, including a red exposure frame generated by the image sensor after red light emission, a green exposure frame generated by the image sensor after green light emission, and a blue exposure frame generated by the image sensor after blue light emission. Fluorescence imaging data can be overlaid on the RGB image frame. The fluorescence imaging data can be extracted from one or more fluorescence exposure frames. A fluorescence exposure frame includes data generated by the image sensor during a readout period 302 after emission of electromagnetic radiation at an excitation wavelength used to excite a fluorescent agent. The data sensed by the pixel array after excitation of the fluorescent agent can be at a relaxation wavelength emitted by the fluorescent agent. The fluorescence exposure frame can include multiple fluorescence exposure frames, each generated by the image sensor after a different type of fluorescence excitation emission. In one embodiment, the fluorescence exposure frame includes multiple fluorescence exposure frames, including a first fluorescence exposure frame generated by the image sensor after emission of electromagnetic radiation having a wavelength of about 770 nm to about 790 nm, and a second fluorescence exposure frame generated by the image sensor after emission of electromagnetic radiation having a wavelength of about 795 nm to about 815 nm. The fluorescence exposure frame can include additional additional fluorescence exposure frames generated by the image sensor after other fluorescence excitation emissions of light as needed for the imaging application.

[0129] In one embodiment, an exposure frame is data sensed by the pixel array during a readout period 302 that occurs after a blanking period 316. The emission of electromagnetic radiation is emitted during the blanking period 316. In one embodiment, a portion of the emission of electromagnetic radiation overlaps with the readout period 316. The blanking period 316 occurs when the optical black pixels of the pixel array are being read, and the readout period 302 occurs when the active pixels of the pixel array are being read. The blanking period 316 can overlap with the readout period 302.

[0130] FIG. 6A and FIG. 6B A process for recording image frames is shown. Multiple image frames can be strung together to generate a video stream. A single image frame can include data from multiple exposure frames, where an exposure frame is data sensed by the pixel array after emission of electromagnetic radiation. FIG. 6AA conventional process is shown that is typically implemented with a color image sensor that has a color filter array (CFA) to filter out certain wavelengths of light for each pixel. FIG. 6B is the process disclosed herein, and can be implemented with a monochrome "color agnostic" image sensor that receives electromagnetic radiation of all wavelengths.

[0131] FIG. 6A The process shown occurs from time t(0) to time t(l). The process starts with white light emission 602 and sensing white light 604. At 606, an image is processed and displayed based on the sensing at 604.

[0132] FIG. 6B The process shown occurs from time t(0) to time t(l). The process starts with green light emission 612 and sensing reflected electromagnetic radiation 614 after the emission of green light 612. The process continues with red light emission 616 and sensing reflected electromagnetic radiation 618 after the emission of red light 616. The process continues with blue light emission 620 and sensing reflected electromagnetic radiation 622 after the emission of blue light 620. The process continues with one or more special 624 emission(s) and sensing reflected electromagnetic energy 626 after each of the one or more special 624 emission(s). Special emissions can include one or more individual emissions such as excitation wavelengths for fluorescent reagents, hyperspectral emissions, and / or laser mapping emissions. Each of the individual multiple special emissions can be independently sensed by the image sensor to generate separate and independent exposure frames. An image is processed and displayed at 628 based on each of the sensed reflected electromagnetic energy instances 614, 618, 622, and 626.

[0133] FIG. 6B The process shown provides a higher resolution image and provides a way for the generated RGB image to also include special data. When using a partitioned spectrum (as FIG. 6B shown), the sensor can be made sensitive to electromagnetic energy of all wavelengths. In FIG. 6B the process shown, a monochrome pixel array is instructed to sense electromagnetic energy from a predetermined partition of the full spectrum of electromagnetic energy in each cycle. Thus, to form an image, the sensor need only cycle with electromagnetic energy from multiple different partitions within the full spectrum. The final image is assembled based on the multiple cycles. Because the image from each color partition frame cycle has a higher resolution (compared to a CFA pixel array), the resulting image created when the partitioned light frames are combined also has a higher resolution. In other words, because every pixel within the array (not every other pixel, as in a sensor with a CFA) senses the amplitude of energy for a given pulse and a given scene, only a period of time apart, a higher resolution image is produced for each scene.

[0134] As can be seen graphically in the embodiment shown in FIG. 6A and FIG. 6B between times t(0) and t(l), the sensor for the partitioned spectral system in FIG. 6B cycles through at least four times in each cycle of the full spectral system in FIG. 6A In one embodiment, the display device (LCD panel) operates at a rate of 50 to 60 frames per second. In such an embodiment, the partitioned light system in FIG. 6B may operate at a rate of 200 to 240 frames per second to maintain continuity and smoothness of the displayed video. In other embodiments, there can be different capture and display frame rates. Further, the average capture rate can be any multiple of the display rate.

[0135] In one embodiment, it can be desirable that not all partitions are represented equally within the system frame rate. In other words, not all light sources must be pulsed with the same regularity in order to emphasize and de-emphasize aspects of the recorded scene as desired by the user. It should also be understood that the non-visible and visible partitions of the electromagnetic spectrum can be pulsed together within the system, where their respective data values are stitched into the video output for display to the user.

[0136] Exemplary embodiments can include the following pulsed cycle pattern:

[0137] i. green pulse;

[0138] ii. red pulse;

[0139] iii. blue pulse;

[0140] iv. green pulse;

[0141] v. red pulse;

[0142] vi. blue pulse;

[0143] vii. laser mapping pulse scheme;

[0144] viii. fluorescence excitation pulse;

[0145] ix. hyperspectral pulse;

[0146] x. (repeat)

[0147] Further exemplary embodiments can include the following pulsed cycle pattern: i. green pulse;

[0148] ii. red pulse;

[0149] iii. blue pulse;

[0150] iv. fluorescence excitation pulse;

[0151] v. hyperspectral pulse;

[0152] vi. green pulse;

[0153] vii. red pulse;

[0154] viii. blue pulse;

[0155] ix. fluorescence excitation pulse;

[0156] x. hyperspectral pulse;

[0157] xi. laser mapping pulse scheme;

[0158] xii. (repeat)

[0159] Embodiments can include the following pulse cycle pattern: i. brightness pulse;

[0160] ii. red chrominance pulse;

[0161] iii. brightness pulse;

[0162] iv. blue chrominance pulse;

[0163] v. hyperspectral pulse;

[0164] vi. fluorescence excitation pulse;

[0165] vii. laser mapping pulse;

[0166] viii. (repeat)

[0167] Embodiments can include the following pulse cycle pattern: i. brightness pulse;

[0168] ii. red chrominance pulse;

[0169] iii. brightness pulse;

[0170] iv. blue chrominance pulse;

[0171] v. brightness pulse;

[0172] vi. red chrominance pulse;

[0173] vii. brightness pulse;

[0174] viii. blue chrominance pulse;

[0175] ix. hyperspectral pulse;

[0176] x. fluorescence excitation pulse;

[0177] xi. laser mapping pulse;

[0178] xii. (repeat)

[0179] The pulse pattern can be changed to suit the imaging objectives of a particular implementation. An example imaging objective is to obtain hyperspectral imaging data and fluorescence imaging data, and also to obtain laser mapping and / or tool tracking data based on analysis of the hyperspectral and / or fluorescence imaging data. In such examples, the laser mapping and / or tool tracking data can be analyzed for certain regions of the scene that have been highlighted by the hyperspectral and / or fluorescence imaging data. A further example imaging objective is to obtain hyperspectral imaging data or fluorescence imaging data, and also to obtain laser mapping and / or tool tracking data. A further example imaging objective is to obtain laser mapping and / or tool tracking data. A further example imaging objective is to obtain hyperspectral imaging data. A further example imaging objective is to obtain fluorescence imaging data. It will be appreciated that the imaging objectives can be specific, depending on the reason for which the imaging system is deployed. Additionally, the imaging objectives can change during a single imaging session, and the pulse pattern can be changed to match the changing imaging objectives.

[0180] As can be seen in this example, the laser mapping partition can pulse at a different rate than the other partitions. Doing so can emphasize a certain aspect of the scene, where laser mapping data is layered with other data in the video output to make the desired emphasis. It should be noted that adding a laser mapping partition on top of the red, green, and blue partitions does not necessarily require a serialized system to operate at four times the rate of a full-spectrum non-serialized system, as each partition does not have to be equally represented in the pulse pattern. As seen in this implementation, adding a partition that is represented less in the pulse pattern (laser mapping in the above example) will result in less than a 20% increase in the cycle speed of the sensor to accommodate the irregular partition sampling.

[0181] In various implementations, the pulse cycle pattern can also include any of the following wavelengths in any suitable order. Such wavelengths can be particularly suitable for exciting a fluorescent reagent to generate fluorescence imaging data by sensing a relaxation emission of the fluorescent reagent based on a relaxation emission of the fluorescent reagent:

[0182] i. 770 ± 20 nm;

[0183] ii. 770 ± 10 nm;

[0184] iii. 770 ± 5 nm;

[0185] iv. 790 ± 20 nm;

[0186] v. 790 ± 10 nm;

[0187] vi. 790 ± 5 nm;

[0188] vii. 795 ± 20 nm;

[0189] viii. 795 ± 10 nm;

[0190] ix. 795 ± 5 nm;

[0191] x. 815 ± 20 nm;

[0192] xi. 815 ± 10 nm;

[0193] xii. 815 ± 5 nm;

[0194] xiii. 770 nm to 790 nm; and / or

[0195] xiv. 795 nm to 815 nm.

[0196] In various embodiments, the pulse cycle can also include any of the following wavelengths in any suitable order. Such wavelengths can be particularly suitable for generating hyperspectral imaging data:

[0197] i. 513 ± 5 nm;

[0198] ii. 565 nm to 585 nm;

[0199] iii. 900 nm to 1000 nm;

[0200] iv. 513 ± 5 nm;

[0201] v. 513 ± 10 nm;

[0202] vi. 513 ± 20 nm;

[0203] vii. 513 ± 30 nm;

[0204] viii. 513 ± 35 nm;

[0205] ix. 545 ± 5 nm;

[0206] x. 545 ± 10 nm;

[0207] xi. 545 ± 20 nm;

[0208] xii. 545 ± 30 nm;

[0209] xiii. 545 ± 35 nm;

[0210] xiv. 565 ± 5 nm;

[0211] xv. 565 ± 10 nm;

[0212] xvi.565±20nm;

[0213] xvii.565±30nm;

[0214] xviii.565±35nm;

[0215] xix.585±5nm;

[0216] xx.585±10nm;

[0217] xxi.585±20nm;

[0218] xxii.585±30nm;

[0219] xxiii.585±35nm;

[0220] xxiv.900±5nm;

[0221] xxv.900±10nm;

[0222] xxvi.900±20nm;

[0223] xxvii.900±30nm;

[0224] xxviii.900±35nm;

[0225] xxix.1000±5nm;

[0226] xxx.1000±10nm;

[0227] xxxi.1000±20nm;

[0228] xxxii. 1000±30nm; or

[0229] xxxiii.1000±35nm.

[0230] Partition loops can be divided to adapt to or approximate various imaging and video standards. In one implementation, a partition loop may include, as follows: FIGS. 7A-7D The pulses of electromagnetic energy in the red, green, and blue spectra are best shown in the image. FIG. 7A Different light intensities have been achieved by modulating the width or duration of the light pulse within the working range, indicated by the vertical gray dashed line. FIG. 7B In this process, different light intensities are achieved by modulating the power of the optical power or the power of the electromagnetic transmitter (which can be a laser or an LED transmitter), while keeping the pulse width or duration constant. FIG. 7CThe case is shown where both optical power and optical pulse width are modulated to allow for greater flexibility. The zoned cycle can use cyan magenta yellow (CMY), infrared, ultraviolet, hyperspectral, and fluorescence, which uses invisible pulse sources mixed with visible pulse sources and any other color space needed to produce an image or approximates a desired video standard currently known or yet to be developed. It should also be understood that the system is capable of switching between color spaces on the fly to provide the required image output quality.

[0231] In one embodiment, the emitter emits one or more hyperspectral emissions to elicit a spectral response. The hyperspectral emissions include one or more of electromagnetic radiation having a wavelength of about 513-545 nm, about 565-585 nm, and / or about 900-1000 nm. In such embodiments, the coherent light source 802 includes at least one laser emitter for the 513-545 nm zone, at least one laser emitter for the 565-585 zone, and at least one laser emitter for the 900-1000 nm zone. It should be understood that additional hyperspectral emissions for eliciting a spectral response can be emitted without departing from the scope of the present disclosure.

[0232] In one embodiment, the emitter emits one or more fluorescence excitation emissions for causing a reagent to fluoresce. The fluorescence excitation emissions include one or more of electromagnetic radiation having a wavelength of about 460-470 nm, 529-537 nm, 633-643 nm, 775-785 nm, 800-810 nm, 970-980 nm, 575-579 nm, 519-527 nm, 770-790 nm, and / or 795-815 nm. In such embodiments, the coherent light source 802 can include at least one laser emitter for each of the aforementioned zones of electromagnetic radiation. It should be understood that additional fluorescence excitation emissions for causing a reagent to fluoresce can be emitted without departing from the scope of the present disclosure.

[0233] In embodiments using the color space green-blue-green-red (as shown in FIG. 7D It can be desirable to pulse the luminance component more frequently than the chroma components, as users are generally more sensitive to light value differences than to light color differences. This principle can be utilized using a monochrome sensor as shown in FIG. 7D In FIG. 7D green, which contains the most luminance information, can be pulsed more frequently or with greater intensity in a (G-B-G-R-G-B-G-R...) scheme to obtain luminance data. Such a configuration will create a video stream with perceptibly more detail without creating and transmitting imperceptible data.

[0234] In one embodiment, pulsing of the weaker partition can be used to produce an output that has been adjusted for the weaker pulse. For example, blue laser light is considered weak relative to the sensitivity of a silicon-based pixel, and is difficult to produce as compared to red or green light, and thus can be pulsed more frequently during a frame cycle to compensate for the weakness of the light. These additional pulses can be done continuously over time, or by pulsing multiple lasers simultaneously to produce the desired compensation effect. It should be noted that by pulsing during a blanking period (time during which the sensor is not reading out the pixel array), the sensor is not sensitive to differences / mismatches between lasers of the same kind, and simply accumulates the light for the desired output. In another embodiment, the range of maximum light pulses can be different from frame to frame. This is shown in FIG. 7E where the light pulses are different from frame to frame. The sensor can be constructed to be able to program different blanking periods in a two-frame or three-frame or four-frame or n-frame repeating pattern.

[0235] In FIG. 7E four different light pulses are shown, and pulse 1 can repeat, for example, after pulse 4, and can have a four-frame pattern with different blanking periods. This technique can be used to place the most powerful partitions on the smallest blanking period, and thus allow the weakest partition to have a wider pulse on one of the subsequent frames without increasing the readout speed. The reconstructed frame can still have a regular pattern from frame to frame, as it is composed of many pulse frames.

[0236] FIG. 8 A graphical display of a delay or jitter between a control signal 802 and an emission 804 of electromagnetic radiation is shown. In one embodiment, the control signal 802 represents a signal provided to a driver of an emitter. The driver is configured to cause the emitter 202 to emit a pulse of electromagnetic radiation. In one embodiment, the driver is a component of the controller 204, or can be independent of the controller 204 and in communication with the controller 204. In one embodiment, the driver is the controller 204. In one embodiment, the driver is a component of or in communication with the emitter 202. As shown, there is a delay ti of a duration of time between the control signal 802 reaching its peak (i.e., turning on) and the emission 804 of electromagnetic radiation by the emitter 202. There is a delay t2 of a duration of time between the control signal 802 decreasing (i.e., turning off) and the end of the emission 804 of electromagnetic radiation.

[0237] For example, the delays ti and t2 can include some constant delay and some non-constant variation caused by jitter in the driver of the emitter. For example, there can be a constant delay when the control signal 802 is transmitted to the driver and when the emitter 202 actually emits the emission 804 of electromagnetic radiation. This delay can be very short and can be based on the time required for electrical communication to occur between the driver and the emitter. The non-constant variation of the delay can be a result of jitter in the driver of the emitter, in the controller 204, and or in the emitter itself.

[0238] The jitter experienced by a system or a component of a system, such as the driver of an emitter, can be described by a value known as a jitter specification. The jitter specification is a numerical value that describes the amount or duration of jitter experienced by the system. In FIG. 8 In the example shown, the delay ti has a shorter duration than the delay t2. In this example, the delay ti can represent the constant delay experienced after the control signal 802 is initiated and the emission 804 of electromagnetic radiation by the emitter. The difference between t2 and ti can represent the jitter experienced by the system. This value can be referred to as a jitter specification.

[0239] In one embodiment, the jitter specification is a numerical value that represents the amount of variation in the constant or predictable delay for initiating or discontinuing the emission of an electromagnetic radiation pulse. In such an embodiment, the system experiences a constant, predictable delay between the driver signaling the emitter to emit an electromagnetic radiation pulse and when the emitter actually initiates the electromagnetic radiation pulse. Similarly, there can be a constant, predictable delay between when the emitter should discontinue an electromagnetic radiation pulse and when the emitter actually discontinues the electromagnetic radiation pulse. This constant, predictable delay does not represent the jitter specification. Rather, the jitter specification is the variation in this constant, predictable delay. In FIG. 8 In the example shown, the difference between times t2 and ti represents the variation in the constant, predictable delay.

[0240] Jitter is not under the control of the user of the system. The jitter specification represents the amount of unpredictable and non-constant variation in time that is present in the system. If the jitter specification is too large relative to the electromagnetic radiation pulse, then there will be a significant reduction in image quality or image brightness variation in the resulting exposure frame. For example, in a video endoscopy system as discussed herein, a long jitter specification can cause different rows of exposure frames within a video stream to have different brightness. This results in flicker and overall quality reduction in the video stream. A long jitter specification can cause light to be emitted during the readout period 302 of the image sensor. If the electromagnetic radiation is pulsed during the readout period 302, then there will be significant variation between pixels and pixel rows in the pixel readout and this reduces the image quality in the resulting video stream.

[0241] In an example implementation, the controller 204 has a jitter specification of 10% of the duration of the pulse of electromagnetic radiation. In this example, the pulse can vary from 90% of its expected duration to 110% of its expected duration. This can result in a variation in brightness of the exposure frame or between rows within an image frame of a video of up to one-third.

[0242] In one embodiment, if the jitter specification has a duration longer than a threshold amount, the pulse of electromagnetic radiation is limited in duration to avoid overlapping into the readout period 302. The limitation on the duration of the pulse can require reducing the frame rate by increasing the time between captured exposure frames and / or increasing the duration of the blanking period 316. This can result in a reduction in image brightness, and this can further reduce the ability of the image sensor to capture detailed images.

[0243] In one embodiment, if the jitter specification has a duration shorter than a threshold amount, the pulse sequence of the emitter 202 and the readout sequence of the image sensor 204 remain unchanged. In one embodiment, the threshold indicates that the jitter specification must be 1 microsecond or less. In one embodiment, the threshold indicates that the jitter specification must be 50 nanoseconds or less. In one embodiment, the threshold indicates that the jitter specification must be less than the time taken for the image sensor to read out one row of the array of pixels. In one embodiment, the threshold indicates that the jitter specification must be less than the time taken for the image sensor to read out a single pixel of the array of pixels. In one embodiment, the threshold indicates that the jitter specification can be less than or equal to 10% to 25% of the readout period 302 of the image sensor, or the time required for the image sensor to read out all of the active pixels in the array of pixels. For example, in such an embodiment, if the array of pixels includes 400 rows, the jitter specification must be less than or equal to the time required to read out 40 to 100 of the 400 rows of the array of pixels. Thus, the amount of variation in the light captured by the array of pixels can be low enough to reduce image flicker and / or provide as much light as possible between the readout periods 302.

[0244] In one embodiment, the jitter specification is reduced (shortened) by implementing a higher clock rate or a more accurate clock in the driver of the controller 204 or the emitter 202. The reduced jitter specification and tolerance of the driver of the emitter 202 can resolve issues of intolerance to driving, resulting in artifacts in the resulting video stream.

[0245] In one embodiment, a camera control unit (CCU) provides a signal to the controller 204 or the emitter 202 to avoid overlapping the electromagnetic radiation pulses with the readout period 302 of the image sensor. The CCU can determine to send a signal to the controller 204 and / or the emitter 202 to avoid overlapping into the readout of active (i.e., not optically black) pixels in the pixel array. The CCU can maximize the duration of the time electromagnetic radiation emitted by the emitter 202 without overlapping the readout period 302 of the image sensor.

[0246] FIG. 9 A cross-section of a fiber bundle 900 for carrying electromagnetic radiation from the emitter 202 to an absence of light environment to illuminate a scene is shown. In the illustrated example embodiment, the fiber bundle 900 includes seven fibers, but it should be understood that the number of fibers is merely illustrative and any suitable number of fibers can be used without departing from the scope of the present disclosure. The fiber bundle includes a central fiber 902 and a plurality of surrounding fibers 904. FIG. 9

[0247] In one embodiment, the total number of fibers is limited to reduce the cross-sectional area of the fiber bundle 900. The fiber bundle 900 can include a suitable number of fibers for providing sufficient light dispersion while allowing for a small cross-sectional area. This can be desirable because the cross-sectional area of an endoscope lumen is critical in some applications that require a small endoscope. In one embodiment, the fiber bundle 900 can include 2 to 150 fibers. A lower number of fibers can reduce the expense and cross-sectional area required to carry the fiber bundle 900. However, a higher number of fibers improves redundancy. In one embodiment, the fiber bundle 900 includes 5 to 100 fibers or 5 to 50 fibers or 7 to 15 fibers. In one embodiment, the fiber bundle includes seven fibers, as shown. FIG. 9

[0248] When the fiber bundle 900 has a small number of fibers, it can be desirable for each fiber to receive the same amount of electromagnetic radiation and the same amount of electromagnetic radiation of a particular wavelength. For example, if electromagnetic radiation is primarily transmitted through the central fiber 902, the central fiber 902 will receive the majority of the electromagnetic radiation and the scene will be illuminated with color or brightness non-uniformity. Additionally, if more light enters one fiber than another, the total amount (power) of electromagnetic radiation that can be carried in the fiber bundle 900 can be reduced. For example, if electromagnetic radiation is provided to a fiber above a certain energy level or intensity, the fiber can have a fusing limit that can cause the fiber to melt or otherwise fail. Thus, if electromagnetic radiation is more evenly distributed across the fibers, it can be possible to increase the power and illumination of the scene.

[0249] ​​In one embodiment, the emitter 202 mixes two or more wavelengths of electromagnetic radiation before providing the electromagnetic radiation to the fiber bundle 900. This can be accomplished when the emitter 202 includes two or more independent laser beams for emitting different wavelengths of electromagnetic radiation. The emitter 202 can include, for example, a first laser beam for emitting a first wavelength and a second laser beam for emitting a second wavelength. The emitter 202 can mix the electromagnetic radiation so that light from the first laser beam and light from the second laser beam enter the jumper waveguide (or another waveguide) at the same or substantially the same angle. The same or substantially the same angle can be achieved by positioning the laser beams at the same angle relative to each other. In one embodiment, a dichroic mirror allows the same or substantially the same angle by reflecting one wavelength of electromagnetic radiation while being transparent to another wavelength. In one embodiment, the emitter 202 includes a diffuser, mixing rod, lens, or other optical element to mix the light before entering the fiber bundle 900.

[0250] In one embodiment, the emitter 202 provides a uniform distribution of light intensity to the waveguide. The peak intensity of light within an area of the waveguide that collects light can be substantially equal to or close to the average intensity of light within that area. The light provided to the collection area can have a top hat profile so that each fiber collects and / or receives the same or similar intensity of light. The emitter 202 can provide or approach a top hat profile by providing the laser light at an angle to the surface of the collection area. For example, the emitter 202 can include a Gaussian or other non-constant intensity profile. By angling the laser beams relative to the collection area, the Gaussian profile is flattened to a more constant profile or a top hat profile. A lens, diffuser, mixing rod, or similar can be used to generate the top hat profile.

[0251] FIG. 10 A top hat profile 1002 and a Gaussian profile 1004 are shown graphically. The horizontal axis represents horizontal distance and the vertical axis represents light intensity. The row of text labeled 1006 in the figure represents the boundary or width of the collection area 1006 of the fiber bundle 900. The row of text labeled 1008 in the figure represents the fusing level 1008 for the fibers or other waveguides.

[0252] With the Gaussian profile 1004, most of the electromagnetic radiation is sent to the center fiber 902. When most of the energy is in the center fiber 902, the remaining surrounding fibers 904 can be well below the fusing level 1008. For example, using the Gaussian profile 1004, an increase in the total amount of energy can result in the center fiber 902 significantly exceeding the fusing level 1008 while multiple surrounding fibers 904 are well below the fusing level 1008.

[0253] With the top hat profile 1002, all fibers carry the same energy level. This energy level can be near the fuse level 1008 or below the fuse level 1008. For example, with the top hat profile 1002, the total energy carried by the fiber bundle 900 can be significantly increased because the fiber bundle 900 can be pushed collectively near the fuse level 1008 without the risk of fusing any individual fiber.

[0254] FIG. 10 It is shown that by implementing the top hat profile 1002, more energy is provided before any individual fiber reaches the fuse level 1008. For example, the Gaussian profile 1004 and the top hat profile 1002 can provide the same amount of watts to the fiber bundle 900, yet the top hat profile 1002 can still be significantly increased before reaching the fuse level 1008. Thus, a significant improvement in the total amount of energy delivery can be achieved using plastic fibers. In some cases, a 50% or greater increase in the watts carried by the fiber bundle 900 can be achieved by implementing the top hat profile 1002. In one embodiment, the plastic fibers can have a fuse energy level for the light / electromagnetic energy emitted by one or more emitters above which the plastic fibers are damaged, where the light energy spreads out across the plurality of plastic fibers to allow the fiber bundle 900 including the plastic fibers to carry a greater amount of energy without reaching the fuse level 1008 of any of the fibers.

[0255] In one embodiment, the top hat profile 1002 and the Gaussian profile 1004 are combined by the emitter 202 for use with the plastic fiber bundle 900. The emitter 202 and / or jumper waveguide can not include plastic waveguides. However, the emitter 202 can mix the Gaussian profile 1004 with the top hat profile 1002 to allow for use at the inner cavity waveguide with the plastic fiber bundle 900. In one embodiment, mixing the top hat profile 1002 allows for greater power delivery in view of the losses that can be incurred when moving electromagnetic radiation between different materials (e.g., from a diffuser to glass fibers, to plastic fibers, and / or back to glass fibers or a diffuser). The greater power delivery can offset losses in prior or subsequent transitions so that enough light can still be delivered to illuminate a scene.

[0256] FIG. 11 is a side view showing the output from the fiber bundle 1102 compared to the camera field of view. In one embodiment, the plastic fibers have a numerical aperture of 0.63 with a field of view of 100 degrees as shown by the dashed line 1106. The glass fibers have a numerical aperture of 0.87 with a field of view of 120 degrees as shown by the solid line 1104. However, the light emitted within the field of view has an approximately Gaussian profile within a cone that is less than the field of view. For example, almost all of the light for the plastic fibers can be within an 80 degree cone as shown by the dash-dot line 1108. Thus, the center region of the exposure frame can be too bright and the edges too dark.

[0257] FIG. 12 It shows relative to FIG. 11 The output shown is a side view of the output from fiber bundle 1202, exhibiting a more uniform light distribution. FIG. 12 In the illustrated embodiment, uniform light distribution is achieved by aiming the light off the ends of the fibers in the fiber bundle 1202. Aiming the fibers away from the center widens the cones in the field of view and eliminates light loss at the output ends. One end of each fiber can be held in a desired position to distribute light, where the combination of light cones from the fibers provides uniform illumination. The fiber bundle 1202 includes multiple fibers and rows 1204 indicating the orientation of the cones output from individual fibers. In one embodiment, a fixing device such as a physical mold or a sheet with holes holds the ends of the fibers in a desired orientation. The fibers can be oriented to the optimal orientation for uniform illumination of the scene. The ends of the fibers in the fiber bundle 1202 can be located near the distal end of an endoscope and can be directed to propagate light around an area centered on the focal point or camera lens axis.

[0258] FIG. 13 This is a side view showing the output of the fiber bundle 1302, which transitions from plastic fiber 1304 to glass fiber 1306 at connector 1308. In this embodiment, the cavity waveguide includes plastic fiber 1304, which then transitions to glass fiber 1306 at or near the output. Glass fiber 1306 typically has a higher numerical aperture and a wider field of view than plastic fiber 1304. Therefore, as shown by light cone 1310, a wider and more uniform light energy distribution is achieved. Light passing through plastic fiber 1304 is guided to glass fiber 1306 via connector 1308. This coupling can occur within the handpiece unit of the endoscope's cavity. Connector 1308 can be positioned in the handpiece or cavity to limit the amount of glass fiber 1306 used. Moving from plastic fiber 1304 to glass fiber 1306 via a tapered member in the handpiece or cavity produces the same field of view as conventional endoscopes. However, compared to the targeting implementation (which does not experience optical loss at the output), optical loss can be significant, such as about 25%.

[0259] FIG. 14 This is a side view showing the light output from the fiber bundle 1402 using a diffuser 1408. In this embodiment, the cavity waveguide includes plastic fibers 1404, which then transition to the diffuser 1408 at or near the output. The diffuser 1408 can include any suitable optical diffuser, such as a hybrid rod. Exemplary diffusers include holographic diffusers. The diffuser 1408 at the output can produce a larger field of view compared to glass fibers. However, the diffuser 1408 is less efficient, such as compared to... FIG. 12 The aiming implementation shown is approximately 40% to 60% more efficient.

[0260] Plastic fibers are generally less expensive than glass fibers. The reduced price can result in significant savings in manufacturing the illumination system. Because glass can only be used for a short distance near the output, or not used at all, significant cost savings can be achieved.

[0261] In one embodiment, a single fiber replaces the bundle of optical fibers. The single fiber can be larger than the typical fibers that make up the bundle of optical fibers, enabling the single fiber to handle more power than a smaller bundle of fibers for the same cross-sectional area. The single fiber can extend from the console and through the lumen to provide light to the interior of the body or other light deficient environment. The single fiber can operate as a lumen waveguide that extends from the emitter 202 or jumper waveguide and through the lumen. Electromagnetic radiation can be provided directly to the single fiber with a tip hat profile by the emitter 202.

[0262] Because plastic fibers can only have a numerical aperture of 0.63 or 0.65, most of the electromagnetic radiation can only be emitted at an angle of 70 or 80 degrees. At the output of the single fiber, a diffuser can be positioned to spread the output light and create more uniform illumination within the field of view of the camera. In one embodiment, the type of diffuser or the presence of a diffuser can be based on the field of view used by the camera. For example, laparoscopic surgery can allow for a narrower field of view, such as 70 degrees, while arthroscopic surgery can use a wider field of view, such as 110 degrees. Thus, a diffuser can be used for arthroscopy, while a diffuser can not be present for laparoscopy.

[0263] It should be appreciated that embodiments for outputting electromagnetic radiation (light) can include a combination of one or more of the embodiments shown in FIGS. 1-8. For example, plastic fibers can be converted to glass fibers, and the glass fibers can be intended to provide more uniform and improved illumination. FIGS. 11-14

[0264] FIG. 15 is a schematic flowchart illustrating an example method 1500 for providing light to an imaged scene in a light deficient environment. The method 1500 can be performed by an illumination system, such as the system 100 of FIG. 1. FIG. 1

[0265] The method 1500 begins, and an image sensor generates and reads out pixel data for an image from the image sensor based on light received by the image sensor at 1502, where a length of time to read out a row of pixel data comprises a row readout length. An emitter emits light for illuminating a scene viewed by the image sensor at 1504. A driver drives the emitter to emit at 1506, where the driver comprises a dithering specification that is less than or equal to the row readout length. A controller controls the driver to drive the emitter to generate a pulse of light between readout periods of the image sensor at 1508.

[0266] ​​FIG. 16 is a schematic flow chart illustrating an example method 1600 for providing light to an imaging scene in a light deficient environment. The method 1600 can be performed by an illumination system, such as the system 100 of FIG. 1 .

[0267] The method 1600 begins and a first emitter and a second emitter emit light including a first wavelength and a second wavelength at 1602. A plurality of optical fibers direct the light generated by the first emitter and the second emitter to a scene in an endoscopic environment at 1604. The plurality of optical fibers receive substantially equal amounts of light (mixed light) from the first emitter and the second emitter at each of the plurality of optical fibers at 1606a.

[0268] FIGS. 17A-17C Each illustrates a light source 1700 having a plurality of emitters. The plurality of emitters can alternatively be referred to as“laser beams,” where each emitter / laser beam can operate independently of the other emitters / laser beams and / or pulse different sections or wavelengths of the electromagnetic spectrum. The light source 1700 can be collectively referred to herein as“emitters.” The plurality of emitters includes a first emitter 1702, a second emitter 1704, and a third emitter 1706. Additional emitters can be included, as discussed further below. The emitters 1702, 1704, and 1706 can include one or more laser emitters that emit light having different wavelengths. For example, the first emitter 1702 can emit wavelengths consistent with blue laser light, the second emitter 1704 can emit wavelengths consistent with green laser light, and the third emitter 1706 can emit wavelengths consistent with red laser light. For example, the first emitter 1702 can include one or more blue lasers, the second emitter 1704 can include one or more green lasers, and the third emitter 1706 can include one or more red lasers. The lasers 1702, 1704, 1706 emit laser beams toward a collection region 1708, which can be a location of a waveguide, a lens, or other optical component for collecting light and / or providing light to a jumper waveguide 206 or a hollow core waveguide 210 of FIG. 2 .

[0269] In one implementation, the emitters 1702, 1704, and 1706 emit electromagnetic radiation at hyperspectral wavelengths. Certain hyperspectral wavelengths can penetrate tissue and enable a practitioner to“see through” tissue in the foreground to identify chemical processes, structures, compounds, biological processes, etc. located behind the tissue in the foreground. The hyperspectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc. that are known to have a particular spectral response.

[0270] In implementations in which an agent or dye that helps identify certain tissues, structures, chemical reactions, biological processes, etc. has been administered to a patient, the emitters 1702, 1704, and 1706 can emit wavelengths that are used to cause the agent or dye to fluoresce. Such wavelengths can be determined based on the agent or dye administered to the patient. In such embodiments, the emitters can need to be highly precise in order to emit the required wavelengths to cause certain agents or dyes to fluoresce or activate.

[0271] In one implementation, the emitters 1702, 1704, and 1706 emit a laser mapping pattern for mapping the topology of a scene and / or for calculating the size and distance between objects in a scene. In one embodiment, an endoscopic imaging system is used in conjunction with multiple tools such as a scalpel, a retractor, forceps, etc. In such embodiments, each of the emitters 1702, 1704, and 1706 can emit a laser mapping pattern such that the laser mapping pattern is projected onto each tool individually. In such embodiments, the laser mapping data for each of the tools can be analyzed to identify the distance between the tool and other objects in the scene.

[0272] In FIG. 17B In embodiments of the system 1700, the emitters 1702, 1704, 1706 each deliver laser light to the collection region 1708 at different angles. The variation in angles can result in a variation in the location of the electromagnetic energy in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at the collection region 1708, the varying angles can result in different amounts of light entering different fibers. For example, the angles can result in a variation in intensity across the collection region 1708. In addition, the light from the different emitters can not be mixed uniformly, so some fibers can receive different amounts of different colors of light. The variation in color or intensity of the light in different fibers can result in non-optimal illumination of the scene. For example, the variation in delivered light or light intensity can result in the scene and the captured image.

[0273] In one embodiment, an intervening optical element can be placed between the fiber bundle and the emitters 1702, 1704, 1706 to mix the different colors (wavelengths) of light before entering the fibers or other waveguide. Exemplary intervening optical elements include a diffuser, a mixing rod, one or more lenses, or other optical components for mixing the light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle can have the same color. This mixing can result in the same color in each fiber, but, in some embodiments, can still result in different total brightness delivered to different fibers. In one embodiment, the intervening optical element can also spread or uniformly spread the light over the collection region so that each fiber carries the same total amount of light (e.g., the light can spread out in a top hat profile). A diffuser or mixing rod can result in loss of light.

[0274] Although the collection region 1708 is shown as a physical component in FIG. 17A , the collection region 1708 can simply be the region that delivers light from the emitters 1702, 1704, and 1706. In some cases, the collection region 1708 can include optical components such as a diffuser, mixing rod, lens, or any other intervening optical components between the emitters 1702, 1704, 1706 and the output waveguide.

[0275] FIG. 17C An embodiment of a light source 1700 is shown with emitters 1702, 1704, 1706 providing light to the collection region 1708 at the same or substantially the same angle. The light is provided at an angle that is substantially normal to the collection region 1708. The light source 1700 includes a plurality of dichroic mirrors including a first dichroic mirror 1710, a second dichroic mirror 1712, and a third dichroic mirror 1714. The dichroic mirrors 1710, 1712, 1714 include mirrors that reflect light of a first wavelength but transmit light of a second wavelength (or are transparent to it). For example, the third dichroic mirror 1714 can reflect blue laser light provided by the third emitter, while being transparent to the red and green light provided by the first and second emitters 1702, 1704, respectively. The second dichroic mirror 1712 can be transparent to the red light from the first emitter 1702, but reflective to the green light from the second emitter 1704. If other colors or wavelengths are included, the dichroic mirrors can be selected to reflect light corresponding to at least one emitter and be transparent to the other emitters. For example, the third dichroic mirror 1714 reflects light from the third emitter 1706, but is transparent to emitters behind it, such as the first and second emitters 1702, 1704. In embodiments where there are tens or hundreds of emitters, each dichroic mirror can reflect the corresponding emitter and the emitters in front of it, while being transparent to the emitters behind it. This can allow tens or hundreds of emitters to emit electromagnetic energy at substantially the same angle to the collection region 1708.

[0276] Because these dichroic mirrors allow other wavelengths to be transmitted or pass through, each of these wavelengths can reach the collection region 1708 from the same angle and / or at the same center point or focal point. Providing light from the same angle and / or the same focal / center point can significantly improve the reception and color mixing at the collection region 1708. For example, a particular fiber can receive different colors in the same proportion as they are transmitted / reflected by the emitters 1702, 1704, 1706 and the mirrors 1710, 1712, 1714. In contrast to embodiments of FIG. 17B , the light mixing can be significantly improved at the collection region. In one embodiment, any of the optical components discussed herein can be used at the collection region 1708 to collect the light before providing it to the fibers or fiber bundle.

[0277] FIG. 17C An embodiment of a light source 1700 is shown with emitters 1702, 1704, 1706 that also provide light to the collection region 1708 at the same or substantially the same angle. For example, the light incident on the collection region 1708 is offset from the vertical by an angle 1716. In one embodiment, the laser emitters 1702, 1704, 1706 can have a Gaussian cross-sectional intensity profile. As previously described, improved distribution of light energy between fibers can be achieved by forming a more flat or top-hat shaped intensity profile. In one embodiment, as the angle 1716 increases, the intensity across the collection region 1708 approaches a top-hat profile. For example, by increasing the angle 1716 until the profile is sufficiently flat, the top-hat profile can even approximate a non-flat output beam. The top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intervening optical components between the emitters 1702, 1704, 1706 and the output waveguide, fiber, or bundle of optical fibers.

[0278] FIG. 18 is a schematic diagram showing a single optical fiber 1802 outputting at the output via a diffuser 1804. In one embodiment, the optical fiber 1802 has a diameter of 500 microns, a numerical aperture of 0.65, and emits a light cone 1806 of about 70 or 80 degrees without the diffuser 1804. With the diffuser 1804, the light cone 1806 can have an angle of about 110 or 120 degrees. The light cone 1806 can be the majority of the place where all the light reaches and is uniformly distributed. The diffuser 1804 can allow for a more uniform distribution of electromagnetic energy of a scene observed by an image sensor.

[0279] In one embodiment, the lumen waveguide 210 includes a single plastic or glass optical fiber of about 500 microns. Plastic fibers are less expensive, but their width by coupling, diffusing, or other losses can allow the fiber to carry a sufficient amount of light to the scene. For example, a smaller fiber can not be able to carry as much light or power as a larger fiber. The lumen waveguide 210 can include a single or multiple optical fibers. The lumen waveguide 210 can receive light directly from a light source or via a jumper waveguide. A diffuser can be used to widen the light output 206 to obtain a desired field of view of the image sensor 214 or other optical components.

[0280] While in FIGS. 17A-17CThree emitters are shown, but in some embodiments, a number of emitters ranging from one to hundreds or more can be used. The emitters can have different wavelengths or spectrums of light that they emit, and these lights can be used to continuously cover a desired portion of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectrums). The emitters can be configured to emit visible light such as red, green, and blue light, and can also be configured to emit hyperspectral emissions of electromagnetic radiation, fluorescence excitation wavelengths for causing reagents to fluoresce, and / or laser mapping modes for calculating parameters and distances between objects in a scene.

[0281] FIG. 19 A portion of the electromagnetic spectrum 1900 is shown that is divided into twenty different sub-spectrums. The number of sub-spectrums is merely exemplary. In at least one embodiment, the spectrum 1900 can be divided into hundreds of sub-spectrums, each having a small waveband. The spectrum can extend from the infrared spectrum 1902, through the visible spectrum 1904, and into the ultraviolet spectrum 1906. The sub-spectrums each have a waveband 1908 that covers a portion of the spectrum 1900. Each waveband can be defined by an upper wavelength and a lower wavelength.

[0282] Hyperspectral imaging includes imaging information from across the electromagnetic spectrum 1900. A hyperspectral pulse of electromagnetic radiation can include multiple sub-pulses across one or more portions of the electromagnetic spectrum 1900 or the entire electromagnetic spectrum 1900. A hyperspectral pulse of electromagnetic radiation can include a single wavelength bin of electromagnetic radiation. The resulting hyperspectral exposure frame includes information sensed by the pixel array after the hyperspectral pulse of electromagnetic radiation. Thus, a hyperspectral exposure frame can include data for any suitable binning of the electromagnetic spectrum 1900, and can include multiple exposure frames for multiple binning of the electromagnetic spectrum 1900. In one embodiment, the hyperspectral exposure frame includes multiple hyperspectral exposure frames such that the combined hyperspectral exposure frames include data for the entire electromagnetic spectrum 1900.

[0283] In one embodiment, at least one emitter, such as a laser emitter, is included in the light source, such as light source 202, 1700, for each sub-spectrum to provide complete and continuous coverage of the entire optical spectrum 1900. For example, a light source for providing coverage of the illustrated sub-spectra can include at least 20 different emitters, one for each sub-spectrum. In one embodiment, each emitter covers a spectrum that covers 40 nanometers. For example, one emitter can emit light within a waveband of 500 nm to 540 nm, while another emitter can emit light within a waveband of 540 nm to 580 nm. In another embodiment, the emitters can cover other sized wavebands, depending on the types of emitters available or the imaging needs. For example, a plurality of emitters can include a first emitter covering a waveband of 500 nm to 540 nm, a second emitter covering a waveband of 540 nm to 640 nm, and a third emitter covering a waveband of 640 nm to 650 nm. Each emitter can cover a different segment of the electromagnetic spectrum ranging from far infrared, mid infrared, near infrared, visible light, near ultraviolet, and / or far ultraviolet. In some cases, multiple emitters of the same type or wavelength can be included to provide sufficient output power for imaging. The number of emitters needed for a particular waveband can depend on the sensitivity of the monochromatic sensor to the waveband and / or the power output capability of the emitters in that waveband.

[0284] The waveband width and coverage provided by the emitters can be selected to provide any desired combination of the optical spectrum. For example, continuous coverage of the optical spectrum using very small waveband widths (e.g., 10 nm or less) can allow for highly selective hyperspectral and / or fluorescence imaging. The waveband width can allow for selective emission of excitation wavelengths for one or more specific fluorescent reagents. Additionally, the waveband width can allow for selective emission of certain portions of the hyperspectral electromagnetic radiation for identifying specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths come from emitters that can be selectively activated, extreme flexibility in causing one or more specific fluorescent reagents to fluoresce during an examination can be achieved. Additionally, extreme flexibility in identifying one or more objects or processes through hyperspectral imaging can be achieved. Thus, more fluorescence and / or hyperspectral information can be achieved in less time and within a single examination, which would otherwise require multiple examinations, delays due to application or staining of dyes, etc.

[0285] FIG. 20is a schematic diagram showing a timing diagram 2000 for generating emission and readout of images. The solid lines represent readout periods (peaks 2002) and blanking periods (valleys) for capturing a series of exposure frames 2004-2014. The series of exposure frames 2004-2014 can include a series of repeated exposure frames that can be used to generate laser mapping data, hyperspectral data, and / or fluorescence data that can be overlaid on an RGB video stream. In one embodiment, a single image frame includes information from multiple exposure frames, where one exposure frame includes red image data, another exposure frame includes green image data, and another exposure frame includes blue image data. Additionally, a single image frame can include one or more of hyperspectral image data, fluorescence image data, and laser mapping data. The multiple exposure frames are combined to produce a single image frame. The single image frame is an RGB image with hyperspectral imaging data. The series of exposure frames includes a first exposure frame 2004, a second exposure frame 2006, a third exposure frame 2008, a fourth exposure frame 2010, a fifth exposure frame 2012, and an Nth exposure frame 2026.

[0286] Additionally, the hyperspectral image data, fluorescence image data, and laser mapping data can be used in combination to identify key tissues or structures and further measure the dimensions of those key tissues or structures. For example, the hyperspectral image data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, cancerous tissue, etc. The location and identification of the key structures can be received from the corresponding system and also used to generate a topology of the key structures using the laser mapping data. For example, the corresponding system determines the location of a cancerous tumor based on the hyperspectral imaging data. Since the location of the cancerous tumor is known based on the hyperspectral imaging data, the topology and distance of the cancerous tumor can then be calculated based on the laser mapping data. This example can also apply when a cancerous tumor or other structure is identified based on fluorescence imaging data.

[0287] In one embodiment, each exposure frame is generated based on at least one pulse of electromagnetic energy. The pulses of electromagnetic energy are reflected and detected by the image sensor and then read out in a subsequent readout (2002). Thus, each blanking period and readout results in an exposure frame for a particular electromagnetic energy spectrum. For example, the first exposure frame 2004 can be generated based on the spectrum of a first one or more pulses 2016, the second exposure frame 2006 can be generated based on the spectrum of a second one or more pulses 2018, the third exposure frame 2008 can be generated based on the spectrum of a third one or more pulses 2020, the fourth exposure frame 2010 can be generated based on the spectrum of a fourth one or more pulses 2022, the fifth exposure frame 2012 can be generated based on the spectrum of a fifth one or more pulses, and the Nth exposure frame 2026 can be generated based on the spectrum of an Nth one or more pulses 2026.

[0288] Pulses 2016-2026 can include energy from a single emitter or a combination from two or more emitters. For example, the spectra included in a single readout cycle or within multiple exposure frames 2004-2014 can be selected for a desired examination or detection of a particular tissue or condition. According to one embodiment, one or more pulses can include visible spectrum light for generating an RGB or black and white image, while one or more additional pulses are emitted to sense a spectral response of electromagnetic radiation for hyperspectral wavelengths. For example, pulse 2016 can include red light, pulse 2018 can include blue light, and pulse 2020 can include green light, while remaining pulses 2022-2026 can include wavelengths and spectra for detecting particular tissue types, fluorescing reagents, and / or mapping scene topography. As another example, the pulses of a single readout cycle include spectra (e.g., different segments of the electromagnetic spectrum) generated by multiple different emitters that can be used to detect a particular tissue type. For example, if the combination of wavelengths causes a pixel to have a value that exceeds or falls below a threshold, the pixel can be classified as corresponding to a particular type of tissue. Each frame can also be used to narrow down the type of tissue present at that pixel (e.g., as well as each pixel in the image) to provide a very specific classification of the tissue and / or the state (diseased / healthy) of the tissue based on the spectral response of the tissue and / or whether a fluorescent reagent is present at the tissue.

[0289] Multiple frames 2004-2014 are shown with different lengths of readout cycles and with pulses of different lengths or intensities. The blanking periods, pulse lengths or intensities, etc. can be selected based on the sensitivity of the monochromatic sensor to particular wavelengths, the power output capability of the emitters, and / or the carrying capacity of the waveguide.

[0290] In one embodiment, a dual image sensor can be used to obtain a three- dimensional image or video feed. Three-dimensional examination can allow for improved understanding of the three-dimensional structure of an examination region and mapping of different tissue or substance types within the region.

[0291] In one example implementation, a patient is provided with a fluorescent reagent, and the fluorescent reagent is configured to attach to cancer cells. The fluorescent reagent is known to fluoresce when irradiated by a particular partition of electromagnetic radiation. The relaxation wavelength of the fluorescent reagent is also known. In the example implementation, the patient is imaged with an endoscopic imaging system as discussed herein. The endoscopic imaging system pulses partitions of red, green, and blue wavelengths of light to generate an RGB video stream of the interior of the patient's body. Additionally, the endoscopic imaging system pulses an electromagnetic radiation excitation wavelength of the fluorescent reagent administered to the patient. In the example, the patient has cancer cells, and the fluorescent reagent attaches to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength of the fluorescent reagent, the fluorescent reagent will fluoresce and emit the relaxation wavelength. If cancer cells are present in the scene imaged by the endoscopic imaging system, then the fluorescent reagent will also be present in the scene, and will emit its relaxation wavelength after fluorescing due to the emission of the excitation wavelength. The endoscopic imaging system senses the relaxation wavelength of the fluorescent reagent, thereby sensing the presence of the fluorescent reagent in the scene. Because the fluorescent reagent is known to attach to cancer cells, the presence of the fluorescent reagent also indicates the presence of cancer cells within the scene. The endoscopic imaging system thereby identifies the location of the cancer cells within the scene. The endoscopic imaging system can also emit a laser mapping pulse scheme for generating a topology of the scene and calculating dimensions of objects within the scene. The location of the cancer cells, as identified by the fluorescent imaging data, can be combined with the topology and dimension information calculated based on the laser mapping data. Thus, the precise location, size, dimensions, and topology of the cancer cells can be identified. This information can be provided to a medical practitioner to aid in resecting the cancer cells. Additionally, this information can be provided to a robotic surgical system to enable the surgical system to resect the cancer cells.

[0292] In another example implementation, a patient is imaged with an endoscopic imaging system to identify quantitative diagnostic information about the patient's tissue pathology. In this example, the patient is suspected of or known to have a disease that can be tracked with hyperspectral imaging to observe the progression of the disease in the patient's tissue. The endoscopic imaging system pulses light of red, green, and blue wavelengths to generate an RGB video stream of the inside of the patient's body. Additionally, the endoscopic imaging system pulses light of one or more hyperspectral wavelengths that allow the system to "see through" some tissue and generate an image of the tissue affected by the disease. The endoscopic imaging system senses the reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue, identifying the location of the diseased tissue within the patient's body. The endoscopic imaging system can also emit a laser mapping pulse scheme to generate a topology of the scene and calculate the dimensions of objects within the scene. The location of the diseased tissue, as identified by the hyperspectral imaging data, can be combined with the topology and dimensional information calculated with the laser mapping data. Thus, the precise location, size, dimensions, and topology of the diseased tissue can be identified. This information can be provided to a medical practitioner to assist in resecting, imaging, or studying the diseased tissue. Additionally, this information can be provided to a robotic surgical system to enable the surgical system to resect the diseased tissue.

[0293] FIG. 21 is a schematic diagram of an imaging system 2100 with a single cut filter. The system 2100 includes an endoscope 2106 or other suitable imaging device with a light source 2108 for a light deficient environment. The endoscope 2106 includes an image sensor 2104 and a filter 2102 to filter out unwanted wavelengths of light or other electromagnetic radiation before reaching the image sensor 2104. The light source 2108 transmits light that can illuminate a surface 2112 in a light deficient environment such as a body cavity. Light 2110 reflects off the surface 2112 and passes through the filter 2102 before hitting the image sensor 2104.

[0294] The filter 2102 can be used in implementations where a fluorescent reagent or dye has been applied. In such embodiments, the light source 2108 emits an excitation wavelength for causing the fluorescent reagent or dye to fluoresce. Typically, the relaxation wavelength emitted by the fluorescent reagent or dye will have a different wavelength than the excitation wavelength. The filter 2102 can be selected to filter out the excitation wavelength and only allow the relaxation wavelength to pass through the filter and be sensed by the image sensor 2104.

[0295] In one embodiment, the optical filter 2102 is configured to filter out excitation wavelengths of electromagnetic radiation that cause a reagent or dye to fluoresce, such that only the intended relaxation wavelengths of the fluorescing reagent or dye are allowed to pass through the optical filter 2102 and reach the image sensor 2104. In one embodiment, the optical filter 2102 filters out fluorescent reagent excitation wavelengths of at least between 770 nm and 790 nm. In one embodiment, the optical filter 2102 filters out fluorescent reagent excitation wavelengths of at least between 795 nm and 815 nm. In one embodiment, the optical filter 2102 filters out fluorescent reagent excitation wavelengths of at least between 770 nm and 790 nm and between 795 nm and 815 nm. In these embodiments, the optical filter 2102 filters out the excitation wavelengths of the reagent and allows only the relaxation wavelengths of the fluorescent reagent to be read by the image sensor 2104. The image sensor 2104 can be a wavelength-agnostic image sensor, and the optical filter 2102 can be configured to allow the image sensor 2104 to receive only the relaxation wavelengths of the fluorescent reagent and not the emission excitation wavelengths of the reagent. The data determined by the image sensor 2104 can then be indicative of the presence of a key bodily structure, tissue, biological process, or chemical process determined by the location of the reagent or dye.

[0296] The optical filter 2102 can also be used in implementations where a fluorescent reagent or dye has not been applied. The optical filter 2102 can be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read by the image sensor 2104. The image sensor 2104 can be a monochromatic image sensor, such that pixels of a captured image above or below a threshold value can be characterized as corresponding to certain spectral responses or fluorescent emissions. The spectral responses or fluorescent emissions determined by the pixels captured by the image sensor 2104 can be indicative of the presence of certain bodily tissues or structures, certain conditions, certain chemical processes, and the like.

[0297] FIG. 22 is a schematic diagram of an imaging system 2200 having multiple cut filters. The system 2200 includes a scope 2206 or other suitable imaging device having a light source 2208 for a light deficient environment. The scope 2206 includes an image sensor 2204 and two optical filters 2202a, 2202b. It should be understood that in alternative embodiments, the system 2200 can include any number of optical filters, and the number of optical filters and the type of optical filters can be selected for certain purposes, e.g., to gather imaging information of particular bodily tissues, bodily conditions, chemical processes, and the like. The optical filters 2202a, 2202b are configured to prevent the image sensor 2204 from sensing light or other electromagnetic radiation of unwanted wavelengths. The optical filters 2202a, 2202b can be configured to filter out unwanted wavelengths from white light or other electromagnetic radiation that can be emitted by the light source 2208.

[0298] With respect toFIG. 21 The present disclosure also describes that filters 2202a and 2202b can be used in embodiments where a fluorescent reagent or dye has been applied. Filters 2202a and 2202b can be configured to block the emission excitation wavelength of the reagent or dye and allow the image sensor 2204 to read only the relaxation wavelength of the reagent or dye. Furthermore, filters 2202a and 2202b can be used in embodiments where a fluorescent reagent or dye has not been applied. In such embodiments, filters 2202a and 2202b can be selected to allow wavelengths corresponding to the desired spectral response to pass through and be read by the image sensor 2204.

[0299] Multiple filters 2202a, 2202b can each be configured to filter out wavelengths of different ranges in the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than the desired wavelength range, and additional filters can be configured to filter out wavelengths shorter than the desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 2204.

[0300] In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 513 nm and 545 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 565 nm and 585 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 900 nm and 1000 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 421 nm and 475 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 520 nm and 545 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 617 nm and 645 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 760 nm and 795 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 795 nm and 815 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 370 nm and 420 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is customized such that electromagnetic radiation between 600 nm and 670 nm contacts the image sensor 2204. In an embodiment, the optical filter 2202a, 2202b is configured to allow only certain fluorescent relaxation emissions to pass through the optical filter 2202a, 2202b and contact the image sensor 2204. In an embodiment, the first optical filter blocks electromagnetic radiation having a wavelength of about 770 nm to about 790 nm, and the second optical filter blocks electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0301] In one embodiment, the system 2200 includes multiple image sensors 2204, and can specifically include two image sensors for generating three-dimensional images. The image sensors 2204 can be color / wavelength agnostic, and configured to read electromagnetic radiation of any wavelength reflected from the surface 2212. In one embodiment, the image sensors 2204 are each color-dependent or wavelength-dependent, and configured to read electromagnetic radiation of a specific wavelength reflected from the surface 2212 and back to the image sensors 2204. Alternatively, the image sensors 2204 can include a single image sensor with multiple different pixel sensors configured to read different wavelengths or colors of light, such as a Bayer filter color filter array. Alternatively, the image sensors 2204 can include one or more color agnostic image sensors, which can be configured to read different wavelengths of electromagnetic radiation according to a pulse schedule, such as those shown in FIGS. 1-2. FIGS. 5-7E

[0302] FIG. 23 is a schematic diagram illustrating a system 2300 for mapping a surface and / or tracking an object in a light deficient environment by laser mapping imaging. In one embodiment, an endoscope 2306 pulses a grid array 2306 (which can be referred to as a laser mapping pattern) onto a surface 2304 in a light deficient environment. In one embodiment as shown in FIG. 23, FIG. 23 In one embodiment, the grid array 2306 includes vertical hashes 2308 and horizontal hashes 2310. It should be understood that the grid array 2306 can include any suitable array for mapping the surface 2304, including for example a raster grid of discrete points, an occupancy grid map, a dot array, etc. Additionally, the endoscope 2306 can pulse multiple grid arrays 2306, and can for example pulse one or more individual grid arrays on each of multiple objects or structures within the light deficient environment.

[0303] In one embodiment, the system 2300 pulses the grid array 2306, which can be used to map the three-dimensional topology of a surface and / or track the location of an object such as a tool or another device in a light deficient environment. In one embodiment, the system 2300 provides data to a third party system or computer algorithm for determining surface dimensions and configurations by way of light detection and ranging (LIDAR) mapping. The system 2300 can pulse light or electromagnetic radiation of any suitable wavelength in the grid array 2306, including for example ultraviolet light, visible light, and / or infrared or near infrared light. The surface 2304 and / or objects within the environment can be mapped and tracked with very high resolution and with very high accuracy and precision.

[0304] ​In one embodiment, system 2300 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. System 2300 can include a light engine having an emitter that generates one or more pulses of electromagnetic radiation and a lumen that transmits the one or more pulses of electromagnetic radiation to a distal tip of an endoscope within a light deficient environment, such as a body cavity. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation includes a laser mapping pattern that is emitted onto a surface within the light deficient environment, such as a surface of body tissue within the body cavity and / or a surface of a tool or other device. Endoscope 2306 can include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations within the light deficient environment.

[0305] In one embodiment, system 2300 includes a processor for determining a distance of an endoscope or tool from an object, such as surface 2304. The processor can also determine an angle between the endoscope or tool and the object. The processor can also determine surface area information about the object, including, for example, dimensions of a surgical tool, dimensions of a structure, dimensions of an anatomical structure, location information, and other location data and metrics. System 2300 can include one or more image sensors that provide image data to a control system for determining a distance of an endoscope or tool from an object, such as surface 2304. The image sensors can output information to a control system for determining an angle between the endoscope or tool and the object. Additionally, the image sensors can output information to a control system for determining surface area information about the object, dimensions of a surgical tool, dimensions of a structure, dimensions of an anatomical structure, location information, and other location data and metrics.

[0306] In one embodiment, the emitter of the endoscope 2306 pulses the grid array 2306 at a sufficient speed such that the grid array 2306 is not visible to the user. In various implementations, seeing the grid array 2306 during an endoscopic imaging procedure and / or endoscopic surgery can distract the user. The grid array 2306 can be pulsed at a sufficiently short period such that the grid array 2306 cannot be detected by the human eye. In alternative embodiments, the endoscope 2306 pulses the grid array 2306 at a sufficient recurrence frequency such that the grid array 2306 is viewable by the user. In such embodiments, the grid array 2306 can be superimposed on the image of the surface 2304 on the display. The grid array 2306 can be superimposed on the black and white or RGB image of the surface 2304 such that the grid array 2306 is visible to the user during use of the system 2300. The user of the system 2300 can indicate whether the grid array 2306 should be superimposed on the image of the surface 2304 and / or whether the grid array 2306 should be visible to the user. The system 2300 can include a display that provides real-time measurements of the distance from the endoscope 2306 to the surface 2304 or another object within the light deficient environment. The display can also provide real-time surface area information about the surface 2304 and / or any objects, structures, or tools within the light deficient environment. The accuracy of the measurements can be precise to less than one millimeter.

[0307] In one embodiment, the system 2300 pulses multiple grid arrays 2306. In one embodiment, each of the multiple grid arrays 2306 corresponds to a tool or other device present within the light deficient environment. The precise position and parameters of each of the tools and other devices can be tracked by pulsing and sensing the multiple grid arrays 2306. The information generated by sensing the reflected grid arrays 2306 can be evaluated to identify the relative positions of the tools and other devices within the light deficient environment.

[0308] The endoscope 2306 can pulse electromagnetic radiation according to pulse schedules such as those shown herein, which can also include pulsing the grid array 2306 and pulsing red, green, and blue light for generating RGB images and further generating grid arrays 2306 that can be superimposed on the RGB images and / or for mapping and tracking surfaces 2304 and objects within the light deficient environment. The grid array 2306 can additionally be pulsed in conjunction with hyperspectral or fluorescence excitation wavelengths of electromagnetic radiation. Data from each of RGB imaging, laser mapping imaging, hyperspectral imaging, and fluorescence imaging can be combined to identify the location, size, and surface topology of key structures within the body.

[0309] In one embodiment, the endoscope 2306 includes one or more color agnostic image sensors. In one embodiment, the endoscope 2306 includes two color agnostic image sensors for generating three-dimensional images or maps of the light deficient environment. The image sensors can generate RGB images of the light deficient environment according to a pulse schedule as disclosed herein. Additionally, the image sensors can determine data for mapping the light deficient environment and tracking one or more objects within the light deficient environment based on data determined at the pulse grid array 2306. Additionally, the image sensors can determine spectral or hyperspectral data and fluorescence imaging data according to a pulse schedule, which can be modified by a user to suit the particular needs of the imaging procedure. In one embodiment, the pulse schedule includes red, green, and blue pulses and pulses of the grid array 2306 and / or pulses for generating hyperspectral image data and / or fluorescence image data. In various implementations, the pulse schedule can include any suitable combination of pulses of electromagnetic radiation according to the needs of the user. The frequency of repetition of different wavelengths of electromagnetic radiation can be determined based on, for example, the energy of certain pulses, the needs of the user, whether certain data (e.g., hyperspectral data and / or fluorescence imaging data) needs to be continuously updated or can be updated less frequently, etc.

[0310] The pulse schedule can be modified in any suitable manner and certain pulses of electromagnetic radiation can be repeated at any suitable frequency according to the needs of the user or a computer-implemented program for certain imaging procedures. For example, in embodiments in which surface tracking data generated based on the grid array 2306 is provided to a computer-implemented program for, e.g., robotic surgery, the grid array 2306 can be pulsed more frequently than if the surface tracking data is provided to a user who visualizes the scene during the imaging procedure. In such embodiments in which the surface tracking data is used for robotic surgery, the surface tracking data can need to be updated more frequently or can need to be extremely accurate so that the computer-implemented program can perform the robotic surgery with precision and accuracy.

[0311] In one embodiment, the system 2300 is configured to generate an occupancy grid map including an array of cells divided into a grid. The system 2300 is configured to store a height value for each of the respective grid cells to determine a surface map of the three-dimensional environment in the light deficient environment.

[0312] FIG. 24A and FIG. 24BPerspective and side views of an implementation of a monolithic sensor 2400 having multiple pixel arrays for producing a three-dimensional image are shown, respectively, in accordance with the teachings and principles of the present disclosure. Such an implementation can be desirable for three-dimensional image capture, where during use the two pixel arrays 2402 and 2404 can be offset. In another implementation, the first pixel array 2402 and the second pixel array 2404 can be dedicated to receiving electromagnetic radiation of a predetermined wavelength range, where the first pixel array is dedicated to electromagnetic radiation of a different wavelength range than the second pixel array.

[0313] FIG. 25A and FIG. 25B Perspective and side views of an implementation of an imaging sensor 2500 built on multiple substrates are shown, respectively. As shown, multiple columns of pixels 2504 forming the pixel array are located on a first substrate 2502 and multiple columns of circuitry 2508 are located on a second substrate 2506. Electrical connections and communication between a column of pixels and its associated or corresponding column of circuitry are also shown. In one implementation, the image sensor can have a pixel array separate from all or most of the supporting circuitry, while it can otherwise be fabricated with its pixel array and supporting circuitry on a single, monolithic substrate / chip. The present disclosure can use at least two substrates / chips that will be stacked together using three-dimensional stacking technology. The first 2502 of the two substrates / chips can be processed using image CMOS technology. The first substrate / chip 2502 can consist of only a pixel array, or can consist of a pixel array surrounded by limited circuitry. The second or subsequent substrate / chip 2506 can be processed using any technology, not necessarily from image CMOS technology. The second substrate / chip 2506 can be, but is not limited to, a high-density digital technology for integrating various and multiple functions into very limited space or area on the substrate / chip, or a mixed-mode or analog technology for integrating, for example, precise analog functions, or an RF technology for enabling wireless capabilities, or a MEMS (Micro-Electro-Mechanical System) for integrating MEMS devices. The image CMOS substrate / chip 2502 can be stacked with the second or subsequent substrate / chip 2506 using any three-dimensional technology. The second substrate / chip 2506 can support the majority or most of the circuitry that would otherwise be implemented in the first image CMOS chip 2502 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increases the overall system area while keeping the pixel array size constant and optimized as much as possible. Electrical connections between the two substrates / chips can be done through interconnects, which can be bond wires, lugs, and / or TSVs (Through Silicon Vias).

[0314] FIG. 26A and FIG. 26BPerspective and side views of a particular implementation of an imaging sensor 2600 with multiple pixel arrays for producing a three-dimensional image are shown. A three-dimensional image sensor can be built on multiple substrates and can include multiple pixel arrays and other associated circuitry, where multiple pixel columns 2602a forming a first pixel array and multiple pixel columns 2602b forming a second pixel array are on respective substrates 2608a and 2608b, respectively, and multiple circuit columns 2606a and 2606b are on a separate substrate 2604. Electrical connections and communication between pixel columns and associated or corresponding circuit columns are also shown.

[0315] The multiple pixel arrays can sense information simultaneously and can combine information from the multiple pixel arrays to generate a three-dimensional image. In one embodiment, an endoscopic imaging system includes two or more pixel arrays that can be deployed to generate three-dimensional imaging. The endoscopic imaging system can include an emitter to emit a pulse of electromagnetic radiation during a blanking period of a pixel array. The pixel arrays can be synchronized such that, for the two or more pixel arrays, an optical black pixel is read simultaneously (i.e., a blanking period occurs). The emitter can emit a pulse of electromagnetic radiation to charge each of the two or more pixel arrays. The two or more pixel arrays can read their respective charged pixels simultaneously, such that a readout period of the two or more pixel arrays occurs simultaneously or approximately simultaneously. In one embodiment, an endoscopic imaging system includes multiple emitters, each emitter being individually synchronized with one or more pixel arrays of a plurality of pixel arrays. Information from the multiple pixel arrays can be combined to generate three-dimensional image frames and video streams.

[0316] It should be understood that the teachings and principles of the present disclosure can be used in a reusable device platform, a limited use device platform, a repositionable use device platform, or a single use / disposable device platform without departing from the scope of the present disclosure. It should be understood that in a reusable device platform, an end user is responsible for cleaning and sterilization of the device. In a limited use device platform, a device can be used a certain defined number of times before becoming inoperable. A typical new device has been sterilized prior to delivery and should be cleaned and sterilized by the end user prior to other uses if used for other purposes. In a repositionable use device platform, a third party can reprocess (e.g., clean, package, and sterilize) single use devices for additional use at a lower cost than a new unit. In a single use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0317] EMBODIMENT

[0318] The following examples relate to preferred features of further embodiments:

[0319] Example 1 is a system. The system includes an emitter to emit pulses of electromagnetic radiation. The emitter includes a first emitter to emit pulses of electromagnetic radiation of a first wavelength. The emitter includes a second emitter to emit pulses of electromagnetic radiation of a second wavelength. The system includes an image sensor including an array of pixels to sense reflected electromagnetic radiation. The system includes a controller in electronic communication with the emitter and the image sensor. The system causes at least a portion of the pulses of electromagnetic radiation emitted by the emitter to include one or more of: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm; electromagnetic radiation having a wavelength of about 565 nm to about 585 nm; electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; an electromagnetic radiation excitation wavelength that causes a reagent to fluoresce; or a laser mapping mode.

[0320] Example 2 is the system of Example 1, wherein: the first emitter emits the pulses of electromagnetic radiation of the first wavelength at a first dichroic mirror that reflects the pulses of electromagnetic radiation of the first wavelength to a plurality of optical fibers; the second emitter emits the pulses of electromagnetic radiation of the second wavelength at a second dichroic mirror that reflects the pulses of electromagnetic radiation of the second wavelength to the plurality of optical fibers; and the first dichroic mirror is transparent to electromagnetic radiation of the second wavelength.

[0321] Example 3 is the system of any one of Examples 1-2, wherein the first dichroic mirror reflects the pulses of electromagnetic radiation of the first wavelength into the plurality of optical fibers at an angle that is offset from perpendicular to the plurality of optical fibers, and the second dichroic mirror reflects the pulses of electromagnetic radiation of the second wavelength into the plurality of optical fibers at an angle that is offset from perpendicular to the plurality of optical fibers.

[0322] Example 4 is the system of any one of Examples 1-3, wherein: the first dichroic mirror reflects the pulses of electromagnetic radiation of the first wavelength into the plurality of optical fibers at an angle that is substantially perpendicular to the first emitter; and the second dichroic mirror reflects the pulses of electromagnetic radiation of the second wavelength into the plurality of optical fibers through the first dichroic mirror at an angle that is substantially perpendicular to the second emitter.

[0323] Example 5 is the system of any of Examples 1-4, wherein: the emitter further comprises a third emitter for emitting pulses of electromagnetic radiation of a third wavelength at a third dichroic mirror, the third dichroic mirror reflects the pulses of electromagnetic radiation of the third wavelength to the plurality of optical fibers; the third dichroic mirror reflects the pulses of electromagnetic radiation of the third wavelength through the first dichroic mirror to the plurality of optical fibers; the first dichroic mirror and the second dichroic mirror are transparent to electromagnetic radiation of the third wavelength; the third dichroic mirror reflects the pulses of electromagnetic radiation of the third wavelength to the plurality of optical fibers at an angle substantially normal to the third emitter; and the third dichroic mirror reflects the pulses of electromagnetic radiation of the third wavelength to the plurality of optical fibers at an angle offset from normal.

[0324] Example 6 is the system of any of Examples 1-5, further comprising: an optical fiber bundle, wherein the emitter emits the pulses of electromagnetic radiation into the optical fiber bundle; wherein the optical fiber bundle comprises plastic fibers and glass fibers, wherein the plastic fibers and the glass fibers are coupled near an output of the optical fiber bundle.

[0325] Example 7 is the system of any of Examples 1-6, further comprising an intervening optical component, wherein the pulses of electromagnetic radiation pass through the intervening optical component before entering the optical fiber bundle.

[0326] Example 8 is the system of any of Examples 1-7, wherein the intervening optical component comprises one or more of a diffuser or a mixing rod.

[0327] Example 9 is the system of any of Examples 1-8, further comprising: an optical fiber bundle comprising a plurality of plastic optical fibers, wherein the emitter emits the pulses of electromagnetic radiation into the optical fiber bundle; and an intervening optical component, wherein the pulses of electromagnetic radiation pass through the intervening optical component before entering the optical fiber bundle; wherein the intervening optical component comprises a plurality of glass fibers.

[0328] Example 10 is the system of any of Examples 1-9, further comprising: an optical fiber bundle, wherein the emitter emits the pulses of electromagnetic radiation into the optical fiber bundle; and a diffuser disposed at a distal end of the optical fiber bundle; wherein the diffuser provides a light cone having an angle of between 110 and 120 degrees or an angle of between 70 and 80 degrees.

[0329] Example 11 is the system of any of Examples 1-10, further comprising a third emitter to emit pulses of electromagnetic radiation of a third wavelength and a fourth emitter to emit pulses of electromagnetic radiation of a fourth wavelength, and wherein: the first wavelength of electromagnetic radiation emitted by the first emitter is red light; the second wavelength of electromagnetic radiation emitted by the second emitter is blue light; the third wavelength of electromagnetic radiation emitted by the third emitter is green light; and the fourth wavelength of electromagnetic radiation emitted by the fourth emitter is a hyperspectral wavelength, the hyperspectral wavelength comprising one or more of: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm, or the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; wherein the fourth emitter to emit the hyperspectral wavelength comprises one or more independent lasers to emit electromagnetic radiation of different hyperspectral wavelengths.

[0330] Example 12 is the system of any of Examples 1-11, wherein the array of pixels of the image sensor senses reflected electromagnetic radiation during a readout period of the array of pixels to generate the plurality of exposure frames, wherein the readout period comprises a time period when active pixels in the array of pixels are read.

[0331] Example 13 is the system of any of Examples 1-12, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter are hyperspectral wavelengths to elicit a spectral response, wherein the hyperspectral wavelengths comprise one or more of: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm, and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0332] Example 14 is the system of any of Examples 1-13, wherein the emitter is configured to emit a plurality of electromagnetic radiation sub-pulses during a pulse duration, the plurality of sub-pulses having a sub-duration shorter than the pulse duration.

[0333] Example 15 is the system of any of Examples 1-14, wherein one or more of the pulses of electromagnetic radiation emitted by the emitter comprise electromagnetic radiation emitted at two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

[0334] Example 16 is the system of any of Examples 1-15, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter are hyperspectral emissions that cause the image sensor to produce a hyperspectral exposure frame, and wherein the controller is configured to provide the hyperspectral exposure frame to a corresponding hyperspectral system that determines a location of the critical tissue structure within the scene based on the hyperspectral exposure frame.

[0335] Example 17 is the system of any of Examples 1-16, wherein the hyperspectral emissions include: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0336] Example 18 is the system of any of Examples 1-17, wherein the controller is further configured to: receive the location of the critical tissue structure from the corresponding hyperspectral system; generate an overlay frame that includes the location of the critical tissue structure; and combine the overlay frame with a color image frame that depicts the scene to indicate the location of the critical tissue structure within the scene.

[0337] Example 19 is the system of any of Examples 1-18, wherein sensing electromagnetic radiation reflected by the pixel array includes generating a laser mapping exposure frame by sensing reflected electromagnetic radiation produced by the emitter pulsing the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and / or a size of one or more objects within the scene; provide the location of the critical tissue structure to the corresponding laser mapping system; and receive a topology and / or a size of the critical tissue structure from the corresponding laser mapping system.

[0338] Example 20 is the system of any of Examples 1-19, wherein the critical structure includes one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

[0339] Example 21 is the system of any of Examples 1-20, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter are fluorescence excitation wavelengths that cause the image sensor to produce a fluorescence exposure frame, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding fluorescence system that determines a location of the critical tissue structure within the scene based on the fluorescence exposure frame.

[0340] Example 22 is the system of any of Examples 1-21, wherein the fluorescent excitation emission comprises one or more of: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0341] Example 23 is the system of any of Examples 1-22, wherein the controller is further configured to: receive the location of the critical tissue structure from the corresponding fluorescence system; generate an overlay frame comprising the location of the critical tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

[0342] Example 24 is the system of any of Examples 1-23, wherein sensing electromagnetic radiation reflected by the pixel array comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and / or dimensions of one or more objects within the scene; provide the location of the critical tissue structure to the corresponding laser mapping system; and receive a topology and / or dimensions of the critical tissue structure from the corresponding laser mapping system.

[0343] Example 25 is the system of any of Examples 1-24, wherein the critical structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

[0344] Example 26 is the system of any of Examples 1-25, wherein the controller is configured to synchronize timing of the electromagnetic radiation pulses during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a start of a next subsequent readout of active pixels in the pixel array.

[0345] Example 27 is the system of any of Examples 1-26, wherein two or more electromagnetic radiation pulses emitted by the emitter produce two or more instances of reflected electromagnetic radiation that are sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.

[0346] Example 28 is the system of any of Examples 1-27, wherein the image sensor comprises a first image sensor and a second image sensor such that the image sensor is capable of generating a three-dimensional image.

[0347] Example 29 is the system of any of Examples 1-28, wherein the emitter is configured to repeatedly emit a sequence of pulses of electromagnetic radiation sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to a pulse of electromagnetic radiation.

[0348] Example 30 is the system of any of Examples 1-29, wherein the pulses of electromagnetic radiation are emitted in a pattern of different wavelengths of electromagnetic radiation, and wherein the emitter repeatedly emits the pattern of different wavelengths of electromagnetic radiation.

[0349] Example 31 is the system of any of Examples 1-30, wherein at least a portion of the pulses of electromagnetic radiation comprise a red wavelength, a green wavelength, a blue wavelength, and a hyperspectral wavelength, such that reflected electromagnetic radiation corresponding to each of the red wavelength, the green wavelength, the blue wavelength, and the hyperspectral wavelength sensed by the pixel array is processable to generate a red-green-blue (RGB) image frame comprising a superposition of hyperspectral imaging data, wherein the hyperspectral wavelength of electromagnetic radiation comprises: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0350] Example 32 is the system of any of Examples 1-31, wherein at least a portion of the pulses of electromagnetic radiation comprise a luminance emission, a red chrominance emission, a blue chrominance emission, and a hyperspectral emission, such that reflected electromagnetic radiation corresponding to each of the luminance emission, the red chrominance emission, the blue chrominance emission, and the hyperspectral emission sensed by the pixel array is processable to generate a YCbCr image frame comprising a superposition of hyperspectral imaging data, wherein the hyperspectral emission of electromagnetic radiation comprises: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0351] Example 33 is the system of any of Examples 1-32, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter is a fluorescence excitation wavelength for causing a reagent to fluoresce, wherein the fluorescence excitation wavelength comprises one or more of: the electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0352] Example 34 is the system of any of Examples 1-33, wherein at least a portion of the pulses of electromagnetic radiation comprises a red wavelength, a green wavelength, a blue wavelength, and a fluorescence excitation wavelength, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red wavelength, the green wavelength, the blue wavelength, and the fluorescence excitation wavelength can be processed to generate a red-green-blue (RGB) image frame comprising a stack of fluorescence imaging data, wherein the fluorescence wavelength of electromagnetic radiation comprises: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0353] Example 35 is the system of any of Examples 1-34, wherein at least a portion of the pulses of electromagnetic radiation comprises a luminance emission, a red chrominance emission, a blue chrominance emission, and a fluorescence excitation emission, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the luminance emission, the red chrominance emission, the blue chrominance emission, and the fluorescence excitation emission can be processed to generate a YCbCr image frame comprising a stack of fluorescence imaging data, wherein the fluorescence wavelength of electromagnetic radiation comprises: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0354] Example 36 is the system of any of Examples 1-35, further comprising a single optical fiber, wherein the emitter emits the pulses of electromagnetic radiation into the single optical fiber.

[0355] Example 37 is the system of any of Examples 1-36, wherein the pixel array is a two- dimensional array of independent pixels, each independent pixel capable of detecting electromagnetic radiation of any wavelength.

[0356] Example 38 is the system of any of Examples 1-37, further comprising a filter that filters electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0357] Example 39 is the system of any of Examples 1-38, further comprising a filter that filters electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0358] Example 40 is the system of any of Examples 1-39, wherein the electromagnetic sensor is a photodiode.

[0359] Example 41 is the system of any of Examples 1-40, wherein sensing reflected electromagnetic radiation by the pixel array comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the laser mapping pattern pulsed by the emitter, wherein the laser mapping exposure frame comprises information for determining real-time measurements including one or more of: a distance from an endoscope to an object; an angle between an endoscope and the object; or surface topology information about the object.

[0360] Example 42 is the system of any of Examples 1-41, wherein the laser mapping exposure frame comprises information for determining the real-time measurements with an accuracy of less than 10 centimeters.

[0361] Example 43 is the system of any of Examples 1-42, wherein the laser mapping exposure frame comprises information for determining the real-time measurements with an accuracy of less than one millimeter.

[0362] Example 44 is the system of any of Examples 1-43, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter comprises a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

[0363] Example 45 is the system of any of Examples 1-44, wherein the laser mapping pattern emitted by the emitter comprises a first output and a second output independent of each other, wherein the first output is for light illumination and the second output is for tool tracking.

[0364] Example 46 is the system of any of Examples 1-45, wherein the first emitter is a first laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulses of the first wavelength and the second emitter is a second laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulses of the second wavelength.

[0365] Example 47 is the system of any of Examples 1-46, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each exposure frame of the plurality of exposure frames corresponds to one or more electromagnetic radiation pulses emitted by the emitter.

[0366] Example 48 is the system of any of Examples 1-47, wherein the fiber optic bundle comprises 2 to 150 fibers.

[0367] Example 49 is the system of any of Examples 1-48, wherein the third emitter is a third laser beam comprising a plurality of lasers for emitting the pulses of electromagnetic radiation of the third wavelength.

[0368] Example 50 is the system of any of Examples 1-49, wherein the fourth emitter is a fourth laser beam comprising a plurality of lasers for emitting the pulses of electromagnetic radiation of the fourth wavelength.

[0369] Example 51 is the system of any of Examples 1-50, wherein the emitters comprise one or more hyperspectral emitters for emitting pulses of electromagnetic radiation of hyperspectral wavelengths to elicit a spectroscopic response.

[0370] Example 52 is the system of any of Examples 1-51, wherein each of the one or more hyperspectral emitters comprises a laser beam comprising a plurality of lasers.

[0371] Example 53 is the system of any of Examples 1-52, wherein the emitters further comprise an optical element for mixing pulses of electromagnetic radiation prior to entering the fiber optic bundle, wherein the optical element comprises one or more of a diffuser, a mixing rod, or a lens.

[0372] Example 54 is the system of any of Examples 1-53, further comprising a dichroic mirror for reflecting electromagnetic radiation of blue wavelengths.

[0373] Example 55 is the system of any of Examples 1-54, further comprising a dichroic mirror for reflecting electromagnetic radiation of green wavelengths.

[0374] Example 56 is the system of any of Examples 1-55, further comprising a dichroic mirror for reflecting electromagnetic radiation of red wavelengths.

[0375] Example 57 is the system of any of Examples 1-56, further comprising a dichroic mirror for reflecting electromagnetic radiation having wavelengths of about 513 nm to about 545 nm.

[0376] Example 58 is the system of any of Examples 1-57, further comprising a dichroic mirror for reflecting electromagnetic radiation having wavelengths of about 565 nm to about 585 nm.

[0377] Example 59 is the system of any of Examples 1-58, further comprising a dichroic mirror for reflecting electromagnetic radiation having wavelengths of about 900 nm to about 1000 nm.

[0378] Example 60 is the system of any of Examples 1-59, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0379] Example 61 is the system of any of Examples 1-60, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0380] Example 62 is the system of any of Examples 1-61, further comprising a dichroic mirror for reflecting a certain band of wavelengths of electromagnetic radiation, wherein the dichroic mirror is transparent to other wavelengths of electromagnetic radiation.

[0381] Example 63 is the system of any of Examples 1-62, further comprising a dichroic mirror for reflecting a certain wavelength of electromagnetic radiation, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a red wavelength.

[0382] Example 64 is the system of any of Examples 1-63, further comprising a dichroic mirror for reflecting a certain wavelength of electromagnetic radiation, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a green wavelength.

[0383] Example 65 is the system of any of Examples 1-64, further comprising a dichroic mirror for reflecting a certain wavelength of electromagnetic radiation, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a blue wavelength.

[0384] Example 66 is the system of any of Examples 1-65, further comprising a dichroic mirror for reflecting a certain wavelength of electromagnetic radiation, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a wavelength of about 513 nm to about 545 nm.

[0385] Example 67 is the system of any of Examples 1-66, further comprising a dichroic mirror for reflecting a certain wavelength of electromagnetic radiation, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a wavelength of about 565 nm to about 585 nm.

[0386] Example 68 is the system of any of Examples 1-67, further comprising a dichroic mirror for reflecting a certain wavelength of electromagnetic radiation, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0387] Example 69 is the system of any of Examples 1-68, further comprising a dichroic mirror for reflecting a certain wavelength of electromagnetic radiation, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0388] Example 70 is the system of any of Examples 1-69, further comprising a dichroic mirror for reflecting electromagnetic radiation of a particular wavelength, wherein the dichroic mirror is at least transparent to electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0389] Example 71 is the system of any of Examples 1-70, wherein the emitter comprises a plurality of laser emitters, and wherein the plurality of laser emitters have a Gaussian cross-sectional intensity profile.

[0390] Example 72 is the system of any of Examples 1-71, wherein the emitter comprises a plurality of laser emitters, and wherein the plurality of laser emitters have a flat or substantially flat intensity profile.

[0391] Example 73 is the system of any of Examples 1-72, wherein the emitter comprises a plurality of laser emitters, and wherein the plurality of laser emitters have a top-hat intensity profile.

[0392] Example 74 is the system of any of Examples 1-73, wherein the first emitter and the second emitter are aimed at a collection region, such that the pulses of electromagnetic radiation of the first wavelength and the pulses of electromagnetic radiation of the second wavelength mix at the collection region and are received by a fiber bundle.

[0393] Example 75 is the system of any of Examples 1-74, further comprising an intervening optical element disposed between the emitter and the fiber bundle, wherein the intervening optical element is configured to mix emissions from the first emitter and the second emitter before the emissions reach the fiber bundle.

[0394] Example 76 is the system of any of Examples 1-75, further comprising an intervening optical element disposed between the emitter and the fiber bundle, the intervening optical element configured to uniformly mix independent emissions of electromagnetic radiation from the first emitter and the second emitter before reaching the fiber bundle.

[0395] It should be appreciated that various features disclosed herein provide significant advantages over the state of the art. The following claims are exemplary of some of the features.

[0396] In the foregoing detailed description of the present disclosure, various features are grouped together in single embodiments for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed disclosure requires more features than are explicitly recited in each claim. Rather, inventive aspects lie in the non-combination of all features of the single embodiments disclosed above.

[0397] It will be appreciated that any features of the above-described arrangements, embodiments and implementations can be combined in single implementations comprising combinations of features obtained from any of the disclosed arrangements, embodiments and implementations.

[0398] It will be appreciated that the above-mentioned arrangements are merely illustrative of the application of the principles of the present disclosure. Numerous modifications and alternative arrangements can be devised by those skilled in the art without departing from the spirit and scope of the present disclosure and the appended claims are intended to cover such modifications and arrangements.

[0399] Thus, although the present disclosure has been described in detail with reference to particular implementations, it will be apparent to those skilled in the art that many modifications and variations can be made to the implementations, including but not limited to variations in size, materials, shapes, forms, function and manner of assembly and use.

[0400] Additionally, where appropriate, functions described herein can be performed in one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the following description and claims to refer to particular system components. As one skilled in the art will appreciate, components can be referred to by different names in different contexts. The use of a term in the following description is therefore not intended to limit the scope of the application to the specific name's definition, but is only meant to refer to that specific component.

[0401] The foregoing detailed description has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the disclosure be limited only by the claims appended hereto, and their equivalents. Additionally, it is intended that any or all appropriate methods of treatment can be used in combination with any or all appropriate compositions of matter to form additional combined methods of treatment.

[0402] In addition, while a particular implementation of the disclosure has been described, the disclosure is not limited to the particular form described. The scope of the disclosure is defined by the claims appended hereto, any future claims submitted by amendment to this application, and their equivalents.

Claims

1. A system comprising: a transmitter that transmits a plurality of pulses of electromagnetic radiation, wherein the transmitter comprises a plurality of electromagnetic sources including: a first transmitter to transmit pulses of electromagnetic radiation of a first wavelength; and a second transmitter to transmit pulses of electromagnetic radiation of a second wavelength; an image sensor comprising an array of pixels to sense reflected electromagnetic radiation; and a controller in electronic communication with the transmitter and the image sensor; and wherein at least a portion of the plurality of pulses of electromagnetic radiation transmitted by the transmitter comprises two or more of: a spectral emission comprising electromagnetic radiation of a wavelength that elicits a spectral response; a fluorescence emission comprising electromagnetic radiation of a fluorescence excitation wavelength; or a laser mapping emission comprising electromagnetic radiation of a laser mapping pattern; wherein the controller instructs the transmitter to cycle the plurality of electromagnetic sources according to a variable pulse cycle, the variable pulse cycle comprising two or more of the spectral emission, the fluorescence emission, or the laser mapping emission; and wherein the controller adjusts the variable pulse cycle based on one or more of user input or scene exposure.

2. The system of claim 1, wherein: the first transmitter transmits the pulses of electromagnetic radiation of the first wavelength at a first dichroic mirror that reflects the pulses of electromagnetic radiation of the first wavelength to a plurality of optical fibers; the second transmitter transmits the pulses of electromagnetic radiation of the second wavelength at a second dichroic mirror that reflects the pulses of electromagnetic radiation of the second wavelength to the plurality of optical fibers; and the first dichroic mirror is transparent to electromagnetic radiation of the second wavelength.

3. The system of claim 2, wherein the first dichroic mirror reflects the pulses of electromagnetic radiation of the first wavelength into the plurality of optical fibers at an angle that is offset from perpendicular to the plurality of optical fibers, and the second dichroic mirror reflects the pulses of electromagnetic radiation of the second wavelength into the plurality of optical fibers at an angle that is offset from perpendicular to the plurality of optical fibers.

4. The system of claim 2, wherein: the first dichroic mirror reflects the pulses of electromagnetic radiation of the first wavelength into the plurality of optical fibers at an angle that is substantially perpendicular to the first transmitter; and the second dichroic mirror reflects the pulses of electromagnetic radiation of the second wavelength into the plurality of optical fibers through the first dichroic mirror at an angle that is substantially perpendicular to the second transmitter.

5. The system of claim 2, wherein: the transmitter further comprises a third transmitter to transmit pulses of electromagnetic radiation of a third wavelength at a third dichroic mirror that reflects the pulses of electromagnetic radiation of the third wavelength to the plurality of optical fibers; the third dichroic mirror reflects the pulses of electromagnetic radiation of the third wavelength into the plurality of optical fibers through the first dichroic mirror; the first dichroic mirror and the second dichroic mirror are transparent to the third wavelength of electromagnetic radiation; the third dichroic mirror reflects the pulse of electromagnetic radiation of the third wavelength into the plurality of optical fibers at an angle that is substantially perpendicular to the third emitter; and the third dichroic mirror reflects the pulse of electromagnetic radiation of the third wavelength into the plurality of optical fibers at an angle that is offset from perpendicular.

6. The system of claim 1, further comprising: a bundle of optical fibers, wherein the emitter emits the pulse of electromagnetic radiation into the bundle of optical fibers; wherein the bundle of optical fibers comprises plastic fibers and glass fibers, wherein the plastic fibers and the glass fibers are coupled near an output of the bundle of optical fibers.

7. The system of claim 6, further comprising an intervening optical component, wherein the pulse of electromagnetic radiation passes through the intervening optical component before entering the bundle of optical fibers.

8. The system of claim 7, wherein the intervening optical component comprises one or more of a diffuser or a mixing rod.

9. The system of claim 1, further comprising: a bundle of optical fibers comprising a plurality of plastic optical fibers, wherein the emitter emits the pulse of electromagnetic radiation into the bundle of optical fibers; and an intervening optical component, wherein the pulse of electromagnetic radiation passes through the intervening optical component before entering the bundle of optical fibers; wherein the intervening optical component comprises a plurality of glass fibers.

10. The system of claim 1, further comprising: a bundle of optical fibers, wherein the emitter emits the pulse of electromagnetic radiation into the bundle of optical fibers; and a diffuser disposed at a distal end of the bundle of optical fibers; wherein the diffuser provides a cone of light having an angle of between 110 and 120 degrees or an angle of between 70 and 80 degrees.

11. The system of claim 1, wherein the emitter further comprises a third emitter for emitting a pulse of electromagnetic radiation of a third wavelength, a fourth emitter for emitting a pulse of electromagnetic radiation of a fourth wavelength, a fifth emitter for emitting a pulse of electromagnetic radiation of a fifth wavelength, and a sixth emitter for emitting the laser mapping pattern pulse, and wherein: the first wavelength of electromagnetic radiation emitted by the first emitter is red light; the second wavelength of electromagnetic radiation emitted by the second emitter is blue light; the third wavelength of electromagnetic radiation emitted by the third emitter is green light; the fourth wavelength of electromagnetic radiation emitted by the fourth emitter is electromagnetic radiation of a wavelength that elicits a spectral response; and the fifth wavelength of electromagnetic radiation emitted by the fifth emitter is electromagnetic radiation of a fluorescence excitation wavelength.

12. The system of claim 1, wherein the first emitter is a first laser bundle comprising a plurality of lasers for emitting the pulse of electromagnetic radiation of the first wavelength, and the second emitter is a second laser bundle comprising a plurality of lasers for emitting the pulse of electromagnetic radiation of the second wavelength. ​ ​ 13. The system of claim 1, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to one or more pulses of electromagnetic radiation emitted by the emitter.

14. The system of claim 13, wherein the array of pixels of the image sensor senses reflected electromagnetic radiation during a readout period of the array of pixels to generate the plurality of exposure frames, wherein the readout period comprises a time period when active pixels in the array of pixels are read.

15. The system of claim 1, wherein the emitter is configured to emit a plurality of sub- pulses of electromagnetic radiation having a sub-duration shorter than a pulse duration during the pulse duration.

16. The system of claim 1, wherein one or more of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises electromagnetic radiation emitted at two or more wavelengths simultaneously as a single pulse or single sub-pulse.

17. The system of claim 1, wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter are the fluorescent emissions, and wherein the array of pixels senses reflected electromagnetic radiation and outputs data used to generate a fluorescence frame, and wherein the controller provides the fluorescence frame to a corresponding fluorescence system that determines a location of a tissue structure within a scene based on the fluorescence frame.

18. The system of claim 17, wherein the fluorescent emissions comprise one or more of: electromagnetic radiation having a wavelength of 770 nm to 795 nm; or electromagnetic radiation having a wavelength of 790 nm to 815 nm.

19. The system of claim 18, wherein the controller is further configured to: receive the location of the tissue structure from the corresponding fluorescence system; generate an overlay frame comprising the location of the tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the tissue structure within the scene.

20. The system of claim 19, wherein the array of pixels senses reflected electromagnetic radiation and outputs data used to generate a laser mapping frame in response to the emitter pulsing the laser mapping emissions, and wherein the controller is configured to: provide the laser mapping frame to a corresponding laser mapping system that determines a topology of the scene and / or dimensions of one or more objects within the scene; provide the location of the tissue structure to the corresponding laser mapping system; and receive a topology and / or dimensions of the tissue structure from the corresponding laser mapping system.

21. The system of claim 20, wherein the tissue structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

22. The system of claim 1, wherein the controller is configured to synchronize timing of the plurality of pulses of electromagnetic radiation during a blanking period of the image sensor, wherein the blanking period corresponds to a time between readout of a last row of active pixels in the pixel array and a start of a next subsequent readout of active pixels in the pixel array.

23. The system of claim 1, wherein two or more pulses of electromagnetic radiation emitted by the emitter produce two or more instances of reflected electromagnetic radiation that are sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.

24. The system of claim 1, wherein the emitter is configured to repeatedly emit a sequence of pulses of electromagnetic radiation sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to one or more pulses of electromagnetic radiation.

25. The system of claim 1, wherein the variable pulse cycle comprises a pattern of electromagnetic radiation of different wavelengths, and wherein the emitter repeats the variable pulse cycle.

26. The system of claim 1, wherein the pixel array senses reflected electromagnetic radiation and outputs data for generating a laser mapping in response to the emitter pulsing the laser mapping emission, and wherein the laser mapping frame comprises data for determining real-time measurements including one or more of: a distance from an endoscope to an object; an angle between the endoscope and the object; or surface topology information about the object.

27. The system of claim 1, wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

28. The system of claim 1, wherein the laser mapping emission comprises a first output and a second output that are independent of each other, and wherein the first output is for light illumination and the second output is for tool tracking.

29. The system of claim 1, wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter is a spectral emission, and wherein the pixel array senses reflected electromagnetic radiation and outputs data for generating a spectral frame in response to the emitter pulsing the spectral emission, and wherein the controller provides the spectral frame to a corresponding spectral system that determines a location of a tissue structure within a scene based on the spectral frame.

30. The system of claim 29, wherein the spectral emission comprises: electromagnetic radiation having a wavelength of 513 nm to 545 nm and electromagnetic radiation having a wavelength of 900 nm to 1000 nm; or electromagnetic radiation having a wavelength of 565 nm to 585 nm and electromagnetic radiation having a wavelength of 900 nm to 1000 nm.

31. The system of claim 30, wherein the controller is further configured to: receive the location of the tissue structure from the corresponding spectral system; generate an overlay frame comprising the location of the tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the tissue structure within the scene.

32. The system of claim 31, wherein the pixel array senses reflected electromagnetic radiation in response to the emitter pulsing the laser mapping emission and outputs data used to generate a laser mapping frame, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and / or a size of one or more objects within the scene; provide the location of the tissue structure to the corresponding laser mapping system; and receive a topology and / or a size of the tissue structure from the corresponding laser mapping system.

33. The system of claim 32, wherein the tissue structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

Images