Offset illumination of a scene using multiple emitters in a fluorescence imaging system

Abstract

Offset illumination using multiple emitters in a fluorescence imaging system is described. The invention discloses a system comprising an emitter for emitting pulses of electromagnetic radiation and an image sensor comprising an array of pixels for sensing reflected electromagnetic radiation. The emitter comprises a first emitter and a second emitter for emitting electromagnetic radiation of different wavelengths. The system is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises electromagnetic radiation having a wavelength of about 770 nm to about 795 nm and / or about 795 nm to about 815 nm.

Description

Technical Field

[0001] This disclosure relates to digital imaging, and more particularly to fluorescence imaging in low-light environments. Background Technology

[0002] Advances in technology have enabled advancements in medical imaging capabilities. Endoscopes can be used to observe the inside of the body and examine the interior of organs or cavities. Endoscopes are used to investigate patient symptoms, confirm diagnoses, or provide medical treatment. Medical endoscopes can be used to observe a variety of body systems and parts, such as the gastrointestinal tract, respiratory tract, urinary tract, and abdominal cavity. Endoscopes are also used in surgical procedures, such as orthopedic procedures, procedures performed on joints or bones, procedures performed on the nervous system, procedures performed within the abdominal cavity, and so on.

[0003] In some cases of endoscopic imaging, viewing the space in color may be beneficial or necessary. A digital color image comprises at least three layers, or "color channels," that accumulate to form an image with a range of hues. Each color channel measures the intensity and chromaticity of light in a spectral band. Typically, a digital color image includes color channels for the red, green, and blue spectral bands (this may be referred to as a red-green-blue or RGB image). Each of the red, green, and blue color channels includes luminance information for the red, green, or blue spectral band. The luminance information from the individual red, green, and blue layers is combined to produce a color image. Because a color image is composed of individual layers, a conventional digital camera image sensor includes an array of color filters that allows red, green, and blue visible light wavelengths to strike selected pixel sensors. Each individual pixel sensor element is sensitive to a red, green, or blue light wavelength and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image. At least three different types of pixel sensors occupy a significant amount of physical space, making a complete pixel array impossible to fit in the small distal end of an endoscope.

[0004] Because conventional image sensors cannot be fitted into the distal end of an endoscope, they are traditionally located in the endoscope's handheld unit, which is held by the operator and not placed within the body cavity. In such endoscopes, light travels along the length of the endoscope from the handheld unit to the distal end. This configuration has significant limitations. Endoscopes with this configuration are delicate and can easily become misaligned or damaged by impacts or shocks during normal use. This can significantly degrade image quality, and the endoscope requires frequent repair or replacement.

[0005] Traditional endoscopes with image sensors housed in a handheld unit are further limited to capturing only color images. However, in some implementations, it may be desirable to capture images using fluorescence image data in addition to color image data. Fluorescence is the light emitted by a substance that has absorbed light or other electromagnetic radiation. When a fluorescent material is exposed to ultraviolet light or other wavelengths of electromagnetic radiation, some fluorescent materials “glow” or emit different colors visible to the human eye. When the radiation source is removed, some fluorescent materials will stop emitting light almost immediately.

[0006] Fluorescence occurs when orbital electrons in molecules, atoms, or nanostructures are excited by light or other electromagnetic radiation and then relax to their ground state by emitting photons from the excited state. The specific frequency of the electromagnetic radiation that excites the orbital electrons or is emitted by photons during relaxation depends on the atom, molecule, or nanostructure. Fluorescence imaging has many practical applications, including mineralogy, geology, medicine, spectroscopy of chemical sensors, detection of biological processes or signals, and more. Fluorescence can be used in biochemistry and medicine as a non-destructive means of tracking or analyzing biomolecules. Some fluorescent reagents or dyes can be configured to attach to certain types of tissue, thereby drawing attention to that type of tissue.

[0007] However, fluorescence imaging requires specific electromagnetic radiation emission and a specific imaging sensor capable of reading the specific relaxation wavelength of a particular fluorescent reagent. Different reagents or dyes are sensitive to electromagnetic radiation of different wavelengths and emit electromagnetic radiation of different wavelengths when fluorescing. Fluorescence imaging systems can be highly specialized and can be tailored for a particular reagent or dye. Such imaging systems are used for limited applications and cannot fluoresce more than one reagent or structure during a single imaging process. Using multiple different imaging systems, each configured to fluoresce different reagents, is very expensive. Alternatively, it may be advantageous to apply multiple fluorescent reagents in a single imaging process and observe multiple reagents in a single superimposed image.

[0008] Based on the foregoing, this paper describes systems, methods, and apparatuses for fluorescence imaging in low-light environments. Such systems, methods, and apparatuses can provide multiple datasets for identifying key structures within the body and provide precise and valuable information about body cavities. Attached Figure Description

[0009] Non-limiting and incomplete embodiments of this disclosure are described with reference to the following accompanying drawings, wherein, unless otherwise specified, similar reference numerals in the various views indicate similar parts. The advantages of this disclosure will be better understood with reference to the following description and drawings, wherein:

[0010] Figure 1 This is a schematic diagram of a system with paired emitters and pixel arrays for digital imaging in low-light environments.

[0011] Figure 2 A system for providing illumination to light-deficient environments for endoscopic imaging;

[0012] Figure 2A A schematic diagram of the hardware of the complementary system;

[0013] Figures 3A to 3D A diagram illustrating the operating loop of the sensor used to construct the exposure frame;

[0014] Figure 4A A diagram illustrating the operation of an electromagnetic transmitter implementation scheme;

[0015] Figure 4B A diagram illustrating how varying the duration and magnitude of emitted electromagnetic pulses can provide exposure control.

[0016] Figure 5 To be Figures 3A to 4B The illustration of an embodiment of the present disclosure is shown, which combines the sensor's operating cycle, electromagnetic transmitter, and emitted electromagnetic pulses, illustrating the imaging system during operation.

[0017] Figure 6A This is a schematic diagram of a method for recording video with full-spectrum light during the time period from t(0) to t(1);

[0018] Figure 6B This is a schematic diagram of the process of recording video by using spectral light in pulse segments over a time period from t(0) to t(1);

[0019] Figures 7A to 7E A schematic diagram illustrates a method for recording video frames of both full-spectrum light and partitioned-spectrum light over a certain time interval;

[0020] Figure 8 It is a graphical display of the delay or jitter between the control signal and the electromagnetic radiation emitted by the transmitter;

[0021] Figure 9 It is a cross-sectional view of an optical fiber bundle including a central fiber and multiple surrounding fibers;

[0022] Figure 10 It is a graphical display of the top-hat profile and Gaussian profile used to transmit electromagnetic radiation to an optical fiber bundle.

[0023] Figure 11 It is a side view showing the output of electromagnetic radiation (light) from the fiber optic bundle compared to the camera's field of view;

[0024] Figure 12 This is a side view showing the output of electromagnetic radiation from a bundle of optical fibers, where the ends of individual fibers are designed to provide a more uniform distribution of electromagnetic radiation.

[0025] Figure 13 This is a side view showing the output of electromagnetic radiation from an optical fiber bundle, wherein the optical fiber bundle includes plastic fibers and glass fibers coupled near the output section;

[0026] Figure 14 This is a side view showing the output of electromagnetic radiation from an optical fiber bundle, which includes a diffuser located near the output.

[0027] Figure 15 It is a schematic flowchart of a method for driving a transmitter to illuminate a scene according to jitter specifications;

[0028] Figure 16 It is a schematic flowchart of a method for providing electromagnetic radiation to imaging scenes in low-light environments;

[0029] Figures 17A to 17C A light source with multiple emitters is shown;

[0030] Figure 18 A single optical fiber is shown, which illuminates a scene in a dark environment by outputting via a diffuser at the output point;

[0031] Figure 19 This illustrates a portion of an electromagnetic spectrum that, according to the principles and teachings of this disclosure, is divided into multiple distinct sub-spectrums that can be emitted by an emitter of a light source;

[0032] Figure 20 This is a schematic diagram illustrating the emission and readout timing for generating an image frame that includes multiple exposure frames produced by different partitions of pulsed light;

[0033] Figure 21 An imaging system including a single cutoff filter for filtering electromagnetic radiation wavelengths is shown;

[0034] Figure 22 An imaging system including multiple cutoff filters for filtering electromagnetic radiation wavelengths is shown;

[0035] Figure 23A and Figure 23B Perspective and side views of a specific implementation of an imaging sensor having multiple pixel arrays for generating three-dimensional images are shown, wherein the multiple pixel arrays and the image sensor are constructed on multiple substrates;

[0036] Figure 24A and Figure 24B A specific embodiment of a plurality of pixel arrays for generating a three-dimensional image is shown, in accordance with the principles and teachings of this disclosure; and

[0037] Figure 25A and Figure 25BPerspective and side views of a specific implementation of an imaging sensor built on multiple substrates are shown, wherein multiple pixel columns forming a pixel array are located on a first substrate and multiple circuit columns are located on a second substrate. The figures show electrical connections and communications between a pixel column and its associated or corresponding circuit column. Detailed Implementation

[0038] This document discloses systems, methods, and apparatuses for digital imaging that are primarily applicable to medical applications such as medical endoscopic imaging. One embodiment of this disclosure is an endoscopic system for fluorescence and color imaging in low-light environments. The methods, systems, and computer-based products disclosed herein provide imaging or diagnostic capabilities for medical robotic applications, such as the use of robots for performing imaging procedures, surgical procedures, etc.

[0039] One embodiment of this disclosure is an endoscopic imaging system including a transmitter for emitting pulses of electromagnetic radiation to illuminate a scene. The transmitter includes multiple laser beams (which may alternatively be referred to herein as separate “emitters” constituting the overall transmitter), which can operate independently of each other and emit electromagnetic radiation of different wavelengths. The multiple laser beams constituting the transmitter can each be configured to pulse electromagnetic radiation in different zones or wavelengths of the electromagnetic spectrum. The electromagnetic radiation pulses can be pulsed to an optical fiber bundle, which can then carry the pulsed electromagnetic radiation to the distal end of the endoscope to illuminate the scene. This specific implementation using multiple laser beams introduces problems when the electromagnetic radiation reaches the optical fiber bundle. For example, individual fibers within the fiber bundle may receive light of different wavelengths, different power levels, or more or less light than other individual fibers. This results in non-uniform illumination of the scene.

[0040] Based on the foregoing, embodiments of this disclosure include an intervening optical element and / or one or more dichroic mirrors. The intervening optical element and the one or more dichroic mirrors can be used in combination to provide uniform light to the fiber bundle. The intervening optical element can be positioned between the transmitter and the fiber bundle. The intervening optical element may include, for example, a diffuser, a mixing rod, a lens, or some other optical component for promoting a uniform light mixture. In one embodiment, a dichroic mirror is present for each laser beam of the transmitter. A first dichroic mirror for a first laser beam may be configured to reflect electromagnetic radiation pulsed by the first laser beam at a certain wavelength. A second dichroic mirror for a second laser beam may be configured to reflect electromagnetic radiation pulsed by the second laser beam at a certain wavelength, etc. Thus, the dichroic mirror can be configured to reflect electromagnetic radiation pulsed by a certain laser beam and be transparent to electromagnetic radiation of other wavelengths that may be pulsed by other laser beams of the transmitter. In one embodiment, the dichroic mirror is angled and positioned within an endoscope system such that electromagnetic radiation pulsed by the transmitter strikes the dichroic mirror and then changes the direction pulsed into the fiber bundle. The fiber optic bundle can then carry light through the endoscope system to illuminate the scene. This system for offset illumination makes it possible to use multiple different lasers, laser beams, or transmitters without the risk of illuminating the scene or non-uniform light transmitted through the fiber optic bundle.

[0041] Conventional endoscopes are designed such that the image sensor is placed at the proximal end of the device within the handheld unit. This configuration requires incident light to travel the length of the endoscope through precisely coupled optics. These precise optics can easily become misaligned during normal use, leading to image distortion or loss. The embodiments disclosed herein place the image sensor in a space-constrained environment within the distal end of the endoscope itself. This offers greater optical simplicity compared to specific implementations known in the art. However, this acceptable solution is not straightforward and presents a number of engineering challenges.

[0042] Significant image quality loss can occur when minimizing the overall size of the image sensor to fit within the distal end of the endoscope. The area of ​​the pixel array of the image sensor can be reduced by decreasing the number of pixels and / or the sensing area of ​​each individual pixel. Each of these modifications affects the resolution, sensitivity, and dynamic range of the resulting image. Conventional endoscopic imaging systems aim to sense stable broadband illumination and provide color information using segmented pixel arrays (such as Bayer patterned arrays). Based on the drawbacks associated with segmented pixel arrays, this paper discloses alternative systems and methods using monochromatic (which may be referred to as "color-indeterminate") pixel arrays that do not include individual pixel filters. In the embodiments disclosed herein, color information is provided by pulsed electromagnetic radiation of different wavelengths to the transmitter. The pulsed imaging system disclosed herein can generate a color image overlaid with fluorescence imaging data.

[0043] In one embodiment, color information is determined by capturing independent exposure frames in response to electromagnetic radiation pulses of different wavelengths. Alternatively, the pulses may include red, green, and blue light wavelengths for generating an RGB image frame consisting of red, green, and blue exposure frames. In an alternative embodiment, the pulses may include luminance (“Y”), redness (“Cr”), and blueness (“Cb”) pulses of light for generating a YCbCr image frame consisting of luminance data, redness data, and blueness data. The color image frame may also include data from fluorescence exposure frames superimposed on the RGB or YCbCr image frame. The fluorescence pulses may include one or more electromagnetic radiation pulses for inducing a spectral response. In one embodiment, the fluorescence emission includes one or more electromagnetic radiations having wavelengths of about 770 nm to about 790 nm; or about 795 nm to about 815 nm. Alternating the wavelengths of the pulsed electromagnetic radiation allows the use of a full-pixel array and avoids artifacts introduced by Bayer patterned pixel arrays.

[0044] In some cases, it is desirable to generate endoscopic images with multiple data types or multiple overlapping images. For example, it is desirable to generate color (RGB or YCbCr) images that also include fluorescence imaging data superimposed on a color image. Superimposed images with this property allow physicians or computer programs to identify key body structures based on fluorescence imaging data. Historically, this has required the use of multiple sensor systems, including an image sensor for color imaging and one or more additional image sensors for fluorescence imaging. In such systems, the multiple image sensors would have multiple types of pixel sensors, each sensitive to different ranges of electromagnetic radiation. In systems known in the art, this includes three separate types of pixel sensors for generating color images, and additional pixel sensors for generating fluorescence image data at different wavelengths of the electromagnetic spectrum. These multiple different pixel sensors occupy excessive physical space and cannot be located at the distal end of the endoscope. In systems known in the art, one or more cameras are not placed at the distal end of the endoscope, but rather within the endoscope handpiece or robotic unit. This causes many disadvantages and results in endoscopes requiring high precision. When an endoscope is subjected to impact or shock during use, the delicate endoscope may be damaged and the image quality may be degraded. In light of the foregoing, this document discloses systems, methods, and apparatuses for endoscopic imaging in low-light environments. The systems, methods, and apparatuses disclosed herein provide means for employing multiple imaging techniques simultaneously in a single imaging process, allowing one or more image sensors to be positioned at the distal end of the endoscope.

[0045] Fluorescence imaging

[0046] The systems, methods, and apparatuses disclosed herein provide means for generating fluorescence imaging data in light-deficient environments. This fluorescence imaging data can be used to identify certain substances, tissues, components, or processes within body cavities or other light-deficient environments. In some embodiments, the fluorescence imaging is provided to a physician or a computer-implemented program to enable the identification of certain structures or tissues within the body. Such fluorescence imaging data can be overlaid on black-and-white or RGB images to provide additional information and context.

[0047] Fluorescence is the light emitted by a substance that has absorbed light or other electromagnetic radiation. When fluorescent substances are exposed to ultraviolet light or other wavelengths of electromagnetic radiation, some fluorescent substances can "glow" or emit different colors visible to the human eye. When the radiation source is removed, some fluorescent materials will stop emitting light almost immediately.

[0048] Fluorescence occurs when orbital electrons in molecules, atoms, or nanostructures are excited by light or other electromagnetic radiation and then relax to their ground state by emitting photons from the excited state. The specific frequency of the electromagnetic radiation that excites the orbital electrons or is emitted by photons during relaxation depends on the specific atom, molecule, or nanostructure. In most cases, the light emitted by the material has a longer wavelength than the radiation absorbed by the material, and therefore has lower energy. However, when the absorbed electromagnetic radiation is strong, an electron can absorb two photons. This two-photon absorption can lead to the emission of radiation with a shorter wavelength than the absorbed radiation, and therefore higher energy. Alternatively, the emitted radiation may have the same wavelength as the absorbed radiation.

[0049] Fluorescence imaging has numerous practical applications, including mineralogy, geology, medicine, spectroscopy of chemical sensors, and the detection of biological processes or signals. Fluorescence is particularly useful in biochemistry and medicine as a non-destructive means of tracking or analyzing biomolecules. Biomolecules, including certain tissues or structures, can be tracked by analyzing their fluorescence emission after excitation by electromagnetic radiation of certain wavelengths. However, relatively few cellular components are naturally fluorescent. In some implementations, it may be advantageous to visualize certain tissues, structures, chemical processes, or biological processes that are inherently non-fluorescent. In such implementations, dyes or reagents can be administered to the body, including molecules, proteins, or quantum dots with fluorescent properties. The reagents or dyes can then fluoresce after excitation by electromagnetic radiation of certain wavelengths. Different reagents or dyes can include different molecules, proteins, and / or quantum dots that will fluoresce under electromagnetic radiation of specific wavelengths. Therefore, it may be necessary to excite reagents or dyes with electromagnetic radiation of specific frequency bands to achieve fluorescence and identify desired tissues, structures, or processes in vivo.

[0050] Fluorescence imaging provides valuable information in the medical field that can be used for diagnostic purposes and / or visualized in real time during medical procedures. Specific reagents or dyes can be applied to the body to cause certain tissues, structures, chemical processes, or biological processes to fluoresce. The fluorescence of the reagents or dyes can highlight body structures such as blood vessels, nerves, and specific organs. Additionally, the fluorescence of the reagents or dyes can highlight symptoms or diseases, such as cancer cells or cells undergoing certain biological or chemical processes that may be associated with a symptom or disease. Fluorescence imaging can be used in real time by practicing physicians or computer programs to differentiate, for example, cancerous and non-cancerous cells during surgical tumor extraction. Fluorescence imaging can also be used non-destructively to track and visualize conditions within the body that are otherwise invisible to the human eye or indistinguishable in RGB images over time.

[0051] Systems, methods, and apparatuses for generating fluorescence imaging data can be used in conjunction with reagents or dyes. Some reagents or dyes are known to attach to certain types of tissue and fluoresce at specific wavelengths of electromagnetic spectrum. In one embodiment, a reagent or dye is applied to a patient configured to fluoresce when activated by light of certain wavelengths. The endoscopic imaging system disclosed herein is used to excite a reagent or dye and cause it to fluoresce. The fluorescence of the reagent or dye is captured by the endoscopic imaging system to aid in the identification of tissues or structures in body cavities. In one embodiment, multiple reagents or dyes are administered to a patient, each configured to fluoresce at different wavelengths and / or provide indications of different structures, tissues, chemical reactions, biological processes, etc. In such embodiments, the endoscopic imaging system emits each of the applicable wavelengths to cause each of the applicable reagents or dyes to fluoresce. This eliminates the need to perform separate imaging procedures for each of the multiple reagents or dyes.

[0052] Imaging reagents enhance imaging capabilities in the pharmaceutical, medical, biotechnology, diagnostics, and medical protocol industries. Many imaging techniques, such as X-rays, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI), and nuclear medicine, primarily analyze anatomical structures and morphology, but cannot detect changes at the molecular level. Fluorescent reagents, dyes, and probes (including quantum dot nanoparticles and fluorescent proteins) assist medical imaging techniques by providing additional information about certain tissues, structures, chemical processes, and / or biological processes present within the imaging region. Imaging with fluorescent reagents enables cell tracking and / or the tracking of certain molecular biomarkers. Fluorescent reagents can be used to image cancer, infection, inflammation, stem cell biology, and more. Many fluorescent reagents and dyes are being developed and applied to visualize and track biological processes in a non-destructive manner. Such fluorescent reagents can be excited by electromagnetic radiation of certain wavelengths or wavelength bands. Similarly, these fluorescent reagents emit relaxation energies of certain wavelengths or wavelength bands when they fluoresce, and the emitted relaxation energies can be read by sensors to determine the location and / or boundaries of the reagent or dye.

[0053] In one embodiment of this disclosure, the endoscopic imaging system pulses electromagnetic radiation to excite electrons in a fluorescent reagent or dye. The endoscopic imaging system can pulse electromagnetic radiation of multiple different wavelengths during a single imaging process to fluoresce a variety of different reagents or dyes. The endoscope includes an image sensor sensitive to the relaxation wavelength of one or more reagents or dyes. Imaging data generated by the image sensor can be used to identify the location and boundaries of one or more reagents or dyes. The endoscopic system can also pulse electromagnetic radiation in the red, green, and blue bands of visible light, allowing fluorescence imaging to be superimposed on an RGB video stream.

[0054] Pulse imaging

[0055] Some specific embodiments of this disclosure include aspects of a sensor and system combination design capable of generating high-resolution images with a reduced number of pixels in confined lighting environments. This is achieved by pulsed single light wavelengths frame by frame, and by using a controlled light source combined with a high frame capture rate and a specially designed corresponding monochromatic sensor to switch or alternate between single different color wavelengths each frame. Additionally, electromagnetic radiation outside the visible spectrum can be pulsed to enable the generation of fluorescence images. Pixels can be color-variable, such that each pixel generates data for each electromagnetic radiation pulse, each pulse including pulses of red, green, and blue visible light wavelengths as well as pulses of other wavelengths that can be used for fluorescence imaging.

[0056] This disclosed system is an endoscopic imaging system for use in low-light environments. The system includes an endoscope comprising an image sensor configured to sense reflected electromagnetic radiation for generating multiple exposure frames, which can be combined to generate a color image frame overlaid with fluorescence data. The system includes a transmitter for emitting pulses of electromagnetic radiation. The system includes a controller (optionally referred to as "control circuitry") in electronic communication with the image sensor and the transmitter. The controller controls the duty cycle of the transmitter in response to a signal corresponding to the duty cycle of the transmitter. The image sensor includes bidirectional pads for transmitting and receiving information. The bidirectional pads of the image sensor operate in a frame period divided into three defined states: a roll-readout state, a service item state, and a configuration state. The system includes an oscillator disposed in the controller and a frequency detector connected to the controller. The frequency detector controls the clock frequency of the image sensor in response to a signal from the controller corresponding to the oscillator frequency. The system enables clock signal data to be transmitted from the bidirectional pads of the image sensor to the controller during the service item and configuration phases. This system enables exposure frames to be synchronized without using an input clock or a data transmission clock.

[0057] For the purpose of facilitating an understanding of the principles of this disclosure, reference will now be made to embodiments shown in the accompanying drawings, and these embodiments will be described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any changes and further modifications to the features of the invention shown herein, as well as any additional applications of the principles of this disclosure as shown herein (which will generally occur to those skilled in the art and those familiar with the contents of this disclosure), will be considered within the scope of the disclosure protected by the claims.

[0058] Before disclosing and describing the structures, systems, and methods for generating images in low-light environments, it should be understood that this disclosure is not limited to the specific structures, configurations, process steps, and materials disclosed herein, as such structures, configurations, process steps, and materials can vary to some extent. Furthermore, it should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the invention will be defined only by the appended claims and their equivalents.

[0059] In describing and claiming the subject matter of this disclosure, the following terms will be used in accordance with the following definitions.

[0060] It should be noted that the singular forms “a,” “an,” and “the” used in this specification and the appended claims include multiple referents unless the context clearly indicates otherwise.

[0061] As used herein, the terms “comprising,” “including,” “characterized by,” and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unmentioned elements or method steps.

[0062] As used herein, the phrase “consisting of” and its grammatical equivalents exclude any element or step not included in the claims.

[0063] As used herein, the phrase “consistent with…” and its grammatical equivalents limit the scope of the claims to the specified materials or steps and to materials or steps that do not substantially affect one or more of the essential and novel features of the claimed disclosure.

[0064] As used in this article, the term "proximal" broadly refers to the concept of the part closer to the starting point.

[0065] As used in this article, the term "distal" generally refers to the opposite of the proximal, and therefore, depending on the context, it refers to the part that is farther from the starting point or the farthest part.

[0066] As used herein, color sensors or multispectral sensors are those known to have a color filter array (CFA) on which incident electromagnetic radiation is filtered to its single component. In the visible range of the electromagnetic spectrum, such CFAs can be based on a Bayer template or a modified form thereof to separate the green, red, and blue spectral components of light.

[0067] As used in this article, a monochrome sensor refers to an imaging sensor without filtering capabilities. Because pixels are color-invariant, its effective spatial resolution is significantly higher than that of pixel color equivalents in traditional single-sensor cameras (which typically employ Bayer pattern filtering). Since fewer incident photons are wasted between individual pixels, monochrome sensors can also achieve higher quantum efficiency.

[0068] As used herein, a transmitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of a transmitter can be configured to emit pulses and have very specific frequencies or frequency ranges from the entire electromagnetic spectrum. Pulses can include wavelengths in both the visible and invisible ranges. The transmitter can be cyclically turned on and off to generate pulses, or pulses can be generated using a shutter mechanism. The transmitter can have a variable power output level, or it can be controlled using auxiliary devices such as apertures or filters. The transmitter can emit broad-spectrum or full-spectrum electromagnetic radiation that can be pulsed through color filtering or shutter action. The transmitter can include multiple electromagnetic sources acting individually or in combination.

[0069] It should be noted that, as used herein, the term "light" refers to both a particle and a wavelength, and is intended to represent electromagnetic radiation detectable by the pixel array 122, and may include wavelengths from both the visible and invisible spectra of electromagnetic radiation. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is smaller than the entire spectrum, or in other words, wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a controllable light source with respect to a portion of the emitted electromagnetic spectrum, or a light source with physical properties of its components, emission intensity, or emission duration, or all of the above. An emitter may emit light in any dithered, diffuse, or collimated emission, and may be controlled digitally or through analog methods or systems. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and includes light sources such as lasers, LEDs, incandescent light, or any digitally controllable light source.

[0070] Now refer to the attached diagram, Figure 1 A schematic diagram of a system 100 for sequential pulse imaging in a dark environment is shown. The system 100 can be deployed to generate an RGB image, with fluorescence imaging data superimposed on the RGB image. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a partition of electromagnetic radiation in the dark environment 112, and the pixel array 122 senses instances of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 operate sequentially such that one or more pulses of the partition of electromagnetic radiation generate image data sensed by the pixel array 122.

[0071] It should be noted that, as used herein, the term "light" refers to both a particle and a wavelength, and is intended to represent electromagnetic radiation detectable by the pixel array 122, and may include wavelengths from both the visible and invisible spectra of electromagnetic radiation. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is smaller than the entire spectrum, or in other words, wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a controllable light source with respect to a portion of the emitted electromagnetic spectrum, or a light source with physical properties of its components, emission intensity, or emission duration, or all of the above. An emitter may emit light in any dithered, diffuse, or collimated emission, and may be controlled digitally or through analog methods or systems. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and includes light sources such as lasers, LEDs, incandescent light, or any digitally controllable light source.

[0072] The pixel array 122 of the image sensor can be electronically paired with the transmitter 102, such that the transmitter 102 and the pixel array 122 are synchronized during operation for receiving emissions and for conditioning within the system. The transmitter 102 can be tuned to emit electromagnetic radiation in the form of a laser, which can be pulsed to illuminate a darkened environment 112. The transmitter 102 can be pulsed at intervals corresponding to the operation and function of the pixel array 122. The transmitter 102 can pulse light in multiple electromagnetic zones, causing the pixel array to receive electromagnetic energy and generate a dataset corresponding in time to each specific electromagnetic zone. For example, Figure 1 A specific embodiment is shown in which emitter 102 emits four different zones of electromagnetic radiation, including red 104, green 106, blue 108, and a fluorescence excitation wavelength 110. The fluorescence excitation wavelength 110 may include multiple different zones of electromagnetic radiation, which are selected to fluoresce various fluorescent reagents present in the dark environment 112. The fluorescence excitation wavelength 110 may be selected to fluoresce a specific fluorescent reagent present in the dark environment 112.

[0073] exist Figure 1 In alternative embodiments not shown, the pulsed emission of light includes luminance (“Y”) emission, red hue (“Cr”) emission, and blue hue (“Cb”) emission, instead of pulsed red 104 emission, pulsed green 106 emission, and pulsed blue 108 emission. In one embodiment, the controller or transmitter 102 modularizes the electromagnetic radiation pulses according to color conversion coefficients to provide luminance and / or chromaticity information, which convert light energy from the red, green, and blue light energy spaces into luminance, red hue, and blue hue light energy spaces. The pulsed emission of light may also include modulated blue hue (“Y+Cb”) pulses and / or modulated red hue (“Y+Cr”) pulses.

[0074] The low-light environment 112 includes structures, tissues, and other elements that reflect a combination of red 114, green 116, and / or blue 118 light. A structure perceived as red 114 reflects pulsed red 104 light back. Reflection from the red structure results in red 105 sensed by the pixel array 122 after the emission of the pulsed red 104. Data sensed by the pixel array 122 generates a red exposure frame. A structure perceived as green 116 reflects pulsed green 106 light back. Reflection from the green structure results in green 107 sensed by the pixel array 122 after the emission of the pulsed green 106. Data sensed by the pixel array 122 generates a green exposure frame. A structure perceived as blue 118 reflects pulsed blue 108 light back. Reflection from the blue structure results in blue 109 sensed by the pixel array 122 after the emission of the pulsed blue 108. Data sensed by the pixel array 122 generates a blue exposure frame.

[0075] When the structure is a combination of colors, it will reflect back a combination of pulsed red 104 emission, pulsed green 106 emission, and / or pulsed blue 108 emission. For example, a structure perceived as purple will reflect back light from pulsed red 104 emission and pulsed blue 108 emission. The resulting data sensed by pixel array 122 will indicate that light is reflected in the same area after pulsed red 104 emission and pulsed blue 108 emission. When the resulting red and blue exposure frames are combined to form an RGB image frame, the RGB image frame will indicate that the structure is purple.

[0076] In embodiments where the light-deficient environment 112 includes a fluorescent reagent or dye, or includes one or more fluorescent structures, tissues, or other elements, the pulsed scheme may include emission of certain fluorescent excitation wavelengths. Certain fluorescent excitation wavelengths may be selected to cause a known fluorescent reagent, dye, or other structure to fluoresce. The fluorescent structure will be sensitive to the fluorescent excitation wavelength and will emit a fluorescent relaxation wavelength. After emission at the fluorescent excitation wavelength, the fluorescence relaxation wavelength will be sensed by the pixel array 122. The data sensed by the pixel array 122 generates a fluorescent exposure frame. The fluorescent exposure frame may be combined with multiple other exposure frames to form an image frame. Data in the fluorescent exposure frame may be superimposed on an RGB image frame, which includes data from red, green, and blue exposure frames.

[0077] In embodiments where the light-deficient environment 112 includes a structure, tissue, or other material that emits a spectral response to certain regions of the electromagnetic spectrum, the pulsed scheme may include the emission of hyperspectral regions of electromagnetic radiation to induce a spectral response from the structure, tissue, or other material present in the light-deficient environment 112. This spectral response includes the emission or reflection of electromagnetic radiation at certain wavelengths. This spectral response may be sensed by the pixel array 122 and generate a hyperspectral exposure frame. The hyperspectral exposure frame may be combined with multiple other exposure frames to form an image frame. Data in the hyperspectral exposure frame may be superimposed on an RGB image frame, which includes data from red, green, and blue exposure frames.

[0078] In one implementation, the pulse scheme includes a laser mapping emission or tool tracking pattern. Following the laser mapping emission or tool tracking pattern, reflected electromagnetic radiation sensed by pixel array 122 results in a laser mapping exposure frame. Data in the laser mapping exposure frame can be provided to a corresponding system to identify, for example, the distance between tools present in the dark environment 112, the three-dimensional surface topology of the scene in the dark environment 112, the distance, size, or location of structures or objects within the scene, etc. This data can be overlaid on an RGB image frame or otherwise provided to the system user.

[0079] Emitter 102 may be a laser emitter capable of emitting pulsed red light 104 to generate sensed red data 105 for identifying red elements 114 within a dark environment 112. Emitter 102 may also emit pulsed green light 106 to generate sensed green data 107 for identifying green elements 116 within a dark environment. Emitter 102 may also emit pulsed blue light 108 to generate sensed blue data 109 for identifying blue elements 118 within a dark environment. Emitter 102 may also emit pulsed electromagnetic radiation of a fluorescence excitation wavelength 110 for identifying fluorescent reagent 120 within the dark environment 112. Fluorescent reagent 120 is identified by exciting it with pulsed fluorescence excitation light 110 and then sensing the fluorescence relaxation wavelength 111 of a specific fluorescent reagent 120 (by pixel array 122). The transmitter 102 is capable of emitting pulses of red 104, pulses of green 106, pulses of blue 108, and pulses of fluorescence excitation 110 wavelengths in any desired order.

[0080] Pixel array 122 senses reflected electromagnetic radiation. Each of the sensed red 105, sensed green 107, sensed blue 109, and sensed fluorescence relaxation 111 data can be referred to as an "exposure frame". A specific color or wavelength partition is assigned to each exposure frame, where the assignment is based on the timing of the color or wavelength partition of the pulse from emitter 102. The combination of exposure frames and the assigned specific color or wavelength partition can be referred to as a dataset. Even if pixel 122 is not a dedicated color, colors can be assigned to any given dataset based on prior information about the emitter.

[0081] For example, during operation, after a pulse of red 104 light in a dark environment 112, pixel array 122 senses the reflected electromagnetic radiation. The reflected electromagnetic radiation generates an exposure frame, and this exposure frame is classified as sensed red 105 data because it corresponds temporally to the pulse of red 104 light. The exposure frame, along with its temporal correspondence to the pulse of red 104 light, is termed a “dataset.” This process is repeated for each allocation of electromagnetic radiation emitted by emitter 102. The data created by pixel array 122 includes sensed red 105 exposure frames that identify the red 114 component in the dark environment and correspond temporally to the pulse of red 104 light. The data also includes sensed green 107 exposure frames that identify the green 116 component in the dark environment and correspond temporally to the pulse of green 106 light. The data also includes sensed blue 109 exposure frames that identify the blue 118 component in the dark environment and correspond temporally to the pulse of blue 108 light. The data also includes a sensed fluorescence relaxation exposure frame 111 that identifies the fluorescent reagent 120 and corresponds temporally to the wavelength of the pulsed fluorescence excitation 110.

[0082] In one implementation, three datasets representing red, green, and blue electromagnetic pulses are combined to form a single image frame. Thus, information from the red, green, and blue exposure frames is combined to form a single RGB image frame. One or more additional datasets representing other wavelength partitions may be superimposed on the single RGB image frame. These additional datasets may represent, for example, fluorescence imaging in response to pulses excited at wavelengths between 770 nm and 790 nm and between 795 nm and 815 nm.

[0083] It should be understood that, without departing from the scope of this disclosure, this disclosure is not limited to any particular color combination or any particular electromagnetic partition, and any color combination or any electromagnetic partition can be used in place of RED, GREEN, and BLUE, such as cyan, magenta, and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible and invisible wavelengths. In the figure, the dark environment 112 to be imaged includes red 114, green 116, and blue 118 portions, and also includes a fluorescent reagent 120. As shown, the reflected light from the electromagnetic pulse contains only data of the portion of the object having a specific color corresponding to the pulse's color partition. These individual color (or color interval) datasets can then be used to reconstruct the image by combining the datasets at 126. Information in each of the multiple exposure frames (i.e., multiple datasets) can be combined by a controller 124, a control unit, a camera control unit, an image sensor, an image signal processing pipeline, or some other computational resource configured to process the multiple exposure frames and combine the datasets at 126. Data sets can be combined to generate a single image frame either within the endoscope unit itself or off-site by some other processing resources.

[0084] Figure 2 System 200 is provided for providing illumination to light-deficient environments, such as those used for endoscopic imaging. System 200 can be used in conjunction with any of the systems, methods, or apparatuses disclosed herein. System 200 includes a transmitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an end-cavity waveguide 210, an end-cavity 212, and an image sensor 214 with accompanying optical components, such as lenses. The transmitter 202 (which may be collectively referred to as the “light source”) generates light that travels through the jumper waveguide 206 and the end-cavity waveguide 210 to illuminate the scene at the distal end of the end-cavity 212. The transmitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, hyperspectral, fluorescence excitation, laser mapping pulse schemes, or other wavelengths. The end-cavity 212 can be inserted into a patient for imaging, such as during procedures or examinations. The output light is shown as dashed line 216. The image sensor 214 can be used to capture the scene illuminated by the light and display the scene to a physician or other medical personnel. Controller 204 can provide control signals to transmitter 202 to control when illumination is provided to a scene. In one embodiment, transmitter 202 and controller 204 are located within a camera control unit (CCU) or external console to which the endoscope is attached. If image sensor 214 includes a CMOS sensor, light can be periodically provided to the scene in a series of illumination pulses between readout cycles of image sensor 214 during a so-called blanking period. Therefore, the light can be pulsed in a controlled manner to avoid overlapping with the readout cycles of image pixels in the pixel array of image sensor 214.

[0085] In one embodiment, the cavity waveguide 210 includes one or more optical fibers. These optical fibers may be made of low-cost materials such as plastic to allow for the handling of the cavity waveguide 210 and / or other parts of the endoscope. In one embodiment, the cavity waveguide 210 is a single glass fiber with a diameter of 500 micrometers. A jumper waveguide 206 may be permanently attached to the transmitter 202. For example, the jumper waveguide 206 may receive light from a transmitter within the transmitter 202 and provide light to the cavity waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 206 includes one or more glass optical fibers. The jumper waveguide may include any other type of waveguide for guiding light to the cavity waveguide 210. The connector 208 may selectively couple the jumper waveguide 206 to the cavity waveguide 210 and allow light within the jumper waveguide 206 to pass through the cavity waveguide 210. In one embodiment, the cavity waveguide 210 is directly coupled to a light source without any intermediary jumper waveguide 206.

[0086] Image sensor 214 includes a pixel array. In one embodiment, image sensor 214 includes two or more pixel arrays for generating a three-dimensional image. Image sensor 214 may constitute two additional image sensors, each having its own independent pixel array and capable of operating independently of each other. The pixel array of image sensor 214 includes active pixels and optical black (“OB”) or optically blind pixels. Active pixels may be transparent, “color-indeterminate” pixels capable of sensing imaging data of electromagnetic radiation at any wavelength. Optical black pixels are read during the blanking period of the pixel array when the pixel array is “reset” or calibrated. In one embodiment, light pulses during the blanking period of the pixel array while the optical black pixels are being read. After the optical black pixels have been read, active pixels are read during the readout period of the pixel array. Active pixels may be charged by the electromagnetic radiation pulsed during the blanking period, such that the active pixels are ready to be read by the image sensor during the readout period of the pixel array.

[0087] Figure 2A This is a schematic diagram of complementary system hardware such as a dedicated or general-purpose computer. Specific embodiments within the scope of this disclosure may also include physical and other non-transitory computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible through a general-purpose or dedicated computer system. A computer-readable medium storing computer-executable instructions is a computer storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, specific embodiments of this disclosure may include at least two distinctly different types of computer-readable media: computer storage media (devices) and transmission media.

[0088] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to store program code tools in the form of desired computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0089] A “network” refers to one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. In one implementation, sensors and camera control units may be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transmitted or provided to a computer via a network or other communication connection (hard-wired, wireless, or a combination of hard-wired and wireless), the computer reasonably considers that connection as a transmission medium. The transmission medium may include networks and / or data links that can be used to carry program code tools in the form of desired computer-executable instructions or data structures and are accessible via general-purpose or special-purpose computers. The above combinations should also be covered within the scope of computer-readable media.

[0090] Furthermore, upon arrival at various computer system components, program code tools in the form of computer-executable instructions or data structures can be automatically transferred by a transmission medium to computer storage media (devices) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or the computer system's non-volatile computer storage media (devices). RAM may also include solid-state drives (SSDs) or PCIx-based real-time memory tiered storage devices, such as FusionIO. Therefore, it should be understood that computer storage media (devices) may be included in computer system components that also (or even primarily) utilize transmission media.

[0091] Computer-executable instructions include, for example, instructions and data that, when executed by one or more processors, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform certain functions or groups of functions. Computer-executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter of the invention has been set forth in terms of language with respect to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or steps described above. Rather, the features and steps described above are disclosed as examples of implementing the claims.

[0092] Those skilled in the art will understand that this disclosure can be implemented in a network computing environment with various types of computer system configurations, including personal computers, desktop computers, laptop computers, information processors, control units, camera control units, handheld devices, handheld components, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, various storage devices, etc. It should be noted that any of the aforementioned computing devices can be provided by or located within an entity. This disclosure can also be implemented in a distributed system environment, wherein local and remote computer systems are connected via a network (through a hard-wired data link, a wireless data link, or a combination of hard-wired and wireless data links), and both can perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0093] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to execute one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. Those skilled in the art will understand that components may have different names. This document is not intended to distinguish between components that are different in name rather than function.

[0094] Figure 2A This is a block diagram illustrating an exemplary computing device 250. The computing device 250 can be used to execute various programs, such as those discussed herein. The computing device 250 can be used as a server, client, or any other computing entity. The computing device 250 can perform various monitoring functions discussed herein and can execute one or more applications, such as those described herein. The computing device 250 can be any of a variety of computing devices, such as a desktop computer, laptop computer, server computer, handheld computer, camera control unit, tablet computer, etc.

[0095] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all of which are coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory devices 254 and / or mass storage devices 258. The processor 252 may also include various types of computer-readable media, such as cache memory.

[0096] The memory device 254 includes various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 264) and / or non-volatile memory (e.g., read-only memory (ROM) 266). The memory device 254 may also include rewritable ROM, such as flash memory.

[0097] Mass storage devices 258 include various computer-readable media, such as magnetic tape, magnetic disks, optical disks, solid-state storage (e.g., flash memory), etc. Figure 2 As shown, a specific mass storage device is hard disk drive 274. Various drives may also be included in mass storage device 258 to enable reading and / or writing from various computer-readable media. Mass storage device 258 includes removable media 276 and / or non-removable media.

[0098] I / O device 260 includes various means that allow input to or retrieval of data and / or other information to or from computing device 250. Exemplary I / O device 260 includes digital imaging device, electromagnetic sensor and transmitter, cursor control device, keyboard, keypad, microphone, monitor or other display device, speaker, printer, network interface card, modem, lens, CCD or other image capture device, etc.

[0099] Display device 280 includes any type of device capable of displaying information to one or more users of computing device 250. Examples of display device 280 include monitors, display terminals, video projection devices, and so on.

[0100] Interface 256 includes various interfaces enabling computing device 250 to interact with other systems, devices, or computing environments. Exemplary interface 256 may include any number of different network interfaces 270, such as interfaces for connecting to a local area network (LAN), wide area network (WAN), wireless network, and the Internet. Other interfaces include user interface 268 and peripheral device interface 272. Interface 256 may also include one or more user interface elements 268. Interface 256 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touchpad, etc.), keyboards, etc.

[0101] Bus 262 enables processor 252, memory device 254, interface 256, mass storage device 258, and I / O device 260 to communicate with each other and with other devices or components coupled to bus 262. Bus 262 represents one or more of several types of bus architectures (such as system bus, PCI bus, IEEE 1394 bus, USB bus, etc.).

[0102] For illustrative purposes, the programs and other executable devices shown herein are discrete entities; however, it should be understood that such programs and components may reside in different storage units of computing device 250 at various times and be executed by processor(s) 252(s). Alternatively, the systems and programs described herein may be implemented in hardware, or by a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays may be programmed to execute one or more systems and programs described herein.

[0103] Figure 3A The diagram illustrates the sensor's operational cycle during scroll readout mode or during sensor readout 300. A frame readout cycle may begin at and be represented by a vertical line 310. Readout cycle 302 is represented by a diagonal or slanted line. The sensor can be read out line by line, with the top of the downward-sloping edge designated as the top row 312 and the bottom of the downward-sloping edge designated as the bottom row 314. The time between the last row readout and the next readout cycle may be referred to as the blanking cycle 316. It should be noted that some of the sensor pixel rows may be covered with light-shielding material (e.g., a metallic coating or any other generally black layer of another material type). These covered pixel rows may be referred to as optical black rows 318 and 320. Optical black rows 318 and 320 can be used as input to a correction algorithm. Figure 3A As shown, these optical black rows 318 and 320 can be located on top of the pixel array, at the bottom of the pixel array, or at both the top and bottom of the pixel array.

[0104] Figure 3BA method is illustrated for controlling the amount of electromagnetic radiation (e.g., light) exposed to pixels so that it is integrated or accumulated by the pixels. It should be understood that photons are the fundamental particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into electrical charge or current. An electronic shutter or rolling shutter (shown as dashed line 322) can be used to begin the integration time by resetting the pixels. Light will then be integrated until the next readout cycle. The position of the electronic shutter 322 can be moved between two readout cycles 302 to control the pixel saturation for a given amount of light. It should be noted that this technique allows for a constant integration time between two different rows, but introduces a delay when moving from the top row to the bottom row.

[0105] Figure 3C The diagram shows the case where the electronic shutter 322 has been removed. In this configuration, the integration of incident light can begin during readout cycle 302 and end in the next readout cycle 302, which also defines the start of the next integration.

[0106] Figure 3D This illustrates a configuration without an electronic shutter 322 but with controlled and pulsed light 330 during the blanking period 316. This ensures that all lines see the same light emitted from the same light pulse 330. In other words, each line will begin its integration in a dark environment, which may be located after the optically black line 320 of the readout frame (m) to obtain the maximum light pulse width, and will then receive the light pass and end its integration in a dark environment, which may be located before the optically black line 318 of the next subsequent readout frame (m+1) to obtain the maximum light pulse width. For example... Figure 3D In this process, the image generated by the light pulse will only be available during the readout of frame (m+1) without interfering with frames (m) and (m+2). It should be noted that the condition for the light pulse to be read out only in one frame and not interfere with adjacent frames is that a given light pulse is fired during the blanking period 316. Because the optical black lines 318 and 320 are not sensitive to light, the optical black line following time 320 of frame (m) and the optical black line preceding time 318 of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 330.

[0107] like Figure 3A As shown, the sensor can cycle multiple times to receive data for each pulse color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum). Each cycle can be timed. In one embodiment, the cycle can be timed to operate at intervals of 16.67 ms. In another embodiment, the cycle can be timed to operate at intervals of 8.3 ms. It should be understood that other timing intervals are contemplated and are intended to fall within the scope of this disclosure.

[0108] Figure 4AThe operation of an embodiment of the electromagnetic transmitter is illustrated graphically. The transmitter can be timed to correspond to the sensor's operating cycle, such that electromagnetic radiation is emitted during and / or a portion of the sensor's operating cycle. Figure 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the transmitter may pulse during readout period 302 of a sensor operating cycle. In one embodiment, the transmitter may pulse during blanking period 316 of a sensor operating cycle. In one embodiment, the transmitter may pulse for a duration that spans a portion of two or more sensor operating cycles. In one embodiment, the transmitter may begin pulsed during blanking period 316 or during optical black portion 320 of readout period 302, and end pulsed during readout period 302 or during optical black portion 318 of the readout period 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of this disclosure, provided that the transmitter's pulses correspond to the sensor's cycles.

[0109] Figure 4B The control of exposure is graphically represented by varying the duration and magnitude of the emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, and pulse 3 at 416). A transmitter with a fixed output magnitude can be combined with the above... Figure 3D and Figure 4A During any given cycle, pulses are applied at regular intervals to provide the required electromagnetic energy to the pixel array. A transmitter with a fixed output value can pulse over longer time intervals, thus providing more electromagnetic energy to the pixels, or it can pulse over shorter time intervals, thus providing less electromagnetic energy. Whether longer or shorter time intervals are required depends on the operating conditions.

[0110] Compared to adjusting the time interval of a fixed output pulse value from the transmitter, increasing the pulse value itself can provide more electromagnetic energy to the pixel. Similarly, decreasing the pulse value provides less electromagnetic energy to the pixel. It should be noted that, if needed, the system implementation can have the ability to adjust both the pulse value and duration simultaneously. Furthermore, the sensor can be adjusted to increase its sensitivity and duration according to the requirements of optimal image quality. Figure 4BThe diagram illustrates variations in the magnitude and duration of pulses. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. Additionally, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416, such that the electromagnetic energy provided by this pulse is represented by the area under the pulse shown in the diagram. In the illustration, pulse 2 at 414 has a relatively lower magnitude or intensity and a longer duration compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration compared to pulse 1 at 412 and pulse 2 at 414.

[0111] Figure 5 For the combination of the principles and teachings of this disclosure Figures 3A to 3D and Figure 4A The illustration shows an embodiment of the imaging system disclosed herein, including its operating cycle, electromagnetic transmitter, and emitted electromagnetic pulses for display during operation. As can be seen in the figure, the electromagnetic transmitter pulses primarily during the blanking cycle 316 of the image sensor, causing the pixels to be charged and ready for reading during the readout cycle 302 of the image sensor cycle. Figure 5 The dashed lines in the image represent pulses of electromagnetic radiation (from...). Figure 4A The electromagnetic radiation pulses are primarily emitted during the blanking period 316 of the image sensor, but can be superimposed with the readout period 302 of the image sensor.

[0112] An exposure frame includes data read from the pixel array of the image sensor during readout cycle 302. The exposure frame can be combined with an indication of the type of pulse emitted by the transmitter prior to readout cycle 302. This combination of exposure frames and pulse type indication can be referred to as a dataset. Multiple exposure frames can be combined to generate a black-and-white or RGB color image. Additionally, hyperspectral, fluorescence, and / or laser mapping imaging data can be overlaid on the black-and-white or RGB image.

[0113] In one embodiment, the exposure frame is data sensed by the pixel array during readout period 302, which occurs after blanking period 316. Electromagnetic radiation is emitted during blanking period 316. In one embodiment, a portion of the electromagnetic radiation emission overlaps with readout period 302. Blanking period 316 occurs while the optical black pixels of the pixel array are being read, and readout period 302 occurs while the active pixels of the pixel array are being read. Blanking period 316 may overlap with readout period 302.

[0114] Figure 6A and Figure 6BThe process for recording image frames is illustrated. Multiple image frames can be chained together to generate a video stream. A single image frame may include data from multiple exposure frames, where the exposure frames are data sensed by a pixel array after the emission of electromagnetic radiation. Figure 6A The diagram illustrates a conventional process typically implemented with a color image sensor that has a color filter array (CFA) to filter out certain wavelengths of light for each pixel. Figure 6B This is the process disclosed in this paper, and it can be achieved using a monochrome "color-indeterminate" image sensor that receives electromagnetic radiation of all wavelengths.

[0115] Figure 6A The process shown occurs between time t(0) and time t(1). The process begins with white light emission 602 and white light sensing 604. At 606, the image is processed and displayed based on the sensing at 604.

[0116] Figure 6B The process shown occurs between time t(0) and time t(1). The process begins with the emission of green light 612, and the reflected electromagnetic radiation 614 is sensed after the emission of green light 612. The process continues with the emission of red light 616, and the reflected electromagnetic radiation 618 is sensed after the emission of red light 616. The process continues with the emission of blue light 620, and the reflected electromagnetic radiation 622 is sensed after the emission of blue light 620. The process continues with one or more emission at a fluorescence excitation wavelength 624, and the reflected electromagnetic energy 626 is sensed after each of the one or more emission at the fluorescence excitation wavelength of electromagnetic radiation 624. At 628, an image is processed and displayed based on each of the sensed reflected electromagnetic energy instances 614, 618, 622, and 626.

[0117] Figure 6B The method shown provides high-resolution images and provides apparatus for generating RGB images that also include fluorescence imaging data. When using a partitioned spectrum of light (such as...) Figure 6B As shown), this allows the sensor to be sensitive to electromagnetic energy across all wavelengths. Figure 6BIn the process illustrated, the monochromatic pixel array is instructed to sense electromagnetic energy from predetermined partitions of the full spectrum of electromagnetic energy in each cycle. Therefore, to form an image, the sensor only needs to cycle through multiple different partitions within the full spectrum of light. The final image is then combined based on these multiple cycles. Because the image from each color partition frame cycle (compared to a CFA pixel array) has higher resolution, the resulting image, when the light frames of the partitions are combined, also has higher resolution. In other words, because each pixel within the array (rather than at most every other pixel in a CFA sensor) senses the amplitude of the energy for a given pulse and a given scene only at intervals, a higher resolution image is produced for each scene.

[0118] As in Figures 6A to 6B The implementation shown can be seen from the graph between times t(0) and t(1) for... Figure 6B Sensors in the partitioned spectral system Figure 6A Each full-spectrum system in the system cycles at least four times. In one embodiment, the display device (LCD panel) operates at a rate of 50 to 60 frames per second. In such embodiments, Figure 6B The localized optical system can operate at rates from 200 frames per second to 240 frames per second to maintain the continuity and smoothness of the displayed video. In other implementations, different capture and display frame rates may exist. Furthermore, the average capture rate can be any multiple of the display rate.

[0119] In one implementation, it may be desirable that not all zones are represented equally within the system frame rate. In other words, not all light sources must pulse with the same regularity in order to emphasize and de-emphasize various aspects of the recorded scene as needed by the user. It should also be understood that invisible and visible zones of the electromagnetic spectrum can be pulsed together within the system, with their corresponding data values ​​stitched into the video output for display to the user.

[0120] The implementation scheme may include the following pulse cycle pattern:

[0121] i. Green pulse;

[0122] ii. Red pulse;

[0123] iii. Blue pulse;

[0124] iv. Green pulse;

[0125] v. Red pulse;

[0126] vi. Blue pulse;

[0127] vii. Fluorescence excitation pulse;

[0128] viii. (repeated)

[0129] The implementation scheme may include the following pulse cycle pattern:

[0130] i. Brightness pulse;

[0131] ii. Redness pulse;

[0132] iii. Luminance pulse;

[0133] iv. Blue light pulse;

[0134] v. Fluorescence excitation pulse;

[0135] vi. (to repeat)

[0136] The implementation scheme may include the following pulse cycle pattern:

[0137] i. Brightness pulse;

[0138] ii. Redness pulse;

[0139] iii. Luminance pulse;

[0140] iv. Blue light pulse;

[0141] v. Brightness pulse;

[0142] vi. Redness pulse;

[0143] vii. Brightness pulse;

[0144] viii. Blue light pulse;

[0145] ix. Fluorescence excitation pulse;

[0146] x. (repeated)

[0147] As can be seen in this example, the fluorescence excitation partition can be pulsed at a rate different from that of the other partitions. This emphasizes a particular aspect of the scene, where the fluorescence imaging data overlaps only with other data in the video output to make the desired emphasis. It should be noted that adding a fluorescence partition above the red, green, and blue partitions does not necessarily require the serialized system to operate at four times the rate of a full-spectrum non-serialized system, as each partition does not need to be represented equally in the pulse pattern. As seen in this embodiment, adding fewer partition pulses representing fewer partitions in the pulse pattern (fluorescence excitation in the example above) will result in an increase of less than 20% in the sensor's cycle speed to accommodate irregular partition sampling.

[0148] In various implementations, the pulse cyclic pattern may also include any of the following wavelengths in any suitable order. Such wavelengths are particularly suitable for exciting fluorescent reagents to generate fluorescence imaging data by sensing the relaxation emission of the fluorescent reagent based on its relaxation emission:

[0149] i.770±20nm;

[0150] ii. 770±10nm;

[0151] iii. 770±5nm;

[0152] iv. 790±20nm;

[0153] v.790±10nm;

[0154] vi. 790±5nm;

[0155] vii.795±20nm;

[0156] viii. 795±10nm;

[0157] ix.795±5nm;

[0158] x.815±20nm;

[0159] xi.815±10nm;

[0160] xii.815±5nm;

[0161] xiii. 770nm to 790nm; and / or

[0162] xiv. 795nm to 815nm.

[0163] Partition loops can be divided to adapt to or approximate various imaging and video standards. In one implementation, a partition loop includes the following: Figures 7A to 7D The pulses of electromagnetic energy in the red, green, and blue spectra are best illustrated in the image. The timing relationship between the emission of electromagnetic radiation pulses by the transmitter and the readout of the pixel array is shown in... Figures 7A to 7D This is further illustrated in the text.

[0164] In one embodiment, the emitter emits one or more hyperspectral emissions to elicit a spectral response. The hyperspectral emissions include one or more of electromagnetic radiation having wavelengths of about 513-545 nm, about 565-585 nm, and / or about 900-1000 nm. In such embodiments, the coherent light source 802 includes at least one laser emitter for the 513-545 nm region, at least one laser emitter for the 565-585 nm region, and at least one laser emitter for the 900-1000 nm region. It should be understood that additional hyperspectral emissions may be emitted to elicit a spectral response without departing from the scope of this disclosure.

[0165] In one embodiment, the emitter emits one or more fluorescence excitations for fluorescing the reagent. The fluorescence excitations include one or more electromagnetic radiations having wavelengths of about 460-470 nm, 529-537 nm, 633-643 nm, 775-785 nm, 800-810 nm, 970-980 nm, 575-579 nm, 519-527 nm, 770-790 nm, and / or 795-815 nm. In such embodiments, the coherent light source 802 may include at least one laser emitter for each of the aforementioned electromagnetic radiation zones. It should be understood that additional fluorescence excitations for fluorescing the reagent may be emitted without departing from the scope of this disclosure.

[0166] exist Figure 7A Different light intensities have been achieved by modulating the width or duration of the light pulse within the working range indicated by the vertical gray dashed line. Figure 7A The diagram illustrates the general timing relationship between the mixing of three wavelength pulses within a four-frame cycle and the readout cycle of the pixel array of the image sensor. In one embodiment, three monochromatic pulse light sources are present under the control of a controller. For example, a periodic sequence of monochromatic red, monochromatic green, and monochromatic blue exposure frames is captured using an RGBG pulse pattern and combined into an sRGB image frame by an image signal processor chain.

[0167] exist Figure 7B In this process, different light intensities are achieved by modulating the power of the optical power or the power of the electromagnetic transmitter (which can be a laser or an LED transmitter), while keeping the pulse width or duration constant.

[0168] Figure 7CThis illustrates a scenario where both optical power and pulse width are modulated for greater flexibility. Zoned cycling can utilize cyan, magenta, yellow (CMY), infrared, ultraviolet, hyperspectral, and fluorescence; invisible pulse sources mixed with visible pulse sources; and any other color space required to generate the image, or a desired video standard approximating currently known or yet-to-be-developed. It should also be understood that the system is capable of switching between operating color spaces to provide the desired image output quality.

[0169] Using the color space green-blue-green-red (e.g.) Figure 7D In the implementation shown, it may be desirable to pulse the luminance component more frequently than the chromaticity component, because users are generally more sensitive to differences in light intensity than to differences in light color. Examples of such implementations include... Figure 7D The monochrome image sensor shown utilizes this principle. Figure 7D In this configuration, green, which contains the most luminance information, can pulse more frequently or have greater intensity in the (GBGRGBGR…) scheme to obtain luminance data. This configuration will create a video stream with noticeably more detail, without creating and transmitting imperceptible data.

[0170] In one implementation, all three light sources are pulsed synchronously with light energy modulated to provide pure luminance information within the same exposure frame. The light energy can be modulated according to color conversion coefficients from the RGB color space to the YCbCr color space. It should be understood that the color conversion can be implemented according to any suitable standard such as ITU-R BT.709HD, ITU-R BT.601, ITU-R BT.2020, or any other suitable standard or formula. This conversion can be performed according to the ITU-R BT.709HD standard as follows:

[0171]

[0172] In addition to modulating luminance information, full-color images also require red and blue chrominance components. However, algorithms applied to the luminance component cannot be directly applied to the chrominance component because the algorithm is signed, as reflected in the fact that some RGB coefficients are negative. In one implementation, the luminance is increased to such that all final pulse energies are positive. As long as the color fusion processes in the image signal processor know the composition of the chrominance exposure frames, they can decode them by subtracting an appropriate amount of luminance from adjacent frames. The pulse energy ratio is given by the following formula:

[0173] Y=0.183·R+0.614·G+0.062·B

[0174] Cb=λ·Y-0.101·R-0.339·G+0.439·B

[0175] Cr=δ·Y+0.439·R-0.399·G-0.040·B

[0176] in

[0177]

[0178]

[0179] If the λ factor equals 0.552, the red and green components are canceled out. In this case, the blue hue information can be provided as pure blue light. Similarly, if the δ factor equals 0.650, the blue and green components are canceled out, and the red hue information can be provided as pure red light. This implementation is a convenient approximation for digital frame reconstruction.

[0180] In an implementation of white balance in the illumination domain, modulation is applied in addition to white balance modulation.

[0181] In one implementation, pulses that replicate weaker regions can be used to generate an output already adjusted for the weaker pulses. For example, blue lasers are considered to be less sensitive relative to silicon-based pixels and harder to generate than red or green light; therefore, they can be pulsed more frequently during frame cycles to compensate for the weakness of the light. These additional pulses can be performed continuously over time or by using multiple lasers pulsed simultaneously to produce the desired compensation effect. It should be noted that by pulses during the blanking period (the time during which the sensor does not read out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same type and simply focuses the light for the desired output. In another implementation, the maximum range of light pulses may differ from frame to frame. This is illustrated in… Figure 7E In this system, the light pulses are different from frame to frame. The sensor can be configured to be programmed with different blanking periods in repeating patterns of two, three, four, or n frames.

[0182] exist Figure 7E The diagram illustrates four distinct optical pulses, with pulse 1 repeating, for example, after pulse 4, and a pattern of four frames with different blanking periods. This technique can be used to place the most powerful partition on the smallest blanking period, thus allowing the weakest partition to have a wider pulse on a subsequent frame without increasing readout speed. The reconstructed frame can still have a regular frame-to-frame pattern because it consists of many pulse frames.

[0183] Figure 8A graphical representation of the delay or jitter between control signal 802 and the emission of electromagnetic radiation 804 is shown. In one embodiment, control signal 802 represents a signal provided to a driver of the transmitter. The driver is configured to cause transmitter 202 to emit electromagnetic radiation pulses. In one embodiment, the driver is a component of controller 204, or may be independent of and communicate with controller 204. In one embodiment, the driver is controller 204. In one embodiment, the driver is a component of transmitter 202 or communicates with transmitter 202. As shown, there is a duration delay t1 between the control signal 802 reaching its peak (i.e., turning on) and transmitter 202 emitting electromagnetic radiation 804. There is a duration delay t2 between the control signal 802 decreasing (i.e., turning off) and the end of electromagnetic radiation emission 804.

[0184] For example, delays t1 and t2 may include some constant delays as well as some non-constant variations caused by jitter in the transmitter's driver. For instance, a constant delay may exist when control signal 802 is transmitted to the driver, and when transmitter 202 actually emits electromagnetic radiation at emission 804. This delay can be very short and can be based on the time required for electrical communication to occur between the driver and the transmitter. Non-constant variations in delay can be a result of jitter in the transmitter's driver, in the controller 204, and / or in the transmitter itself.

[0185] The jitter experienced by a system or its components (such as the transmitter's driver) can be described by a value known as a jitter specification. A jitter specification is a numerical value that describes the amount or duration of jitter experienced by the system. Figure 8 In the example shown, delay t1 has a shorter duration than delay t2. In this example, delay t1 can represent the constant delay experienced after the start control signal 802 and the transmitter emits electromagnetic radiation at emission 804. The difference between t2 and t1 can represent the jitter experienced by the system. This value can be referred to as the jitter specification.

[0186] In one implementation, the jitter specification is a numerical value representing the amount of variation in a constant or predictable delay used to initiate or deactivate an electromagnetic radiation pulse (EMIP). In such implementations, the system experiences a constant, predictable delay between when the driver signals the transmitter to emit the EMIP and when the transmitter actually initiates the EMIP. Similarly, a constant, predictable delay may exist between when the transmitter should deactivate the EMIP and when the transmitter actually deactivates the EMIP. This constant, predictable delay does not represent the jitter specification. Rather, the jitter specification is the variation of this constant, predictable delay. Figure 8 In the example shown, the difference between times t2 and t1 represents a change in constant, predictable delay.

[0187] Jitter is beyond the user's control over the system. Jitter specification represents a quantity of unpredictable and non-constant time-varying variations present in the system. If the jitter specification is too large relative to an electromagnetic radiation pulse, a significant degrade in image quality or variation in image brightness will occur in the resulting exposure frames. For example, in a video endoscope system as discussed herein, a long jitter specification can cause exposure frames in different rows within the video stream to have different brightness. This results in flickering and overall quality degradation in the video stream. A long jitter specification can cause light to be emitted during the readout cycle 302 of the image sensor. If pulsed electromagnetic radiation occurs during readout cycle 302, significant variations will occur between pixels and pixel rows in the pixel readings, and this degrades the image quality in the resulting video stream.

[0188] In an exemplary implementation, controller 204 has a jitter specification of 10% of the duration of the electromagnetic radiation pulse. In this example, the pulse can vary from 90% to 110% of its desired duration. This can result in brightness variations of up to one-third between exposure frames or rows within the image frame of the video.

[0189] In one implementation, if the jitter specification has a duration longer than a threshold amount, the electromagnetic radiation pulse is limited for that duration to avoid overlapping into the readout period 302. Limiting the pulse duration may necessitate reducing the frame rate by increasing the time between captured exposure frames and / or increasing the duration of the blanking period 316. This can result in a reduction in image brightness, and this can further reduce the image sensor's ability to capture detailed images.

[0190] In one embodiment, if the jitter specification has a duration shorter than a threshold amount, the pulse sequence of transmitter 202 and the readout sequence 204 of the image sensor remain unchanged. In one embodiment, the threshold indicates that the jitter specification must be 1 microsecond or less. In one embodiment, the threshold indicates that the jitter specification must be 50 nanoseconds or less. In one embodiment, the threshold indicates that the jitter specification must be less than the time it takes for the image sensor to read out one row of the pixel array. In one embodiment, the threshold indicates that the jitter specification must be less than the time it takes for the image sensor to read out a single pixel of the pixel array. In one embodiment, the threshold indicates that the jitter specification may be less than or equal to 10% to 25% of the readout period 302 of the image sensor, or the time required for the image sensor to read out all valid pixels in the pixel array. For example, in such an embodiment, if the pixel array comprises 400 rows, the jitter specification must be less than or equal to the time required to read out 40 to 100 rows of the 400 rows in the pixel array. Therefore, the amount of variation in the light captured by the pixel array can be low enough to reduce image flicker and / or provide as much light as possible between readout periods 302.

[0191] In one implementation, jitter specifications are reduced (shortened) by implementing a higher clock rate or a more accurate clock in the driver of controller 204 or transmitter 202. Reduced jitter specifications and tolerances in the driver of transmitter 202 address issues of driver intolerance that lead to artifacts in the resulting video stream.

[0192] In one embodiment, the camera control unit (CCU) signals the controller 204 or transmitter 202 to avoid overlapping electromagnetic radiation pulses with the readout period 302 of the image sensor. The CCU can determine the signals to be sent to the controller 204 and / or transmitter 202 to avoid overlap with the readout of valid (i.e., not optically black) pixels in the pixel array. The CCU can maximize the duration of the temporal electromagnetic radiation emitted by the transmitter 202 without overlapping the readout period 302 of the image sensor.

[0193] Figure 9 A cross-section of an optical fiber bundle 900 used to carry electromagnetic radiation from transmitter 202 into a dark environment to illuminate the scene is shown. Figure 9 In the exemplary embodiment shown, the fiber bundle 900 includes seven fibers; however, it should be understood that the number of fibers is merely illustrative and any suitable number of fibers may be used outside the scope of this disclosure. The fiber bundle includes a central fiber 902 and a plurality of surrounding fibers 904.

[0194] In one embodiment, the total number of fibers is limited to reduce the cross-sectional area of ​​the fiber bundle 900. The fiber bundle 900 may include a suitable number of fibers to provide sufficient light dispersion while allowing a small cross-sectional area. This may be desirable because the cross-sectional area of ​​the endoscope lumen is critical in some applications that require a small endoscope. In one embodiment, the fiber bundle 900 may include 2 to 150 fibers. A smaller number of fibers reduces the cost and cross-sectional area required to carry the fiber bundle 900. However, a larger number of fibers improves redundancy. In one embodiment, the fiber bundle 900 includes 5 to 100 fibers, 5 to 50 fibers, or 7 to 15 fibers. In one embodiment, the fiber bundle includes seven fibers, such as... Figure 9 As shown.

[0195] When the fiber bundle 900 has a small number of fibers, it may be desirable for each fiber to receive the same amount of electromagnetic radiation and the same amount of electromagnetic radiation of a specific wavelength. For example, if the electromagnetic radiation is primarily transmitted through the central fiber 902, the central fiber 902 will receive most of the electromagnetic radiation, and the scene will be illuminated in a colored or unevenly bright manner. Furthermore, if more light enters one fiber than another, the total amount (power) of electromagnetic radiation that can be carried in the fiber bundle 900 will be reduced. For example, if electromagnetic radiation above a certain energy level or intensity is supplied to a fiber, the fiber may have melting limits that can cause it to melt or otherwise fail. Therefore, if the electromagnetic radiation is distributed more evenly across the fibers, it is possible to increase the power and illumination of the scene.

[0196] In one embodiment, transmitter 202 mixes electromagnetic radiation of two or more wavelengths before providing electromagnetic radiation to fiber bundle 900. This can be achieved when transmitter 202 includes two or more independent laser beams for emitting electromagnetic radiation of different wavelengths. Transmitter 202 may include, for example, a first laser beam for emitting a first wavelength and a second laser beam for emitting a second wavelength. Transmitter 202 may mix the electromagnetic radiation such that light from the first laser beam and light from the second laser beam enter the jumper waveguide (or another waveguide) at the same or substantially the same angle. The same or substantially the same angle can be achieved by positioning the laser beams at the same angle relative to each other. In one embodiment, a dichroic mirror allows the same or substantially the same angle by reflecting electromagnetic radiation of one wavelength while being transparent to another wavelength. In one embodiment, transmitter 202 includes a diffuser, mixing rod, lens, or other optical element to mix the light before it enters fiber bundle 900.

[0197] In one embodiment, emitter 202 provides a uniformly distributed light intensity to the waveguide. The peak intensity of the light in the region where it collects light for the waveguide may be substantially equal to or close to the average intensity of the light in that region. The light provided to the collection region may have a top-hat profile, such that each fiber collects and / or receives light of the same or similar intensity. Emitter 202 can provide or approximate a top-hat profile by providing a laser beam at an angle to the surface of the collection region. For example, emitter 202 may include a Gaussian or other non-constant intensity profile. By angulating the laser beam relative to the collection region, the Gaussian profile is flattened into a more constant profile or a top-hat profile. Lenses, diffusers, mixing rods, or the like can be used to generate the top-hat profile.

[0198] Figure 10The top-hat profile 1002 and the Gaussian profile 1004 are illustrated graphically. The horizontal axis represents the horizontal distance, and the vertical axis represents the light intensity. The row labeled 1006 in the figure represents the boundary or width of the collection area 1006 of the fiber bundle 900. The row labeled 1008 in the figure represents the fusion level 1008 for the fiber or other waveguide.

[0199] Using a Gaussian profile 1004, most of the electromagnetic radiation is directed to the central fiber 902. With most of the energy located in the central fiber 902, the remaining surrounding fibers 904 can remain well below the fusing level 1008. For example, using the Gaussian profile 1004, an increase in the total energy can cause the central fiber 902 to significantly exceed the fusing level 1008, while multiple surrounding fibers 904 remain well below the fusing level 1008.

[0200] Using the top-hat profile 1002, all fibers carry the same energy level. This energy level can be close to or below the fusion break level 1008. For example, using the top-hat profile 1002, the total energy carried by the fiber bundle 900 can be significantly increased because the fiber bundle 900 can be concentrated near the fusion break level 1008 without the risk of fusing any individual fiber.

[0201] Figure 10 This illustrates how implementing a top-hat profile 1002 can provide more energy before any single fiber reaches the fusing level 1008. For example, Gaussian profile 1004 and top-hat profile 1002 can provide the same amount of watts to the fiber bundle 900, while top-hat profile 1002 can still significantly increase the amount of watts before reaching the fusing level 1008. Therefore, a significant improvement in the total amount of energy delivered can be achieved using plastic fibers. In some cases, an increase of 50% or more in the watts carried by the fiber bundle 900 can be achieved by implementing top-hat profile 1002. In one embodiment, the plastic fibers may have a fusing energy level for optical / electromagnetic energy emitted by one or more transmitters, above which would damage the plastic fibers, wherein the optical energy propagates across multiple plastic fibers to allow the fiber bundle 900, including the plastic fibers, to carry a greater amount of energy without reaching the fusing level 1008 of any single fiber.

[0202] In one embodiment, the top-hat profile 1002 and the Gaussian profile 1004 are combined by the transmitter 202 for use with the plastic fiber bundle 900. The transmitter 202 and / or the jumper waveguide may not include a plastic waveguide. However, the transmitter 202 may blend the Gaussian profile 1004 with the top-hat profile 1002 to allow use with the plastic fiber bundle 900 at the cavity waveguide. In one embodiment, the blended top-hat profile 1002 allows for greater power delivery, considering the losses that can occur when electromagnetic radiation is moved between different materials (e.g., from diffuser to glass fiber, to plastic fiber, and / or back to glass fiber or diffuser). This greater power delivery can offset losses in previous or subsequent transitions, ensuring that sufficient light is still delivered to illuminate the scene.

[0203] Figure 11 This is a side view showing the output from the fiber bundle 1102 compared to the camera's field of view. In one embodiment, the plastic fiber has a numerical aperture of 0.63, with a field of view of 100 degrees, as shown by dashed line 1106. The glass fiber has a numerical aperture of 0.87, with a field of view of 120 degrees, as shown by solid line 1104. However, light emitted within the field of view has an approximately Gaussian profile within a light cone smaller than that field of view. For example, almost all the light used for the plastic fiber can be within an 80-degree cone, as shown by dashed line 1108. Therefore, the central region of the exposure frame may be too bright and the edges too dark.

[0204] Figure 12 It shows relative to Figure 11 The output shown is a side view of the output from fiber bundle 1202, exhibiting a more uniform light distribution. Figure 12 In the illustrated embodiment, uniform light distribution is achieved by aiming the light off the ends of the fibers in the fiber bundle 1202. Aiming the fibers away from the center widens the cones in the field of view and eliminates light loss at the output ends. One end of each fiber can be held in a desired position to distribute light, where the combination of light cones from the fibers provides uniform illumination. The fiber bundle 1202 includes multiple fibers and rows 1204 indicating the orientation of the cones output from individual fibers. In one embodiment, a fixing device such as a physical mold or a sheet with holes holds the ends of the fibers in a desired orientation. The fibers can be oriented to the optimal orientation for uniform illumination of the scene. The ends of the fibers in the fiber bundle 1202 can be located near the distal end of an endoscope and can be directed to propagate light around an area centered on the focal point or camera lens axis.

[0205] Figure 13This is a side view showing the output of the fiber bundle 1302, which transitions from plastic fiber 1304 to glass fiber 1306 at connector 1308. In this embodiment, the cavity waveguide includes plastic fiber 1304, which then transitions to glass fiber 1306 at or near the output. Glass fiber 1306 typically has a higher numerical aperture and a wider field of view than plastic fiber 1304. Therefore, as shown by light cone 1310, a wider and more uniform light energy distribution is achieved. Light passing through plastic fiber 1304 is guided to glass fiber 1306 via connector 1308. This coupling can occur within the handpiece unit of the endoscope's cavity. Connector 1308 can be positioned in the handpiece or cavity to limit the amount of glass fiber 1306 used. Moving from plastic fiber 1304 to glass fiber 1306 via a tapered member in the handpiece or cavity produces the same field of view as conventional endoscopes. However, compared to the targeting implementation (which does not experience optical loss at the output), optical loss can be significant, such as about 25%.

[0206] Figure 14 This is a side view showing the light output from the fiber bundle 1402 using a diffuser 1408. In this embodiment, the cavity waveguide includes plastic fibers 1404, which then transition to the diffuser 1408 at or near the output. The diffuser 1408 can include any suitable optical diffuser, such as a hybrid rod. Exemplary diffusers include holographic diffusers. The diffuser 1408 at the output can produce a larger field of view compared to glass fibers. However, the diffuser 1408 is less efficient, such as compared to... Figure 12 The targeting implementation shown is approximately 40% to 60% more efficient.

[0207] Plastic fibers are generally cheaper than glass fibers. This lower price can lead to significant savings in manufacturing lighting systems. Significant cost savings are possible because glass can only be used in short distances near the output, or not at all.

[0208] In one implementation, a single fiber replaces the fiber bundle. The single fiber can be larger than the typical fibers constituting the fiber bundle, allowing it to handle greater power than a smaller fiber bundle for the same cross-sectional area. The single fiber can extend from the console and through the cavity to provide light to the interior of the body or other light-deficient environments. The single fiber can operate as an intracavity waveguide extending from transmitter 202 or a jumper waveguide and through the cavity. Electromagnetic radiation can be directly supplied by transmitter 202 to the single fiber, which has a top-cap profile.

[0209] Because plastic fibers can have numerical apertures of only 0.63 or 0.65, most electromagnetic radiation can be emitted at angles of only 70 or 80 degrees. At the output of a single fiber, a diffuser can be positioned to propagate the output light and create more uniform illumination within the camera's field of view. In one implementation, the type of diffuser or its presence can be based on the field of view used by the camera. For example, laparoscopic surgery may allow a narrower field of view, such as 70 degrees, while arthroscopic surgery may use a wider field of view, such as 110 degrees. Therefore, a diffuser may be used for arthroscopy, while it may not be necessary for laparoscopy.

[0210] It should be understood that the implementation scheme for outputting electromagnetic radiation (light) may include Figures 11 to 14 The embodiments shown are combinations of one or more of the embodiments. For example, plastic fibers can be converted to glass fibers, and glass fibers can be designed to provide more uniform and improved lighting.

[0211] Figure 15 This is a schematic flowchart illustrating an exemplary method 1500 for providing light to an imaging scene in a low-light environment. Method 1500 can be provided by a lighting system (such as...) Figure 1 (System 100) executes.

[0212] Method 1500 begins, and at 1502, the image sensor generates and reads out pixel data for an image from the image sensor based on light received by the image sensor, wherein the time length for reading out a row of pixel data includes a row readout length. At 1504, the transmitter emits light to illuminate the scene observed by the image sensor. At 1506, the driver drives the transmitter to emit, wherein the driver includes a jitter specification less than or equal to the row readout length. At 1508, the controller controls the driver to drive the transmitter to generate light pulses between readout cycles of the image sensor.

[0213] Figure 16 This is a schematic flowchart illustrating an exemplary method 1600 for providing light to an imaging scene in a low-light environment. Method 1600 can be provided by a lighting system (such as...) Figure 1 (System 100) executes.

[0214] Method 1600 begins, and the first and second transmitters emit light comprising a first wavelength and a second wavelength at 1602. Multiple optical fibers at 1604 guide the light generated by the first and second transmitters into the scene within the endoscopic environment. At 1606a, the multiple optical fibers receive substantially equal amounts of light (mixed light) from the first and second transmitters at each of the multiple optical fibers.

[0215] Figures 17A to 17CLight sources 1700 with multiple emitters are shown. These emitters may alternatively be referred to as “laser beams,” wherein each emitter / laser beam may operate independently of the other emitters / laser beams and / or pulse different zones or wavelengths of the electromagnetic spectrum. Light sources 1700 may be collectively referred to herein as “emitters.” The multiple emitters include a first emitter 1702, a second emitter 1704, and a third emitter 1706. Additional emitters may be included, as discussed further below. Emitters 1702, 1704, and 1706 may include one or more laser emitters that emit light with different wavelengths. For example, the first emitter 1702 may emit a wavelength consistent with blue laser light, the second emitter 1704 may emit a wavelength consistent with green laser light, and the third emitter 1706 may emit a wavelength consistent with red laser light. For example, the first emitter 1702 may include one or more blue lasers, the second emitter 1704 may include one or more green lasers, and the third emitter 1706 may include one or more red lasers. Lasers 1702, 1704, and 1706 emit laser beams toward a collection region 1708, which may be a waveguide, a lens, or a device for collecting light and / or directing it toward the waveguide (such as...). Figure 2 The jumper waveguide 206 or cavity waveguide 210 provides the location of other optical components for light.

[0216] In one specific implementation, transmitters 1702, 1704, and 1706 emit electromagnetic radiation at hyperspectral wavelengths. Certain hyperspectral wavelengths can penetrate tissue and enable a physician to "see through" the tissue in front to identify chemical processes, structures, compounds, biological processes, etc., located behind the tissue. Hyperspectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc., known to have specific spectral responses.

[0217] In specific embodiments where reagents or dyes that aid in the identification of certain tissues, structures, chemical reactions, biological processes, etc., have been administered to a patient, emitters 1702, 1704, and 1706 may emit wavelengths intended to fluoresce the reagents or dyes. Such wavelengths may be determined based on the reagents or dyes administered to the patient. In such embodiments, the emitters may need to be highly precise in order to emit the desired wavelengths to fluoresce or activate certain reagents or dyes.

[0218] In one implementation, transmitters 1702, 1704, and 1706 emit laser mapping patterns for mapping the topography of a scene and / or for calculating the dimensions and distances between objects in the scene. In one implementation, the endoscopic imaging system is used in conjunction with multiple tools such as surgical scalpels, retractors, clamps, etc. In such implementations, each of transmitters 1702, 1704, and 1706 can emit a laser mapping pattern such that the laser mapping pattern is projected individually onto each tool. In such implementations, the laser mapping data of each tool can be analyzed to identify the distances between the tool and other objects in the scene.

[0219] exist Figure 17B In this implementation, transmitters 1702, 1704, and 1706 each deliver laser light to collection region 1708 at different angles. The change in angle can cause a change in the position of electromagnetic energy within the output waveguide. For example, if light enters the fiber bundle (glass or plastic) immediately at collection region 1708, the changing angle can cause different amounts of light to enter different fibers. For example, the angle can cause an intensity variation across collection region 1708. Furthermore, light from different transmitters may not be uniformly mixed, so some fibers may receive different amounts of different colors of light. Variations in the color or intensity of light in different fibers can lead to suboptimal lighting of the scene. For example, variations in delivered light or light intensity can cause this in both the scene and the captured image.

[0220] In one embodiment, an intervening optics element may be placed between the fiber bundle and emitters 1702, 1704, 1706 to mix light of different colors (wavelengths) before it enters the fiber or other waveguide. Exemplary intervening optics elements include diffusers, mixing rods, one or more lenses, or other optical components for mixing light such that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. This mixing may result in the same color in each fiber; however, in some embodiments, it may still result in different total brightness delivered to different fibers. In one embodiment, the intervening optics element may also propagate or uniformly distribute light over the collection area such that each fiber carries the same total amount of light (e.g., the light may be diffused in a top-hat profile). Diffusers or mixing rods may result in light loss.

[0221] Although the collection area 1708 is in Figure 17A The term 1708 is used to refer to physical components, but the collection region 1708 may simply be the region that delivers light from transmitters 1702, 1704, and 1706. In some cases, the collection region 1708 may include optical components such as diffusers, mixing rods, lenses, or any other intermediary optical components located between transmitters 1702, 1704, 1706, and the output waveguide.

[0222] Figure 17C An embodiment of a light source 1700 is shown, having emitters 1702, 1704, 1706 that provide light to a collection region 1708 at the same or substantially the same angle. The light is provided at an angle substantially perpendicular to the collection region 1708. The light source 1700 includes a plurality of dichroic mirrors, including a first dichroic mirror 1710, a second dichroic mirror 1712, and a third dichroic mirror 1714. Dichroic mirrors 1710, 1712, and 1714 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 1714 may reflect blue laser light provided by a third emitter, while being transparent to red and green light provided by the first emitter 1702 and the second emitter 1704, respectively. The second dichroic mirror 1712 may be transparent to red light from the first emitter 1702 but reflect green light from the second emitter 1704. If other colors or wavelengths are included, dichroic mirrors can be selected to reflect light corresponding to at least one emitter and be transparent to the other emitters. For example, a third dichroic mirror 1714 reflects light from a third emitter 1706 but is transparent to emitters "behind" it, such as a first emitter 1702 and a second emitter 1704. In embodiments where dozens or hundreds of emitters exist, each dichroic mirror can reflect light from its corresponding emitter and the emitters in front of it, while being transparent to the emitters behind it. This allows dozens or hundreds of emitters to project electromagnetic energy into the collection area 1708 at substantially the same angle.

[0223] Because these dichroic mirrors allow other wavelengths to transmit or pass through, each of these wavelengths can reach the collection area 1708 from the same angle and / or at the same center point or focal point. Providing light from the same angle and / or the same focal point / center point significantly improves reception and color mixing at the collection area 1708. For example, a particular fiber can receive different colors in proportions identical to the proportions of transmission / reflection by emitters 1702, 1704, 1706 and mirrors 1710, 1712, 1714. Figure 17B Compared to the previous implementation, this significantly improves light mixing at the collection region. In one implementation, any of the optical components discussed herein may be used at the collection region 1708 to collect light before it is supplied to the fiber or fiber bundle.

[0224] Figure 17CAn embodiment of a light source 1700 is shown having emitters 1702, 1704, 1706 that also provide light to a collection region 1708 at the same or substantially the same angle. For example, the light incident on the collection region 1708 is deflected from the vertical. Angle 1716 indicates the angle of deflection from the vertical. In one embodiment, laser emitters 1702, 1704, 1706 may have a Gaussian cross-sectional intensity profile. As previously mentioned, an improved distribution of optical energy between fibers can be achieved by forming a flatter or cap-shaped intensity profile. In one embodiment, as angle 1716 increases, the intensity across the collection region 1708 approaches a cap-shaped profile. For example, by increasing angle 1716 until the profile is sufficiently flat, a cap-shaped profile can even approximate a non-flat output beam. A cap-shaped profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intermediary optical components located between emitters 1702, 1704, 1706 and the output waveguide, fiber, or fiber bundle.

[0225] Figure 18 This is a schematic diagram showing a single optical fiber 1802 output via a diffuser 1804 at the output point. In one embodiment, the optical fiber 1802 has a diameter of 500 micrometers, a numerical aperture of 0.65, and emits a light cone 1806 of approximately 70 or 80 degrees without the diffuser 1804. Using the diffuser 1804, the light cone 1806 can have an angle of approximately 110 or 120 degrees. The light cone 1806 can be the majority of the area where all light arrives and is uniformly distributed. The diffuser 1804 allows for a more uniform distribution of electromagnetic energy in the scene observed by the image sensor.

[0226] In one embodiment, the cavity waveguide 210 includes a single plastic or glass optical fiber of approximately 500 micrometers. Plastic fibers are less expensive, but their width, through coupling, diffusion, or other losses, allows the fiber to carry a sufficient amount of light into the scene. For example, a smaller fiber may not be able to carry as much light or power as a larger fiber. The cavity waveguide 210 may include a single or multiple optical fibers. The cavity waveguide 210 may receive light directly from a light source or via a jumper waveguide. A diffuser may be used to widen the light output 206 to obtain the desired field of view for the image sensor 214 or other optical components.

[0227] Although Figures 17A to 17CThree emitters are shown, but in some embodiments, the number of emitters can range from one to hundreds or more. The emitters may have different wavelengths or spectra of light emitted, and these lights can be used to continuously cover desired portions of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra). The emitters may be configured to emit visible light such as red, green, and blue light, and may be further configured to emit hyperspectral emission of electromagnetic radiation, fluorescence excitation wavelengths for fluorescing reagents, and / or laser mapping patterns for calculating parameters and distances between objects in a scene.

[0228] Figure 19 A portion of an electromagnetic spectrum 1900 divided into twenty distinct sub-spectrums is shown. The number of sub-spectrums is merely exemplary. In at least one embodiment, spectrum 1900 may be divided into hundreds of sub-spectrums, each having a wavelet band. The spectrum may extend from the infrared spectrum 1902, through the visible spectrum 1904, and into the ultraviolet spectrum 1906. Each sub-spectrum has a wavelet band 1908 covering a portion of spectrum 1900. Each wavelet may be defined by an upper wavelength and a lower wavelength.

[0229] Hyperspectral imaging includes imaging information from across the electromagnetic spectrum 1900. Hyperspectral pulses of electromagnetic radiation may include multiple sub-pulses spanning one or more portions of the electromagnetic spectrum 1900 or the entire electromagnetic spectrum 1900. A hyperspectral pulse of electromagnetic radiation may include a single partition of the wavelength of the electromagnetic radiation. The resulting hyperspectral exposure frame includes information sensed by the pixel array following the hyperspectral pulse of electromagnetic radiation. Therefore, a hyperspectral exposure frame may include data from any suitable partition of the electromagnetic spectrum 1900, and may include multiple exposure frames from multiple partitions of the electromagnetic spectrum 1900. In one embodiment, a hyperspectral exposure frame includes multiple hyperspectral exposure frames such that a combined hyperspectral exposure frame includes data from the entire electromagnetic spectrum 1900.

[0230] In one embodiment, for each sub-spectrum, at least one emitter (such as a laser emitter) is included in a light source (such as light source 202, 1700) to provide complete and continuous coverage of the entire spectrum 1900. For example, the light source used to provide coverage of the illustrated sub-spectrum may include at least 20 different emitters, with at least one emitter for each sub-spectrum. In one embodiment, each emitter covers a 40-nanometer spectral band. For example, one emitter may emit light in a band from 500 nm to 540 nm, while another emitter may emit light in a band from 540 nm to 580 nm. In another embodiment, the emitters may cover bands of other sizes, depending on the type of emitter available or the imaging requirements. For example, multiple emitters may include a first emitter covering a band from 500 nm to 540 nm, a second emitter covering a band from 540 nm to 640 nm, and a third emitter covering a band from 640 nm to 650 nm. Each transmitter can cover different segments of the electromagnetic spectrum, ranging from far-infrared, mid-infrared, near-infrared, visible, near-ultraviolet, and / or far-ultraviolet. In some cases, multiple transmitters of the same type or wavelength may be included to provide sufficient output power for imaging. The number of transmitters required for a particular waveband may depend on the monochromatic sensor's sensitivity to the waveband and / or the power output capability of the transmitters in that waveband.

[0231] The bandwidth and coverage provided by the emitter can be selected to provide any desired combination of spectra. For example, continuous coverage of the spectrum using a very small bandwidth (e.g., 10 nm or less) allows for highly selective hyperspectral and / or fluorescence imaging. Bandwidth allows for the selective emission of excitation wavelengths of one or more specific fluorescent reagents. Additionally, bandwidth allows for the selective emission of certain zones of hyperspectral electromagnetic radiation for the identification of specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths originate from a selectively activated emitter, great flexibility is achieved in fluorescing to one or more specific fluorescent reagents during examination. Furthermore, extreme flexibility is achieved in identifying one or more objects or processes via hyperspectral imaging. Therefore, more fluorescence and / or hyperspectral information can be obtained in less time and in a single examination, which would otherwise require multiple examinations, be delayed due to dye application or staining, etc.

[0232] Figure 20This is a schematic diagram illustrating the timing diagram 2000 for emission and readout used to generate an image. The solid lines represent the readout period (peak 2002) and blanking period (valley) for capturing a series of exposure frames 2004 to 2014. This series of exposure frames 2004 to 2014 may include a series of repeated exposure frames that can be used to generate laser mapping data, hyperspectral data, and / or fluorescence data that can be overlaid on an RGB video stream. In one embodiment, a single image frame includes information from multiple exposure frames, one of which includes red image data, another includes green image data, and yet another includes blue image data. Additionally, a single image frame may include one or more of hyperspectral image data, fluorescence image data, and laser mapping data. Multiple exposure frames are combined to produce a single image frame. The single image frame is an RGB image with hyperspectral imaging data. The series of exposure frames includes a first exposure frame 2004, a second exposure frame 2006, a third exposure frame 2008, a fourth exposure frame 2010, a fifth exposure frame 2012, and an Nth exposure frame 2026.

[0233] Furthermore, hyperspectral imaging data, fluorescence imaging data, and laser mapping data can be used in combination to identify key tissues or structures, and further measure the dimensions of those key tissues or structures. For example, hyperspectral imaging data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, cancerous tissue, etc. The location and identification of key structures can be received from the corresponding system and can also be used to generate the topology of key structures using laser mapping data. For example, the corresponding system determines the location of a cancerous tumor based on hyperspectral imaging data. Since the location of the cancerous tumor is known based on the hyperspectral imaging data, the morphology and distance of the cancerous tumor can be calculated based on the laser mapping data. This example can also be applied when identifying cancerous tumors or other structures based on fluorescence imaging data.

[0234] In one implementation, each exposure frame is generated based on at least one pulse of electromagnetic energy. The electromagnetic energy pulse is reflected and detected by an image sensor and subsequently read out in a subsequent readout (2002). Thus, each blanking period and readout results in an exposure frame for a specific electromagnetic energy spectrum. For example, a first exposure frame 2004 may be generated based on the spectrum of a first one or more pulses 2016, a second exposure frame 2006 may be generated based on the spectrum of a second one or more pulses 2018, a third exposure frame 2008 may be generated based on the spectrum of a third one or more pulses 2020, a fourth exposure frame 2010 may be generated based on the spectrum of a fourth one or more pulses 2022, a fifth exposure frame 2012 may be generated based on the spectrum of a fifth one or more pulses, and an Nth exposure frame 2026 may be generated based on the spectrum of an Nth one or more pulses 2026.

[0235] Pulses 2016 to 2026 may include energy from a single emitter or a combination of two or more emitters. For example, the spectrum may be selected to be included within a single readout cycle or multiple exposure frames 2004 to 2014 for desired examination or detection of a specific tissue or condition. According to one embodiment, one or more pulses may include visible light for generating RGB or black-and-white images, while one or more additional pulses are emitted to sense a spectral response to electromagnetic radiation at hyperspectral wavelengths. For example, pulse 2016 may include red light, pulse 2018 may include blue light, and pulse 2020 may include green light, while the remaining pulses 2022 to 2026 may include wavelengths and spectra for detecting specific tissue types, fluorescing reagents, and / or mapping scene morphology. Again, pulses in a single readout cycle may include spectra generated by multiple different emitters (e.g., different segments of the electromagnetic spectrum) that can be used to detect a specific tissue type. For example, if a combination of wavelengths results in a pixel having a value above or below a threshold, that pixel may be classified as corresponding to a specific type of tissue. Each frame can be used to further narrow down the type of tissue present at that pixel (e.g., and every pixel in the image) to provide a very specific classification of the tissue and / or the state (disease / health) of the tissue based on the spectral response of the tissue and / or the presence of fluorescent reagents in the tissue.

[0236] Multiple frames from 2004 to 2014 are shown as readout periods of varying lengths and pulses of varying lengths or intensities. The blanking period, pulse length, or intensity, etc., can be selected based on the monochromatic sensor's sensitivity to a specific wavelength, the transmitter's power output capability, and / or the waveguide's carrying capacity.

[0237] In one implementation, dual image sensors can be used to acquire three-dimensional images or video feeds. Three-dimensional inspection allows for a better understanding of the three-dimensional structure of the inspected area and the mapping of different tissue or material types within that area.

[0238] In one exemplary embodiment, a fluorescent reagent is provided to a patient, and the fluorescent reagent is configured to adhere to cancer cells. The fluorescent reagent is known to fluoresce when irradiated by a specific zone of electromagnetic radiation. The relaxation wavelength of the fluorescent reagent is also known. In this exemplary embodiment, the patient is imaged using an endoscopic imaging system as described herein. The endoscopic imaging system pulses zones of light with red, green, and blue wavelengths to generate an RGB video stream of the patient's interior. Additionally, the endoscopic imaging system pulses electromagnetic radiation at the excitation wavelength of the fluorescent reagent applied to the patient. In this example, the patient has cancer cells, and the fluorescent reagent has adhered to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength of the fluorescent reagent, the fluorescent reagent fluoresces and emits a relaxation wavelength. If cancer cells are present in the scene imaged by the endoscopic imaging system, the fluorescent reagent will also be present in the scene and, after fluorescing, emit its relaxation wavelength due to the emission of the excitation wavelength. The endoscopic imaging system senses the relaxation wavelength of the fluorescent reagent, thereby sensing the presence of the fluorescent reagent in the scene. Because the fluorescent reagent is known to adhere to cancer cells, the presence of the fluorescent reagent further indicates the presence of cancer cells in the scene. Endoscopic imaging systems can thus identify the location of cancer cells within a scene. These systems can also emit laser mapping pulses to generate the scene's topology and calculate the dimensions of objects within it. The location of cancer cells (as identified by fluorescence imaging data) can be combined with the topological and dimensional information calculated based on the laser mapping data. Therefore, the precise location, size, dimensions, and topological structure of cancer cells can be identified. This information can be provided to physicians to aid in cancer cell removal. Additionally, this information can be provided to robotic surgical systems to enable them to remove cancer cells.

[0239] In another exemplary implementation, an endoscopic imaging system is used to image a patient to identify quantitative diagnostic information about lesions in the patient's tissues. In this example, the patient is suspected or known to have a disease whose progression in the patient's tissues can be observed using hyperspectral imaging tracking. The endoscopic imaging system pulses partitions of red, green, and blue wavelengths of light to generate an RGB video stream inside the patient's body. Additionally, the endoscopic imaging system pulses one or more hyperspectral wavelengths of light, allowing the system to "see through" some tissue and generate images of the tissue affected by the disease. The endoscopic imaging system senses reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue, thereby identifying the location of the diseased tissue within the patient's body. The endoscopic imaging system may also emit a laser mapping pulse scheme to generate the topology of the scene and calculate the dimensions of objects within the scene. The location of the diseased tissue (as identified by the hyperspectral imaging data) can be combined with the topological and dimensional information calculated using the laser mapping data. Therefore, the precise location, size, dimensions, and topology of the diseased tissue can be identified. This information can be provided to a physician to assist in the removal, imaging, or study of the diseased tissue. Additionally, this information can be provided to a robotic surgical system so that the system can remove diseased tissue.

[0240] Figure 21 This is a schematic diagram of an imaging system 2100 with a single cutoff filter. System 2100 includes an endoscope 2106 or other suitable imaging device with a light source 2108 for use in low-light environments. Endoscope 2106 includes an image sensor 2104 and a filter 2102 for filtering out unwanted wavelengths of light or other electromagnetic radiation before reaching the image sensor 2104. Light source 2108 transmits light that can illuminate a surface 2112 in low-light environments such as body cavities. Light 2110 is reflected from surface 2112 and passes through filter 2102 before striking image sensor 2104.

[0241] Filter 2102 can be used in embodiments where a fluorescent reagent or dye has been applied. In such embodiments, light source 2108 emits an excitation wavelength to cause the fluorescent reagent or dye to fluoresce. Typically, the relaxation wavelength emitted by the fluorescent reagent or dye will have a different wavelength than the excitation wavelength. Filter 2102 can be selected to filter out the excitation wavelength and allow only the relaxation wavelength to pass through the filter and be sensed by image sensor 2104.

[0242] In one embodiment, filter 2102 is configured to filter out excitation wavelengths of electromagnetic radiation that cause fluorescence in the reagent or dye, such that only the expected relaxation wavelength of the fluorescent reagent or dye is allowed to pass through filter 2102 and reach image sensor 2104. In one embodiment, filter 2102 filters out fluorescent reagent excitation wavelengths at least between 770 nm and 790 nm. In one embodiment, filter 2102 filters out fluorescent reagent excitation wavelengths at least between 795 nm and 815 nm. In one embodiment, filter 2102 filters out fluorescent reagent excitation wavelengths at least between 770 nm and 790 nm and between 795 nm and 815 nm. In these embodiments, filter 2102 filters out the excitation wavelength of the reagent and allows only the relaxation wavelength of the fluorescent reagent to be read by image sensor 2104. Image sensor 2104 may be a wavelength-variable image sensor, and filter 2102 may be configured to allow image sensor 2104 to receive only the relaxation wavelength of the fluorescent reagent and not the emission excitation wavelength of the reagent. The data determined by the image sensor 2104 can then indicate the presence of key body structures, tissues, biological processes, or chemical processes determined by the location of the reagent or dye.

[0243] Filter 2102 can also be used in embodiments where no fluorescent reagent or dye has been applied. Filter 2102 can be selected to allow wavelengths corresponding to the desired spectral response to pass through and be read by image sensor 2104. Image sensor 2104 can be a monochrome image sensor, such that pixels in the captured image that are above or below a threshold can be characterized as corresponding to certain spectral responses or fluorescence emissions. The spectral response or fluorescence emission determined by the pixels captured by image sensor 2104 can indicate the presence of certain body tissues or structures, certain diseases, certain chemical processes, etc.

[0244] Figure 22 This is a schematic diagram of an imaging system 2200 with multiple cutoff filters. System 2200 includes an endoscope 2206 or other suitable imaging device with a light source 2208 for use in low-light environments. Endoscope 2206 includes an image sensor 2204 and two filters 2202a, 2202b. It should be understood that in alternative embodiments, system 2200 may include any number of filters, and the number and type of filters may be selected for specific purposes, such as for acquiring imaging information of specific body tissues, bodily conditions, chemical processes, etc. Filters 2202a, 2202b are configured to prevent the image sensor 2204 from sensing light or other electromagnetic radiation of unwanted wavelengths. Filters 2202a, 2202b may be configured to filter out unwanted wavelengths from white light or other electromagnetic radiation that may be emitted by the light source 2208.

[0245] Compared to Figure 21 The present disclosure also describes that filters 2202a and 2202b can be used in embodiments where a fluorescent reagent or dye has been applied. Filters 2202a and 2202b can be configured to block the emission excitation wavelength of the reagent or dye and allow the image sensor 2204 to read only the relaxation wavelength of the reagent or dye. Furthermore, filters 2202a and 2202b can be used in embodiments where a fluorescent reagent or dye has not been applied. In such embodiments, filters 2202a and 2202b can be selected to allow wavelengths corresponding to the desired spectral response to pass through and be read by the image sensor 2204.

[0246] Multiple filters 2202a, 2202b can each be configured to filter out wavelengths of different ranges in the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than the desired wavelength range, and additional filters can be configured to filter out wavelengths shorter than the desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 2204.

[0247] In one embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 513 nm and 545 nm contacts the image sensor 2204. In another embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 565 nm and 585 nm contacts the image sensor 2204. In yet another embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 900 nm and 1000 nm contacts the image sensor 2204. In one embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 421 nm and 475 nm contacts the image sensor 2204. In yet another embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 520 nm and 545 nm contacts the image sensor 2204. In one embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 617 nm and 645 nm contacts the image sensor 2204. In one embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 760 nm and 795 nm contacts the image sensor 2204. In one embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 795 nm and 815 nm contacts the image sensor 2204. In one embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 370 nm and 420 nm contacts the image sensor 2204. In one embodiment, filters 2202a and 2202b are configured such that electromagnetic radiation between 600 nm and 670 nm contacts the image sensor 2204. In one embodiment, filters 2202a and 2202b are configured to allow only certain fluorescence relaxation emissions to pass through filters 2202a and 2202b and contact image sensor 2204. In one embodiment, a first filter blocks electromagnetic radiation with wavelengths from about 770 nm to about 790 nm, and a second filter blocks electromagnetic radiation with wavelengths from about 795 nm to about 815 nm.

[0248] In one embodiment, system 2200 includes a plurality of image sensors 2204, and may specifically include two image sensors for generating three-dimensional images. Image sensors 2204 may be color / wavelength indeterminate and are configured to read electromagnetic radiation of any wavelength reflected from surface 2212. In one embodiment, image sensors 2204 are each color-dependent or wavelength-dependent and are configured to read electromagnetic radiation of a specific wavelength reflected from surface 2212 and returning to image sensor 2204. Alternatively, image sensors 2204 may include a single image sensor having a plurality of different pixel sensors configured to read light of different wavelengths or colors, such as a Bayer color filter array. Alternatively, image sensors 2204 may include one or more color-indeterminate image sensors configured to read light according to pulse scheduling (such as... Figures 5 to 7E Those shown) read electromagnetic radiation of different wavelengths.

[0249] Figure 23A and Figure 23B Perspective and side views of a specific embodiment of an imaging sensor 2300 having multiple pixel arrays for generating three-dimensional images are shown, respectively. The three-dimensional image sensor can be constructed on multiple substrates and may include multiple pixel arrays and other associated circuitry, wherein multiple pixel columns 2302a forming a first pixel array and multiple pixel columns 2302b forming a second pixel array are located on respective substrates 2308a and 2308b, and multiple circuit columns 2306a and 2306b are located on a separate substrate 2304. Electrical connections and communications between pixel columns and associated or corresponding circuit columns are also shown.

[0250] Multiple pixel arrays can simultaneously sense information, and information from the multiple pixel arrays can be combined to generate a three-dimensional image. In one embodiment, the endoscopic imaging system includes two or more pixel arrays that can be deployed to generate a three-dimensional image. The endoscopic imaging system may include a transmitter for emitting electromagnetic radiation pulses during a blanking period of the pixel arrays. The pixel arrays can be synchronized such that optical black pixels are read out simultaneously for two or more pixel arrays (i.e., a blanking period occurs). The transmitter can emit electromagnetic radiation pulses to charge each of the two or more pixel arrays. The two or more pixel arrays can simultaneously read out their respective charged pixels such that readout periods of the two or more pixel arrays occur simultaneously or substantially simultaneously. In one embodiment, the endoscopic imaging system includes multiple transmitters, wherein each transmitter is individually synchronized with one or more pixel arrays of the multiple pixel arrays. Information from the multiple pixel arrays can be combined to generate three-dimensional image frames and video streams.

[0251] Figure 24A and Figure 24B Perspective and side views, respectively, are shown of a specific embodiment of a monolithic sensor 2400 according to the teachings and principles of this disclosure, which has multiple pixel arrays for generating three-dimensional images. Such an embodiment may be desired for three-dimensional image capture, wherein two pixel arrays 2402 and 2404 can be offset during use. In another embodiment, the first pixel array 2402 and the second pixel array 2404 may be dedicated to receiving electromagnetic radiation within a predetermined wavelength range, wherein the first pixel array is dedicated to electromagnetic radiation within a different wavelength range than the second pixel array.

[0252] Figure 25A and Figure 25B Perspective and side views of a specific embodiment of an imaging sensor 2500 constructed on multiple substrates are shown. As shown, multiple pixel columns 2504 forming the pixel array are located on a first substrate 2502, and multiple circuit columns 2508 are located on a second substrate 2506. The electrical connections and communications between a pixel column and its associated or corresponding circuit column are also shown. In one embodiment, the image sensor may have a pixel array separate from all or most of the supporting circuitry, and it may otherwise be manufactured such that its pixel array and supporting circuitry are on a single, monolithic substrate / chip. This disclosure may use at least two substrates / chips, which will be stacked together using a three-dimensional stacking technique. The first 2502 of the two substrates / chips may be fabricated using an image CMOS process. The first substrate / chip 2502 may consist solely of a pixel array, or it may consist of a pixel array surrounded by a limited circuitry. The second or subsequent substrate / chip 2506 may be fabricated using any process, and is not necessarily derived from an image CMOS process. The second substrate / chip 2506 can be, but is not limited to, a high-density digital process for integrating various and multiple functions into a very limited space or area on the substrate / chip, a mixed-mode or analog process for integrating, for example, precise analog functions, an RF process for enabling wireless capabilities, or a MEMS (Micro-Electro-Mechanical Systems) process for integrating MEMS devices. The image CMOS substrate / chip 2502 can be stacked with the second or subsequent substrate / chip 2506 using any three-dimensional technology. The second substrate / chip 2506 can support the majority or most of the circuitry that may be additionally implemented as peripheral circuitry in the first image CMOS chip 2502 (if implemented on a monolithic substrate / chip), thus increasing the overall system area while keeping the pixel array size constant and optimized to the maximum extent possible. Electrical connections between the two substrates / chips can be accomplished via interconnects, which can be bonding leads, lugs, and / or TSVs (Through Silicon Vias).

[0253] It should be understood that the teachings and principles of this disclosure can be applied to reusable device platforms, limited-use device platforms, reconfigurable device platforms, or single-use / disposable device platforms without departing from the scope of this disclosure. It should be understood that in a reusable device platform, the end user is responsible for cleaning and sterilizing the device. In a limited-use device platform, the device can be used a predetermined number of times before becoming inoperable. Typically, new devices are sterilized before delivery and should be cleaned and sterilized by the end user before any other use if intended for other purposes. In a reconfigurable device platform, a third party can reprocess (e.g., clean, package, and sterilize) a single-use device for further use at a lower cost than a new unit. In a single-use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0254] Example

[0255] The following embodiments relate to preferred features of other implementations:

[0256] Example 1 is a system. The system includes a transmitter for emitting electromagnetic radiation pulses. The transmitter includes a first transmitter for emitting electromagnetic radiation pulses of a first wavelength. The transmitter includes a second transmitter for emitting electromagnetic radiation pulses of a second wavelength. The system includes an image sensor comprising a pixel array for sensing reflected electromagnetic radiation. The system includes a controller in electronic communication with the transmitter and the image sensor. The system is such that at least a portion of the electromagnetic radiation pulses emitted by the transmitter includes one or more of the following: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0257] Example 2 is a system according to Example 1, wherein: the first transmitter emits an electromagnetic radiation pulse of the first wavelength at a first dichroic mirror, and the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength to multiple optical fibers; the second transmitter emits an electromagnetic radiation pulse of the second wavelength at a second dichroic mirror, and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength to the multiple optical fibers; and the first dichroic mirror is transparent to electromagnetic radiation of the second wavelength.

[0258] Example 3 is a system according to any one of Examples 1 to 2, wherein the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength into the plurality of optical fibers at an angle offset from the perpendicular to the plurality of optical fibers, and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength into the plurality of optical fibers at an angle offset from the perpendicular to the plurality of optical fibers.

[0259] Example 4 is a system according to any one of Examples 1 to 3, wherein: the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength into the plurality of optical fibers at an angle substantially perpendicular to the first transmitter; and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength into the plurality of optical fibers through the first dichroic mirror at an angle substantially perpendicular to the second transmitter.

[0260] Example 5 is a system according to any one of Examples 1 to 4, wherein: the transmitter further includes a third transmitter for emitting an electromagnetic radiation pulse of a third wavelength at a third dichroic mirror, the third dichroic mirror reflecting the electromagnetic radiation pulse of the third wavelength to the plurality of optical fibers; the third dichroic mirror reflecting the electromagnetic radiation pulse of the third wavelength through the first dichroic mirror into the plurality of optical fibers; the first dichroic mirror and the second dichroic mirror are transparent to the electromagnetic radiation of the third wavelength; the third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength at an angle substantially perpendicular to the third transmitter into the plurality of optical fibers; and the third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength at an angle offset from the vertical into the plurality of optical fibers.

[0261] Example 6 is a system according to any one of Examples 1 to 5, further comprising: an optical fiber bundle, wherein the transmitter transmits the electromagnetic radiation pulse into the optical fiber bundle; wherein the optical fiber bundle comprises plastic fibers and glass fibers, wherein the plastic fibers and the glass fibers are coupled near the output portion of the optical fiber bundle.

[0262] Example 7 is a system according to any one of Examples 1 to 6, further comprising an intermediary optical component, wherein the electromagnetic radiation pulse passes through the intermediary optical component before entering the fiber bundle.

[0263] Example 8 is a system according to any one of Examples 1 to 7, wherein the intervening optical component includes one or more of a diffuser or a mixing rod.

[0264] Example 9 is a system according to any one of Examples 1 to 8, further comprising: an optical fiber bundle including a plurality of plastic optical fibers, wherein the transmitter transmits the electromagnetic radiation pulse into the optical fiber bundle; and an intermediary optical component, wherein the electromagnetic radiation pulse passes through the intermediary optical component before entering the optical fiber bundle; wherein the intermediary optical component includes a plurality of glass fibers.

[0265] Example 10 is a system according to any one of Examples 1 to 9, further comprising: an optical fiber bundle, wherein the transmitter emits the electromagnetic radiation pulse into the optical fiber bundle; and a diffuser disposed at the distal end of the optical fiber bundle; wherein the diffuser provides an optical cone having an angle between 110 degrees and 120 degrees or an angle between 70 degrees and 80 degrees.

[0266] Example 11 is a system according to any one of Examples 1 to 10, further comprising a third transmitter for emitting electromagnetic radiation pulses of a third wavelength and a fourth transmitter for emitting electromagnetic radiation pulses of a fourth wavelength, wherein: the electromagnetic radiation of the first wavelength emitted by the first transmitter is red light; the electromagnetic radiation of the second wavelength emitted by the second transmitter is blue light; the electromagnetic radiation of the third wavelength emitted by the third transmitter is green light; and the electromagnetic radiation of the fourth wavelength emitted by the fourth transmitter is a hyperspectral wavelength, the hyperspectral wavelength including one or more of the following: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, electromagnetic radiation having a wavelength of about 565 nm to about 585 nm, or electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; wherein the fourth transmitter for emitting the hyperspectral wavelength includes one or more independent lasers for emitting electromagnetic radiation of different hyperspectral wavelengths.

[0267] Example 12 is a system according to any one of Examples 1 to 11, wherein the pixel array of the image sensor senses reflected electromagnetic radiation during the readout period of the pixel array to generate the plurality of exposure frames, wherein the readout period includes the time period during which effective pixels in the pixel array are read.

[0268] Example 13 is a system according to any one of Examples 1 to 12, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a hyperspectral wavelength for inducing a spectral response, wherein the hyperspectral wavelength includes one or more of the following: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or electromagnetic radiation having a wavelength of about 565 nm to about 585 nm, and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0269] Example 14 is a system according to any one of Examples 1 to 13, wherein the transmitter is configured to emit a plurality of electromagnetic radiation sub-pulses during the pulse duration, the plurality of sub-pulses having a sub-duration shorter than the pulse duration.

[0270] Example 15 is a system according to any one of Examples 1 to 14, wherein one or more electromagnetic radiation pulses emitted by the transmitter include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

[0271] Example 16 is a system according to any one of Examples 1 to 15, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a hyperspectral emission that causes the image sensor to generate a hyperspectral exposure frame, and wherein the controller is configured to provide the hyperspectral exposure frame to a corresponding hyperspectral system that determines the location of a key tissue structure within the scene based on the hyperspectral exposure frame.

[0272] Example 17 is a system according to any one of Examples 1 to 16, wherein the hyperspectral emission includes: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0273] Example 18 is a system according to any one of Examples 1 to 17, wherein the controller is further configured to: receive the location of the key tissue structure from the corresponding hyperspectral system; generate an overlapping frame including the location of the key tissue structure; and combine the overlapping frame with a color image frame depicting the scene to indicate the location of the key tissue structure within the scene.

[0274] Example 19 is a system according to any one of Examples 1 to 18, wherein sensing the electromagnetic radiation reflected by the pixel array includes generating a laser mapping exposure frame by sensing the reflected electromagnetic radiation generated by the transmitter pulses the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system, the corresponding laser mapping system determining the topography of the scene and / or the size of one or more objects within the scene; provide the location of the key tissue structure to the corresponding laser mapping system; and receive the topology and / or size of the key tissue structure from the corresponding laser mapping system.

[0275] Example 20 is a system according to any one of Examples 1 to 19, wherein the key structure includes one or more of a nerve, ureter, blood vessel, artery, blood flow or tumor.

[0276] Example 21 is a system according to any one of Examples 1 to 20, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a fluorescence excitation wavelength that causes the image sensor to generate a fluorescence exposure frame, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding fluorescence system, the corresponding fluorescence system determining the location of a key tissue structure within the scene based on the fluorescence exposure frame.

[0277] Example 22 is a system according to any one of Examples 1 to 21, wherein the fluorescence excitation emission includes one or more of the following: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or the electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0278] Example 23 is a system according to any one of Examples 1 to 22, wherein the controller is further configured to: receive the location of the key tissue structure from the corresponding fluorescence system; generate an overlapping frame including the location of the key tissue structure; and combine the overlapping frame with a color image frame depicting the scene to indicate the location of the key tissue structure within the scene.

[0279] Example 24 is a system according to any one of Examples 1 to 23, wherein sensing the electromagnetic radiation reflected by the pixel array includes generating a laser mapping exposure frame by sensing the reflected electromagnetic radiation generated by the transmitter pulses the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system, the corresponding laser mapping system determining the topography of the scene and / or the size of one or more objects within the scene; provide the location of the key tissue structure to the corresponding laser mapping system; and receive the topology and / or size of the key tissue structure from the corresponding laser mapping system.

[0280] Example 25 is a system according to any one of Examples 1 to 24, wherein the key structure includes one or more of a nerve, ureter, blood vessel, artery, blood flow or tumor.

[0281] Example 26 is a system according to any one of Examples 1 to 25, wherein the controller is configured to synchronize the timing of the electromagnetic radiation pulses during the blanking period of the image sensor, wherein the blanking period corresponds to the time between the readout of the last row of active pixels in the pixel array and the start of the next subsequent readout of the active pixels in the pixel array.

[0282] Example 27 is a system according to any one of Examples 1 to 26, wherein two or more electromagnetic radiation pulses emitted by the transmitter generate two or more instances of reflected electromagnetic radiation, the two or more instances of reflected electromagnetic radiation are sensed by the pixel array to generate two or more exposure frames, the two or more exposure frames being combined to form an image frame.

[0283] Example 28 is a system according to any one of Examples 1 to 27, wherein the image sensor includes a first image sensor and a second image sensor, such that the image sensor is capable of generating a three-dimensional image.

[0284] Example 29 is a system according to any one of Examples 1 to 28, wherein the transmitter is configured to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each exposure frame corresponds to an electromagnetic radiation pulse.

[0285] Example 30 is a system according to any one of Examples 1 to 29, wherein the electromagnetic radiation pulse is emitted in a pattern of electromagnetic radiation of different wavelengths, and wherein the transmitter repeats the pattern of electromagnetic radiation of different wavelengths.

[0286] Example 31 is a system according to any one of Examples 1 to 30, wherein at least a portion of the electromagnetic radiation pulse includes a red light wavelength, a green light wavelength, a blue light wavelength, and a hyperspectral wavelength, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red light wavelength, the green light wavelength, the blue light wavelength, and the hyperspectral wavelength can be processed to generate a red-green-blue (RGB) image frame including superimposed hyperspectral imaging data, wherein the hyperspectral wavelength of the electromagnetic radiation includes: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0287] Example 32 is a system according to any one of Examples 1 to 31, wherein at least a portion of the electromagnetic radiation pulse includes luminance emission, red hue emission, blue hue emission, and hyperspectral emission, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the luminance emission, the red hue emission, the blue hue emission, and the hyperspectral emission can be processed to generate a YCbCr image frame including superimposed hyperspectral imaging data, wherein the hyperspectral emission of the electromagnetic radiation includes: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0288] Example 33 is a system according to any one of Examples 1 to 32, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a fluorescence excitation wavelength for causing the reagent to fluoresce, wherein the fluorescence excitation wavelength includes one or more of the following: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0289] Example 34 is a system according to any one of Examples 1 to 33, wherein at least a portion of the electromagnetic radiation pulse includes a red light wavelength, a green light wavelength, a blue light wavelength, and a fluorescence excitation wavelength, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red light wavelength, the green light wavelength, the blue light wavelength, and the fluorescence excitation wavelength can be processed to generate a red-green-blue (RGB) image frame comprising a stack of fluorescence imaging data, wherein the electromagnetic radiation of the fluorescence wavelength includes electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0290] Example 35 is a system according to any one of Examples 1 to 34, wherein at least a portion of the electromagnetic radiation pulse includes luminance emission, red hue emission, blue hue emission, and fluorescence excitation emission, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the luminance emission, the red hue emission, the blue hue emission, and the fluorescence excitation emission can be processed to generate a stacked YCbCr image frame including fluorescence imaging data, wherein the electromagnetic radiation of the fluorescence wavelength includes: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0291] Example 36 is a system according to any one of Examples 1 to 35, further comprising a single optical fiber, wherein the transmitter transmits the electromagnetic radiation pulse into the single optical fiber.

[0292] Example 37 is a system according to any one of Examples 1 to 36, wherein the pixel array is a two-dimensional array of independent pixels, each of which is capable of detecting electromagnetic radiation of any wavelength.

[0293] Example 38 is a system according to any one of Examples 1 to 37, further comprising a filter for filtering electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0294] Example 39 is a system according to any one of Examples 1 to 38, further comprising a filter for filtering electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0295] Example 40 is a system according to any one of Examples 1 to 39, wherein the electromagnetic sensor is a photodiode.

[0296] Example 41 is a system according to any one of Examples 1 to 40, wherein sensing reflected electromagnetic radiation by the pixel array includes: generating a laser mapping exposure frame by sensing reflected electromagnetic radiation generated by the transmitter pulses the laser mapping pattern, wherein the laser mapping exposure frame includes information for determining real-time measurements, the information including one or more of the following: the distance from the endoscope to the object; the angle between the endoscope and the object; or surface topology information about the object.

[0297] Example 42 is a system according to any one of Examples 1 to 41, wherein the laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 cm.

[0298] Example 43 is a system according to any one of Examples 1 to 42, wherein the laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

[0299] Example 44 is a system according to any one of Examples 1 to 43, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter includes multiple tool-specific laser mapping patterns for each of a plurality of tools in the scene.

[0300] Example 45 is a system according to any one of Examples 1 to 44, wherein the laser mapping pattern emitted by the transmitter includes a first output and a second output that are independent of each other, wherein the first output is used for illumination and the second output is used for tool tracking.

[0301] Example 46 is a system according to any one of Examples 1 to 45, wherein the first transmitter is a first laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the first wavelength, and the second transmitter is a second laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the second wavelength.

[0302] Example 47 is a system according to any one of Examples 1 to 46, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to one or more electromagnetic radiation pulses emitted by the transmitter.

[0303] Example 48 is a system according to any one of Examples 1 to 47, wherein the fiber bundle comprises 2 to 150 fibers.

[0304] Example 49 is a system according to any one of Examples 1 to 48, wherein the third transmitter is a third laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the third wavelength.

[0305] Example 50 is a system according to any one of Examples 1 to 49, wherein the fourth transmitter is a fourth laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the fourth wavelength.

[0306] Example 51 is a system according to any one of Examples 1 to 50, wherein the transmitter includes one or more hyperspectral transmitters for emitting electromagnetic radiation pulses of hyperspectral wavelengths to elicit a spectral response.

[0307] Example 52 is a system according to any one of Examples 1 to 51, wherein each of the one or more hyperspectral emitters includes a laser beam, and the laser beam includes a plurality of lasers.

[0308] Example 53 is a system according to any one of Examples 1 to 52, wherein the transmitter further includes an optical element for mixing electromagnetic radiation pulses before entering the fiber bundle, wherein the optical element includes one or more of a diffuser, a mixing rod, or a lens.

[0309] Example 54 is a system according to any one of Examples 1 to 53, further comprising a dichroic mirror for reflecting electromagnetic radiation of blue light wavelength.

[0310] Example 55 is a system according to any one of Examples 1 to 54, further comprising a dichroic mirror for reflecting electromagnetic radiation of green wavelength.

[0311] Example 56 is a system according to any one of Examples 1 to 55, further comprising a dichroic mirror for reflecting electromagnetic radiation of red wavelength.

[0312] Example 57 is a system according to any one of Examples 1 to 56, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 513 nm to about 545 nm.

[0313] Example 58 is a system according to any one of Examples 1 to 57, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 565 nm to about 585 nm.

[0314] Example 59 is a system according to any one of Examples 1 to 58, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0315] Example 60 is a system according to any one of Examples 1 to 59, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0316] Example 61 is a system according to any one of Examples 1 to 60, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0317] Example 62 is a system according to any one of Examples 1 to 61, further comprising a dichroic mirror for reflecting electromagnetic radiation of a certain wavelength band, wherein the dichroic mirror is transparent to electromagnetic radiation of other wavelengths.

[0318] Example 63 is a system according to any one of Examples 1 to 62, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a red light wavelength at least.

[0319] Example 64 is a system according to any one of Examples 1 to 63, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a green light wavelength.

[0320] Example 65 is a system according to any one of Examples 1 to 64, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a blue light wavelength at least.

[0321] Example 66 is a system according to any one of Examples 1 to 65, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 513 nm to about 545 nm.

[0322] Example 67 is a system according to any one of Examples 1 to 66, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 565 nm to about 585 nm.

[0323] Example 68 is a system according to any one of Examples 1 to 67, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0324] Example 69 is a system according to any one of Examples 1 to 68, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0325] Example 70 is a system according to any one of Examples 1 to 69, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0326] Example 71 is a system according to any one of Examples 1 to 70, wherein the emitter includes a plurality of laser emitters, and wherein the plurality of laser emitters have a Gaussian cross-sectional intensity profile.

[0327] Example 72 is a system according to any one of Examples 1 to 71, wherein the emitter includes a plurality of laser emitters, and wherein the plurality of laser emitters have a flat or substantially flat intensity profile.

[0328] Example 73 is a system according to any one of Examples 1 to 72, wherein the emitter includes a plurality of laser emitters, and wherein the plurality of laser emitters have a top-hat shaped intensity profile.

[0329] Example 74 is a system according to any one of Examples 1 to 73, wherein the first transmitter and the second transmitter are designed to collect an area such that the electromagnetic radiation pulse of the first wavelength and the electromagnetic radiation pulse of the second wavelength are mixed in the collection area and received by the fiber bundle.

[0330] Example 75 is a system according to any one of Examples 1 to 74, further comprising an intermediary optical element disposed between the transmitter and the fiber bundle, wherein the intermediary optical element is configured to mix the emissions from the first transmitter and the second transmitter before the emissions reach the fiber bundle.

[0331] Example 76 is a system according to any one of Examples 1 to 75, further comprising an intermediary optical element disposed between the transmitter and the fiber bundle, the intermediary optical element being configured to uniformly mix the independent emission of electromagnetic radiation from the first transmitter and the second transmitter before reaching the fiber bundle.

[0332] It should be understood that the various features disclosed herein offer significant advantages and advancements in the art. The following claims are examples of some of those features.

[0333] In the specific embodiments described above, for the purpose of simplification, the various features of this disclosure are concentrated in a single embodiment. The method of this disclosure should not be construed as implying an intention that the disclosure protected by the claims requires more features than expressly listed in each claim. Rather, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0334] It should be understood that any feature of the above-described arrangement, embodiments, and implementations may be combined in a single implementation that includes a combination of features obtained from any of the disclosed arrangement, embodiments, and implementations.

[0335] It should be understood that the above-described configuration is merely an exemplary application of the principles of this disclosure. Many modifications and alternative configurations can be devised by those skilled in the art without departing from the spirit and scope of this disclosure, and the appended claims are intended to cover such modifications and configurations.

[0336] Therefore, when this disclosure is illustrated and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications can be made without departing from the principles and ideas set forth herein, including but not limited to changes in size, material, shape, form, function and operation, assembly and use.

[0337] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to execute one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. Those skilled in the art will understand that components may have different names. This document is not intended to distinguish between components that are different in name rather than function.

[0338] For illustrative and descriptive purposes, the specific embodiments described above have been provided. These specific embodiments are not intended to be exhaustive or to limit this disclosure to the specific forms disclosed. Many modifications and changes can be made to this disclosure based on the foregoing teachings. Furthermore, it should be noted that any or all of the foregoing alternative embodiments can be used in any desired combination to form further hybrid embodiments of this disclosure.

[0339] Furthermore, while specific embodiments of this disclosure have been described and illustrated, this disclosure is not limited to the particular form or arrangement of components as described and illustrated. The scope of this disclosure will be defined by the appended claims, any future claims filed herein and in different applications, and their equivalents.

Claims

1. A system comprising: A transmitter for emitting electromagnetic radiation pulses, wherein the transmitter includes a first transmitter for emitting electromagnetic radiation pulses of a first wavelength and a second transmitter for emitting electromagnetic radiation pulses of a second wavelength; An image sensor, the image sensor comprising an array of pixels for sensing reflected electromagnetic radiation; as well as The controller is in electronic communication with the transmitter and the image sensor; The image sensor is configured to generate a plurality of exposure frames, each of the plurality of exposure frames corresponding to one or more electromagnetic radiation pulses emitted by the transmitter; When the effective pixels of the pixel array are not read, the transmitter actively pulses electromagnetic radiation during the blanking period of the image sensor; and At least a portion of the electromagnetic radiation pulse emitted by the transmitter includes a fluorescence excitation pulse; At least a portion of the multiple exposure frames include a fluorescence exposure frame corresponding to a fluorescence excitation pulse; The controller instructs the transmitter to change the duration and / or magnitude of the fluorescence excitation pulses at a constant display frame rate to change the amount of electromagnetic radiation received by the pixel array, wherein the fluorescence excitation zone is able to pulse at a rate different from that of other zones.

2. The system according to claim 1, wherein: The first transmitter emits the electromagnetic radiation pulse of the first wavelength at the first dichroic mirror, and the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength to multiple optical fibers. The second transmitter emits the electromagnetic radiation pulse of the second wavelength at the second dichroic mirror, and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength back to the plurality of optical fibers; and The first dichroic mirror is transparent to electromagnetic radiation of the second wavelength.

3. The system of claim 2, wherein the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength into the plurality of optical fibers at an angle offset from the perpendicular to the plurality of optical fibers, and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength into the plurality of optical fibers at an angle offset from the perpendicular to the plurality of optical fibers.

4. The system according to claim 2, wherein: The first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength into the plurality of optical fibers at an angle substantially perpendicular to the first transmitter; and The second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength through the first dichroic mirror at an angle substantially perpendicular to the second transmitter into the plurality of optical fibers.

5. The system according to claim 2, wherein: The transmitter also includes a third transmitter for emitting an electromagnetic radiation pulse of a third wavelength at a third dichroic mirror, the third dichroic mirror reflecting the electromagnetic radiation pulse of the third wavelength to the plurality of optical fibers; The third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength through the first dichroic mirror into the multiple optical fibers; The first and second dichroic mirrors are transparent to electromagnetic radiation of the third wavelength; The third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength at an angle substantially perpendicular to the third transmitter into the plurality of optical fibers; and The third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength into the multiple optical fibers at an angle offset from the vertical.

6. The system according to claim 1, further comprising: An optical fiber bundle, wherein the transmitter emits the electromagnetic radiation pulse into the optical fiber bundle; The optical fiber bundle includes plastic fibers and glass fibers, wherein the plastic fibers and the glass fibers are coupled near the output portion of the optical fiber bundle.

7. The system of claim 6 further includes an intermediary optics component, wherein the electromagnetic radiation pulse passes through the intermediary optics component before entering the fiber bundle.

8. The system of claim 7, wherein the intervening optical component comprises one or more of a diffuser or a mixing rod.

9. The system according to claim 1, further comprising: An optical fiber bundle comprising a plurality of plastic optical fibers, wherein the transmitter transmits the electromagnetic radiation pulse into the optical fiber bundle; as well as Intermediate optical component, wherein the electromagnetic radiation pulse passes through the intermediate optical component before entering the fiber bundle; The intervening optical component comprises multiple glass fibers.

10. The system according to claim 1, further comprising: An optical fiber bundle, wherein the transmitter emits the electromagnetic radiation pulse into the optical fiber bundle; as well as A diffuser, wherein the diffuser is disposed at the distal end of the fiber bundle; The diffuser provides a light cone having an angle between 110 and 120 degrees or between 70 and 80 degrees.

11. The system of claim 1, further comprising a third transmitter for emitting electromagnetic radiation pulses of a third wavelength and a fourth transmitter for emitting electromagnetic radiation pulses of a fourth wavelength, wherein: The electromagnetic radiation of the first wavelength emitted by the first transmitter is red light; The electromagnetic radiation of the second wavelength emitted by the second transmitter is blue light; The electromagnetic radiation of the third wavelength emitted by the third transmitter is green light; and The electromagnetic radiation of the fourth wavelength emitted by the fourth emitter is a fluorescence excitation wavelength including one or more of the following: electromagnetic radiation having a wavelength of 770 nm to 790 nm, or electromagnetic radiation having a wavelength of 795 nm to 815 nm. The fourth emitter, used to emit the fluorescence excitation wavelength, includes one or more independent lasers for emitting electromagnetic radiation at different fluorescence excitation wavelengths.

12. The system of claim 1, wherein the first transmitter is a first laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the first wavelength, and the second transmitter is a second laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the second wavelength.

13. The system of claim 1, wherein the pixel array of the image sensor senses reflected electromagnetic radiation during a readout period of the pixel array to generate the plurality of exposure frames, wherein the readout period includes a time period during which valid pixels in the pixel array are read out.

14. The system of claim 1, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a fluorescence excitation wavelength for fluorescing a reagent, wherein the fluorescence excitation wavelength includes one or more of the following: The electromagnetic radiation having a wavelength of 770 nm to 790 nm; or The electromagnetic radiation having a wavelength of 795 nm to 815 nm.

15. The system of claim 1, wherein the transmitter is configured to emit a plurality of electromagnetic radiation sub-pulses during the pulse duration, the sub-pulses having a sub-duration shorter than the pulse duration.

16. The system of claim 1, wherein one or more of the electromagnetic radiation pulses emitted by the transmitter comprise electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

17. The system of claim 1, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a fluorescence excitation emission that causes the image sensor to generate a fluorescence exposure frame, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding system that determines the location of a key tissue structure within the scene based on the fluorescence exposure frame.

18. The system of claim 17, wherein the fluorescence excitation emission comprises each of the following: The electromagnetic radiation having wavelengths from 770 nm to 790 nm; and The electromagnetic radiation having a wavelength of 795 nm to 815 nm.

19. The system of claim 18, wherein the controller is further configured to: Receive the location of the key organizational structure from the corresponding system; Generate an overlapping frame including the location of the key organizational structure; and The overlapping frame is combined with a color image frame depicting the scene to indicate the location of the key organizational structure within the scene.

20. The system of claim 19, wherein the critical tissue structure comprises one or more of a nerve, ureter, blood vessel, artery, blood flow, or tumor.

21. The system of claim 1, wherein the controller is configured to synchronize the timing of the electromagnetic radiation pulses during a blanking period of the image sensor, wherein the blanking period corresponds to the time between the readout of the last row of active pixels in the pixel array and the start of the next subsequent readout of the active pixels in the pixel array.

22. The system of claim 1, wherein two or more pulses of electromagnetic radiation emitted by the transmitter generate two or more instances of reflected electromagnetic radiation, the two or more instances being sensed by the pixel array to generate two or more exposure frames, the two or more exposure frames being combined to form an image frame.

23. The system of claim 1, wherein the transmitter is configured to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each exposure frame corresponds to one or more electromagnetic radiation pulses.

24. The system of claim 1, wherein the electromagnetic radiation pulse is emitted in a pattern of electromagnetic radiation of different wavelengths, and wherein the transmitter repeats the pattern of electromagnetic radiation of different wavelengths.

25. The system of claim 1 further includes a filter that filters electromagnetic radiation having wavelengths of 770 nm to 790 nm.

26. The system of claim 1 further includes a filter that filters electromagnetic radiation having wavelengths from 795 nm to 815 nm.

27. The system of claim 1, wherein at least a portion of the electromagnetic radiation pulse comprises a red light wavelength, a green light wavelength, a blue light wavelength, and a fluorescence excitation wavelength, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red light wavelength, the green light wavelength, the blue light wavelength, and the fluorescence excitation wavelength can be processed to generate a red-green-blue (RGB) image frame comprising a stack of fluorescence imaging data, wherein the electromagnetic radiation at the fluorescence excitation wavelength comprises one or more of the following: The electromagnetic radiation having a wavelength of 770 nm to 790 nm; or The electromagnetic radiation having a wavelength of 795 nm to 815 nm.

28. The system of claim 1, wherein at least a portion of the electromagnetic radiation pulse comprises luminance emission, red hue emission, blue hue emission, and fluorescence excitation emission, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the luminance emission, the red hue emission, the blue hue emission, and the fluorescence excitation emission can be processed to generate a YCbCr image frame comprising a stack of fluorescence imaging data, wherein the electromagnetic radiation of the fluorescence excitation emission comprises one or more of the following: The electromagnetic radiation having a wavelength of 770 nm to 790 nm; or The electromagnetic radiation having a wavelength of 795 nm to 815 nm.

Images