Hyperspectral imaging in light-poor environments

Abstract

The present invention discloses a system, method and apparatus for performing hyperspectral imaging in a light-deficient environment. The system includes a transmitter and an image sensor, the transmitter being used to transmit electromagnetic radiation pulses, and the image sensor comprising a pixel array for sensing reflected electromagnetic radiation. The system includes a controller, the controller comprising a processor in electrical communication with the image sensor and the transmitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor. The system causes at least a portion of the electromagnetic radiation pulses emitted by the transmitter to include one or more of the following: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm, electromagnetic radiation having a wavelength from about 565 nm to about 585 nm, or electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm.

Description

Technical Field

[0001] This patent application relates to digital imaging, and in particular to hyperspectral imaging in light-deficient environments. Background Art

[0002] Advances in technology have provided advances in medical imaging capabilities. Endoscopes can be used to view the inside of the body and examine the inside of an organ or cavity of the body. Endoscopes are used to investigate a patient's symptoms, confirm a diagnosis, or provide medical treatment. Medical endoscopes can be used to view a variety of body systems and parts, such as the gastrointestinal tract, respiratory tract, urinary tract, abdominal cavity, etc. Endoscopes can also be used for surgical procedures, such as plastic surgery, surgery on a joint or bone, surgery on the nervous system, surgery within the abdominal cavity, etc.

[0003] In some cases of endoscopic imaging, it may be beneficial or necessary to observe the color space. A digital color image includes at least three layers or "color channels" that accumulate to form an image with a range of tones. Each of the color channels measures the intensity and chromaticity of light of a spectral band. Typically, a digital color image includes color channels for a red spectral band, a green spectral band, and a blue spectral band (this may be referred to as a red-green-blue or RGB image). Each of the red, green, and blue color channels includes brightness information for a red, green, or blue spectral band. The brightness information of the individual red, green, and blue layers is combined to generate a color image. Because the color image is composed of individual layers, a conventional digital camera image sensor includes a color filter array that allows red, green, and blue visible light wavelengths to hit the selected pixel sensor. Each individual pixel sensor element is sensitive to a red, green, or blue wavelength and will only return image data for that wavelength. Image data from the total array of pixel sensors is combined to generate an RGB image. The at least three different types of pixel sensors consume a large amount of physical space, so that the complete pixel array cannot fit in the small distal end of the endoscope.

[0004] Because conventional image sensors cannot fit in the distal end of an endoscope, the image sensor is traditionally located in the handpiece unit of the endoscope, which is held by the endoscope operator and is not placed in the body cavity. In such endoscopes, light is transmitted from the handpiece unit along the length of the endoscope to the distal end of the endoscope. This configuration has significant limitations. Endoscopes with this configuration are delicate and can be easily misaligned or damaged when they are bumped or impacted during regular use. This can significantly reduce the quality of the image, and the endoscope requires frequent repair or replacement.

[0005] Conventional endoscopes with image sensors placed in the handpiece unit are also limited to capturing only color images. However, in some implementations, it may be desirable to capture images with hyperspectral image data in addition to color image data. Color images reflect what the human eye detects when observing the environment. However, the human eye is limited to observing visible light, and cannot detect other wavelengths of the electromagnetic spectrum. At other wavelengths of the electromagnetic spectrum beyond the "visible light" wavelength, additional information about the environment may be obtained. One way to obtain image data outside the visible light spectrum is to apply hyperspectral imaging.

[0006] Hyperspectral imaging is used to identify different materials or objects, and to identify different processes by providing information beyond what is visible to the human eye. Unlike ordinary camera images that provide limited information to the human eye, hyperspectral imaging can identify specific compounds and biological processes based on their unique spectral signatures. Hyperspectral imaging is complex and requires fast computer processing power, sensitive detectors, and large data storage capacity.

[0007] Hyperspectral imaging traditionally requires a dedicated image sensor that consumes significant physical space and cannot be mounted within the distal end of an endoscope. In addition, if a hyperspectral image is superimposed on a black and white or color image to provide context to the practitioner, the camera (or cameras) capable of generating the stacked image may have many different types of pixel sensors that are sensitive to different ranges of electromagnetic radiation. This would include three separate types of pixel sensors for generating RGB color images, and additional pixel sensors for generating hyperspectral image data at different wavelengths of the electromagnetic spectrum. This would consume significant physical space and require a larger pixel array to ensure satisfactory image resolution. In the case of endoscopic imaging, one or more cameras would be too large to be placed at the distal end of the endoscope, and therefore could not be placed in an endoscope hand unit or robotic unit. This introduces the same disadvantages as described above, and can result in an endoscope that is very delicate, such that image quality is significantly reduced when the endoscope is bumped or impacted during use.

[0008] In view of the above, systems, methods and apparatus for improved endoscopic imaging in light-poor environments are described herein.The systems, methods and apparatus disclosed herein provide an apparatus for color and hyperspectral imaging with an endoscopic apparatus. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A non-limiting and non-exhaustive embodiment of the present disclosure is described with reference to the following drawings, wherein like reference numerals refer to like parts in the various views unless otherwise specified. The advantages of the present disclosure will be better understood with reference to the following description and drawings, wherein:

[0010] Figure 1 is a schematic diagram of a system for digital imaging in a light-deficient environment having paired emitters and a pixel array;

[0011] Figure 2 A system for providing illumination to a light-deficient environment for endoscopic imaging;

[0012] Figure 2A is a schematic diagram of the complementary system hardware;

[0013] FIG. 3A to FIG. 3D is a diagram of the operating cycle of the sensor for constructing one image frame;

[0014] Figure 4A is a graphical representation of the operation of an embodiment of an electromagnetic transmitter;

[0015] Figure 4B is a graphical representation of varying the duration and magnitude of emitted electromagnetic pulses to provide exposure control;

[0016] Figure 5 Yes FIG. 3A to FIG. 4B An illustration of an embodiment of the present disclosure of an operating cycle of a sensor, an electromagnetic transmitter, and a combination of transmitted electromagnetic pulses showing the imaging system during operation;

[0017] Fig. 6A is a schematic diagram of a method for recording a video using full spectrum light during a time period from t(0) to t(1);

[0018] Figure 6B is a schematic diagram of the process of recording a video by pulsing a portion of the spectrum of light during the time period t(0) to t(1);

[0019] 7A to 7E A schematic diagram showing a method for recording video frames of both full spectrum light and partitioned spectrum light within a time interval;

[0020] Figure 8 is a schematic diagram of a process flow for simultaneously adjusting an electromagnetic emitter and an image sensor;

[0021] Fig. 9 is a schematic diagram of a process flow for adjusting the sensitivity of an image sensor;

[0022] Fig.10 is a schematic diagram of a process flow for simultaneously adjusting an image signal processor and a transmitter based on a frame cycle histogram;

[0023] Fig.11 is a schematic diagram of a process flow for limiting adjustments to an image signal processor and / or a transmitter based on a desired output;

[0024] Fig.12 is a schematic diagram of a process flow for increasing the dynamic range of an image by cycling a sensor through a first emission intensity and a second emission intensity and combining data from the first emission intensity and the second emission intensity;

[0025] Fig.13 is a schematic diagram of a process flow for increasing the dynamic range of an image by cycling a sensor through a plurality of luminous intensities and combining data from each of the plurality of luminous intensities;

[0026] Fig.14 is a schematic diagram of a process flow for performing corrections and adjustments on digital image data;

[0027] Fig.15 is a schematic diagram of system hardware for writing, storing, and reading data from a digital video data stream;

[0028] Fig.16 is a schematic diagram illustrating methods and hardware for use with a zoned light system;

[0029] Fig.17 is a schematic diagram of a pattern reconstruction process for generating an RGB image with hyperspectral data superimposed thereon by pulse partitioning the spectrum;

[0030] FIG. 18A to FIG. 18B is a schematic diagram of a timing example for deploying two different pixel sensitivities in a dual-sensitivity pixel array;

[0031] FIG. 19A to FIG. 19C The use of pulsed and / or synchronized white light emission and corresponding color sensors is shown;

[0032] FIG. 20A to FIG. 20C A light source having a plurality of emitters is shown;

[0033] Fig.21 A single optical fiber is shown outputting at the output via a diffuser to illuminate a scene in a light-deficient environment;

[0034] Fig. 22 shows a portion of an electromagnetic spectrum divided into a plurality of different sub-spectra that may be emitted by an emitter of a light source in accordance with the principles and teachings of the present disclosure;

[0035] Fig.23 is a schematic diagram showing the timing of emission and readout for generating an image frame including a plurality of exposure frames produced by different partitions of pulsed light;

[0036] Fig.24A and Fig. 24B Specific implementations with multiple pixel arrays for generating three-dimensional images according to the principles and teachings of the present disclosure are shown;

[0037] Fig.25A and 25B A perspective view and a side view, respectively, of a specific implementation of an imaging sensor constructed on multiple substrates, wherein multiple columns of pixels forming a pixel array are located on a first substrate, and multiple columns of circuits are located on a second substrate, showing electrical connections and communications between a column of pixels and its associated or corresponding column of circuits; and

[0038] Fig.26A and Fig.26B A perspective view and a side view, respectively, of a specific implementation of an imaging sensor having multiple pixel arrays for producing a three-dimensional image are shown, wherein the multiple pixel arrays and the image sensor are constructed on multiple substrates. DETAILED DESCRIPTION

[0039] Disclosed herein are systems, methods, and apparatus for digital imaging that may be primarily applicable to medical applications such as medical endoscopic imaging.An embodiment of the present disclosure is an endoscopic system for hyperspectral and / or color imaging in a light-deficient environment.

[0040] In one embodiment, the system includes an emitter and an image sensor, the emitter for emitting electromagnetic radiation pulses, and the image sensor includes a pixel array for sensing reflected electromagnetic radiation. The system also includes a controller, the controller including a processor in electrical communication with the image sensor and the emitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor. The system is such that at least a portion of the electromagnetic radiation pulses emitted by the emitter include hyperspectral electromagnetic radiation, such as electromagnetic radiation having a wavelength from about 513nm to about 545nm, from about 565nm to about 585nm and / or from about 900nm to about 1000nm. The system can be implemented to generate a color image of a scene in a light-deficient environment, wherein the hyperspectral imaging data is superimposed on the color image. The hyperspectral imaging data can be analyzed by a corresponding system to identify the location and identity of key tissue structures in the body, such as the location and identity of neural tissue, cancerous tissue, blood vessels, arteries, blood flow, ureters, etc.

[0041] Conventional endoscopes are designed so that the image sensor is placed at the proximal end of the device within the handpiece unit. This configuration requires that the incident light travel the length of the endoscope through precisely coupled optical elements. The precise optical elements may not be easy to align during normal use, and this can result in image distortion or image loss. The embodiments of the present disclosure place the image sensor within the distal end of the endoscope itself. This provides greater optical simplicity when compared to specific implementations known in the art. However, an acceptable solution to this approach is non-trivial and introduces its own set of engineering challenges, especially that the image sensor must fit within a highly constrained area. Disclosed herein are systems, methods, and devices for digital imaging in light-starved environments that employ minimal area image sensors and can be configured to enable hyperspectral and color imaging.

[0042] In some cases, it is desirable to generate endoscopic imaging with multiple images of multiple data types or stacked on top of each other. For example, it may be desirable to generate a color ("RGB") image that also includes hyperspectral imaging data stacked on the RGB image. Stacked images of this nature can enable a practitioner or computer program to identify key body structures based on hyperspectral imaging data. Historically, this would require the use of multiple sensor systems that include image sensors for color imaging and one or more additional image sensors for hyperspectral imaging. In such systems, the multiple image sensors would have multiple different types of pixel sensors that are each sensitive to different ranges of electromagnetic radiation. In systems known in the art, this includes three separate types of pixel sensors for generating RGB color images, and additional pixel sensors for generating hyperspectral image data at different wavelengths of the electromagnetic spectrum. These multiple different image sensors consume too much physical space and cannot be located at the distal end of the endoscope. In systems known in the art, one or more cameras are not placed at the distal end of the endoscope, and are instead placed in an endoscope handpiece or a robotic unit. This introduces many disadvantages and causes the endoscope to be very delicate. When an endoscope is bumped or impacted during use, the delicate endoscope may be damaged and the image quality may deteriorate. In view of the above, systems, methods and devices for endoscopic imaging in a light-deficient environment are disclosed herein. The systems, methods and devices disclosed herein provide a device for employing multiple imaging techniques in a single imaging process while allowing one or more image sensors to be disposed in the distal end of the endoscope.

[0043] Hyperspectral imaging

[0044] In one embodiment, the systems, methods, and devices disclosed herein provide a device for generating hyperspectral imaging data in a light-deficient environment. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from conventional cameras that only capture light across three wavelengths (including red, green, and blue wavelengths) based on the visible spectrum that can be distinguished by the human eye to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands.

[0045] Hyperspectral imaging was originally developed for applications in mining and geology. Unlike normal camera images that provide limited information to the human eye, hyperspectral imaging can identify specific minerals based on the spectral signatures of different minerals. Hyperspectral imaging is useful even when captured in aerial images and can provide information about, for example, oil or gas leaks from pipelines or natural wells and their impact on nearby vegetation. This information is gathered based on the spectral signatures of certain materials, objects, or processes that can be identified through hyperspectral imaging. Hyperspectral imaging can also be used in medical imaging applications where certain tissues, chemical processes, biological processes, and diseases can be identified based on unique spectral signatures.

[0046] In one embodiment of hyperspectral imaging, a complete spectrum or some spectral information is collected at each pixel in the image plane. A hyperspectral camera can use special hardware to capture any suitable number of wavelength bands for each pixel, which can be interpreted as a complete spectrum. The goals of hyperspectral imaging vary for different applications. In one application, the goal is to obtain imaging data of the entire electromagnetic spectrum for each pixel in the image scene. In another application, the goal is to obtain imaging data of certain partitions of the electromagnetic spectrum for each pixel in the image scene. Certain partitions of the electromagnetic spectrum can be selected based on what may be identified in the image scene. These applications enable accurate identification of certain materials, tissues, chemical processes, biological processes, and diseases when they may not be identifiable under the visible light wavelength band. In some medical applications, hyperspectral imaging includes one or more specific partitions of the electromagnetic spectrum that have been selected to identify certain tissues, diseases, chemical processes, etc. Some exemplary partitions of the electromagnetic spectrum that can be pulsed for hyperspectral imaging in medical applications include electromagnetic radiation with wavelengths from about 513nm to about 545nm, from about 565nm to about 585nm, and / or from about 900nm to about 1000nm.

[0047] Hyperspectral imaging achieves many advantages over conventional imaging and achieves specific advantages in medical applications. Endoscopic hyperspectral imaging allows physicians or computer-implemented programs to identify nerve tissue, muscle tissue, blood vessels, cancer cells, typical non-cancerous cells, blood flow direction, etc. Hyperspectral imaging enables atypical cancer tissue to be accurately distinguished from typical healthy tissue, and thus enables practitioners or computer-implemented programs to distinguish the boundaries of cancerous tumors during surgery or research imaging. The information obtained by hyperspectral imaging enables accurate identification of certain tissues or conditions that may not be diagnosed or may be less accurate when using conventional imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, enabling practitioners to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells that contrast with typical healthy cells, identify certain tissues or conditions, identify key structures, etc. Hyperspectral imaging provides specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0048] In one embodiment of the present disclosure, the endoscope system illumination source and pulses electromagnetic radiation for spectral or hyperspectral imaging. The pulsed hyperspectral imaging discussed herein includes one or more bands of the pulsed electromagnetic spectrum, and may include infrared wavelengths, visible spectrum, ultraviolet spectrum, x-ray wavelengths, or any suitable combination of various wavelength bands. In one embodiment, hyperspectral imaging includes pulses of electromagnetic radiation having a wavelength from about 513nm to about 545nm, from about 565nm to about 585nm, and / or from about 900nm to about 1000nm.

[0049] Pulse imaging

[0050] Some specific implementations of the present disclosure include various aspects of sensor and system combination designs that allow high-definition images to be generated with a reduced number of pixels in a restricted lighting environment. This is achieved by pulsing a monochromatic wavelength frame by frame, and switching or alternating between a single different color wavelength each frame using a controlled light source in combination with a high frame capture rate and a specially designed corresponding monochromatic sensor. In addition, electromagnetic radiation outside the visible spectrum can be pulsed to enable the generation of hyperspectral images. The pixels can be color-indeterminate so that each pixel generates data for each electromagnetic radiation pulse, including pulses of red visible light wavelengths, green visible light wavelengths, and blue visible light wavelengths, as well as other wavelengths for hyperspectral imaging.

[0051] For the purpose of promoting understanding of the principles according to the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe these embodiments. However, it should be understood that it is not intended to limit the scope of the present disclosure. Any changes and further modifications to the inventive features shown herein, and any additional applications of the principles of the present disclosure as shown herein (which will generally occur for those skilled in the relevant art and those who understand the present disclosure) will be considered to be within the scope of the disclosure protected by the claims.

[0052] Before disclosing and describing structures, systems and methods for generating images in a light-deficient environment, it should be understood that the present disclosure is not limited to the specific structures, configurations, process steps and materials disclosed herein, because such structures, configurations, process steps and materials may vary to a certain extent. It should also be understood that the terminology used herein is only for the purpose of describing specific embodiments and is not intended to be limiting, because the scope of the present invention will be limited only by the appended claims and their equivalents.

[0053] In describing and claiming the presently disclosed subject matter, the following terminology will be used in accordance with the following definitions.

[0054] It must be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0055] As used herein, the terms "comprising," "including," "characterized by," and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unrecited elements or method steps.

[0056] As used herein, the phrase "consisting of and its grammatical equivalents exclude any elements or steps not listed in a claim.

[0057] As used herein, the phrase "consisting essentially of and its grammatical equivalents limit the scope of a claim to the specified materials or steps and those materials or steps that do not materially affect one or more of the basic and novel characteristics of the claimed disclosure.

[0058] As used herein, the term "proximal side" broadly refers to the concept of a portion close to a starting point.

[0059] As used herein, the term "distal" generally refers to the opposite of the proximal side, and thus refers to the concept of a portion farther from a starting point or a portion farthest from a starting point according to the context.

[0060] As used herein, color sensors or multispectral sensors are those sensors known to have a color filter array (CFA) thereon to filter incident electromagnetic radiation into its individual components. In the visible range of the electromagnetic spectrum, such CFAs may be based on a Bayer template or a modified form thereof to separate the green, red and blue spectral components of light.

[0061] As used herein, a monochrome sensor refers to an imaging sensor without filtering. Because the pixels are color amorphous, their effective spatial resolution is significantly higher than their pixel color (usually filtered with a Bayer pattern) counterparts in traditional single-sensor cameras. Monochrome sensors can also have higher quantum efficiency because fewer incident photons are wasted between individual pixels.

[0062] As used herein, a transmitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of the transmitter can be configured to emit pulses and have very specific frequencies or frequency ranges from within the entire electromagnetic spectrum. The pulses may include wavelengths in the visible range and the invisible range. The transmitter can be cycled on and off to generate pulses, or can generate pulses with a shutter mechanism. The transmitter can have a variable power output level, or can be controlled by auxiliary devices such as apertures or filters. The transmitter can emit wide spectrum or full spectrum electromagnetic radiation that can generate pulses by color filtering or shading. The transmitter may include multiple electromagnetic sources that act alone or in concert.

[0063] Now refer to the accompanying drawings, Figure 1 A schematic diagram of a system 100 for sequential pulse imaging in an aphotic environment is shown. The system 100 can be deployed to generate an RGB image, wherein hyperspectral data is superimposed on the RGB image. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a partition of electromagnetic radiation in an aphotic environment 112, and the pixel array 122 senses an instance of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 operate sequentially so that one or more pulses of a partition of electromagnetic radiation produce an exposure frame including image data sensed by the pixel array 122.

[0064] It should be noted that, as used herein, the term "light" is both a particle and a wavelength, and is intended to represent electromagnetic radiation that can be detected by the pixel array 122, and may include wavelengths from the visible and invisible spectrum of electromagnetic radiation. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is less than the entire spectrum, or in other words, the wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that is controllable with respect to a portion of the electromagnetic spectrum emitted, or a light source that can manipulate the physical properties of its components, the emission intensity, or the emission duration, or all of the above. An emitter can emit light in any dithered, diffuse, or collimated emission, and can be controlled digitally or by an analog method or system. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and it includes light sources such as lasers, LEDs, incandescent lights, or any light source that can be digitally controlled.

[0065] The pixel array 122 of the image sensor may be electronically paired with the emitter 102 such that the emitter 102 and the pixel array 122 are synchronized during operation for both receiving emissions and adjustments made within the system. The emitter 102 may be tuned to emit electromagnetic radiation in the form of a laser, which may be pulsed to illuminate the light-deficient environment 112. The emitter 102 may pulse at intervals corresponding to the operation and functionality of the pixel array 122. The emitter 102 may pulse light in multiple electromagnetic partitions such that the pixel array receives electromagnetic energy and generates a data set corresponding in time to each specific electromagnetic partition. For example, Figure 1 A specific implementation is shown in which the emitter 102 emits four different sub-regions of electromagnetic radiation, including red 104 wavelengths, green 106 wavelengths, blue 108 wavelengths, and hyperspectral 110 emissions. The hyperspectral 110 emissions may include wavelength bands in the electromagnetic spectrum that induce a spectral response. The hyperspectral 110 emissions may include multiple individual emissions that are separate and independent from each other.

[0066] The aphotic environment 112 includes structures, tissues, and other elements that reflect a combination of red 114, green 116, and / or blue 118 light. Red 114 tissue is sensitive to the pulsed red 104 light and will be processed by the human eye as having a red 114 hue. The scene within the aphotic environment 112 includes elements that are sensitive to hyperspectral 120 radiation.

[0067] The emitter 102 may be a laser emitter capable of emitting pulsed red 104 light for generating sensed red 105 data to identify red 114 elements within a light-deficient environment 112. The emitter 102 may also be capable of emitting pulsed green 106 light for generating sensed green 107 data to identify green 116 elements within a light-deficient environment. The emitter 102 may also be capable of emitting pulsed blue 108 light for generating sensed blue 109 data to identify blue 118 elements within a light-deficient environment. The emitter 102 may also be capable of emitting hyperspectral 110 emissions to identify elements sensitive to hyperspectral 120 radiation. The emitter 102 may be capable of emitting pulsed red 104, pulsed green 106, pulsed blue 108, and pulsed hyperspectral 110 emissions in any desired order.

[0068] Pixel array 122 senses reflected electromagnetic radiation. Each of sensing red 105, sensing green 107, sensing blue 109, and sensing hyperspectral 111 data may be referred to as an "exposure frame." Sensing hyperspectral 111 may result in multiple individual exposure frames that are separate and independent from each other. For example, sensing hyperspectral 111 may produce a first hyperspectral exposure frame at a first partition of electromagnetic radiation, a second hyperspectral exposure frame at a second partition of electromagnetic radiation, and so on. Each exposure frame is assigned a specific color or wavelength partition, where the assignment is based on the timing of the pulse color or wavelength partition from emitter 102. The combination of an exposure frame and the assigned specific color or wavelength partition may be referred to as a data set. Even if pixel 122 is not a dedicated color, a color may be assigned to any given data set based on a priori information about the emitter.

[0069] For example, during operation, after the pulsed red 104 light is pulsed in the light-starved environment 112, the pixel array 122 senses the reflected electromagnetic radiation. The reflected electromagnetic radiation generates an exposure frame, and the exposure frame is classified as sensed red 105 data because it corresponds in time to the pulsed red 104 light. The exposure frame, along with an indication that it corresponds in time to the pulsed red 104 light, is a "data set." This process is repeated for each partition of electromagnetic radiation emitted by the emitter 102. The data created by the pixel array 122 includes a sensed red 105 exposure frame that identifies the red 114 component in the light-starved environment and corresponds in time to the pulsed red 104 light. The data also includes a sensed green 107 exposure frame that identifies the green 116 component in the light-starved environment and corresponds in time to the pulsed green 106 light. The data also includes a sensed blue 109 exposure frame that identifies the blue 118 component in the light-starved environment and corresponds in time to the pulsed blue 108 light. The data also includes sensing hyperspectral 111 exposure frames that identify elements sensitive to hyperspectral 120 radiation and correspond in time to hyperspectral 110 emissions.

[0070] In one embodiment, three data sets representing red, green, and blue electromagnetic pulses are combined to form a single image frame. Thus, the information in the red exposure frame, the green exposure frame, and the blue exposure frame are combined to form a single RGB image frame. One or more additional data sets representing other wavelength partitions may be superimposed on the single RGB image frame. The one or more additional data sets may represent, for example, laser scanning data, fluorescence imaging data, and / or hyperspectral imaging data.

[0071] It should be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition without departing from the scope of the present disclosure, and any color combination or any electromagnetic partition can be used to replace RED, GREEN and BLUE, such as cyan, magenta and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible wavelengths and invisible wavelengths. In the figure, the light-deficient environment 112 to be imaged includes a red 114 portion, a green 116 portion and a blue 118 portion, and also includes elements that can be sensed and mapped into a three-dimensional rendering that are sensitive to hyperspectral 120 radiation. As shown, the reflected light from the electromagnetic pulse only contains data for the portion of the object that has a specific color corresponding to the pulse color partition. These separate color (or color interval) data sets can then be used to reconstruct an image by combining the data sets at 126. The information in each of the multiple exposure frames (i.e., the multiple data sets) can be combined by a controller 124, a control unit, a camera control unit, an image sensor, an image signal processing pipeline, or some other computing resources that can be configured to process the multiple exposure frames and combine the data sets at 126. The data sets may be combined to generate a single image frame within the endoscopy unit itself or off-site by some other processing resource.

[0072] Figure 2The invention discloses a system 200 for providing illumination to a light-deficient environment such as for endoscopic imaging. The system 200 may be used in combination with any of the systems, methods or devices disclosed herein. The system 200 includes a transmitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an inner cavity waveguide 210, an inner cavity 212 and an image sensor 214 with accompanying optical components (such as lenses). The transmitter 202 (generally referred to as a "light source") generates light that passes through the jumper waveguide 206 and the inner cavity waveguide 210 to illuminate the scene at the distal end of the inner cavity 212. The transmitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, hyperspectral, fluorescence excitation, laser scanning pulse schemes or other wavelengths. The inner cavity 212 can be inserted into a patient for imaging, such as during surgery or examination. The light is output as shown by the dashed line 216. The scene illuminated by the light can be captured using an image sensor 214 and displayed to a doctor or some other medical personnel. The controller 204 may provide control signals to the emitter 202 to control when illumination is provided to a scene. In one embodiment, the emitter 202 and the controller 204 are located within a camera control unit (CCU) or an external console to which the endoscope is connected. If the image sensor 214 comprises a CMOS sensor, light may be periodically provided to the scene as a series of illumination pulses between readout cycles of the image sensor 214 during so-called blanking periods. Thus, light may be pulsed in a controlled manner to avoid stacking into the readout cycles of image pixels in the pixel array of the image sensor 214.

[0073] In one embodiment, the lumen waveguide 210 includes one or more optical fibers. These optical fibers can be made of low-cost materials (such as plastic) to allow processing of the lumen waveguide 210 and / or other parts of the endoscope. In one embodiment, the lumen waveguide 210 is a single glass fiber with a diameter of 500 microns. The jumper waveguide 206 can be permanently attached to the transmitter 202. For example, the jumper waveguide 206 can receive light from a transmitter in the transmitter 202 and provide light to the lumen waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 106 includes one or more glass optical fibers. The jumper waveguide can include any other type of waveguide for guiding light to the lumen waveguide 210. The connector 208 can selectively couple the jumper waveguide 206 to the lumen waveguide 210 and allow the light in the jumper waveguide 206 to pass through the lumen waveguide 210. In one embodiment, the intracavity waveguide 210 is coupled directly to the light source without any intervening jumper waveguide 206 .

[0074] The image sensor 214 includes a pixel array. In one embodiment, the image sensor 214 includes two or more pixel arrays for generating a three-dimensional image. The image sensor 214 may constitute two additional image sensors, each having an independent pixel array and operable independently of one another. The pixel array of the image sensor 214 includes effective pixels and optically black ("OB") pixels or light-blind pixels. An effective pixel may be a transparent "color-indeterminate" pixel capable of sensing imaging data of electromagnetic radiation of any wavelength. When the pixel array is "reset" or calibrated, the optically black pixels are read during the blanking period of the pixel array. In one embodiment, when reading the optically black pixels, light pulses during the blanking period of the pixel array. After the optically black pixels have been read, the effective pixels are read during the readout period of the pixel array. The effective pixels may be charged by the electromagnetic radiation pulsed during the blanking period, so that the effective pixels are ready to be read by the image sensor during the readout period of the pixel array.

[0075] Figure 2A is a schematic diagram of complementary system hardware such as a special or general-purpose computer. Implementations within the scope of the present disclosure may also include physical and other non-transient computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media that can be accessed by a general or special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a computer storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, a specific implementation of the present disclosure may include at least two distinct types of computer-readable media: a computer storage medium (device) and a transmission medium.

[0076] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general purpose or special purpose computer.

[0077] "Network" refers to one or more data links that enable electronic data to be transmitted between computer systems and / or modules and / or other electronic devices. In one specific implementation, the sensor and camera control unit can be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer reasonably regards the connection as a transmission medium. The transmission medium may include a network and / or data link that can be used to carry the desired program code tools in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer. The above combinations should also be included in the scope of computer-readable media.

[0078] In addition, upon reaching various computer system components, program code tools in the form of computer executable instructions or data structures can be automatically transmitted to computer storage media (devices) by transmission media (or vice versa). For example, computer executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., "NIC") and then ultimately transmitted to the computer system RAM and / or the non-volatile computer storage media (devices) of the computer system. RAM can also include solid state drives (SSDs or PCIx-based real-time memory hierarchical storage devices, such as FusionIO). Therefore, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0079] Computer executable instructions include, for example, instructions and data that cause a general purpose computer, a special purpose computer, or a special purpose processing device to perform certain functions or groups of functions when executed by one or more processors. Computer executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter of the present invention is described linguistically with respect to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or steps described above. Rather, the features and steps described above are disclosed as example forms of implementing the claims.

[0080] It should be understood by those skilled in the art that the present disclosure can be implemented in a network computing environment having various types of computer system configurations, including personal computers, desktop computers, laptop computers, information processors, control units, camera control units, handheld devices, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the above computing devices can be provided by a physical institution or located within a physical institution. The present disclosure can also be implemented in a distributed system environment, in which local and remote computer systems are connected via a network (via a hard-wired data link, a wireless data link, or a combination of a hard-wired data link and a wireless data link), and both can perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0081] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0082] Figure 2A is a block diagram illustrating an exemplary computing device 250. Computing device 250 may be used to execute various programs, such as those discussed herein. Computing device 250 may be used as a server, a client, or any other computing entity. Computing device 250 may perform the various monitoring functions discussed herein, and may execute one or more applications, such as the applications described herein. Computing device 250 may be any of a variety of computing devices, such as a desktop computer, a laptop computer, a server computer, a handheld computer, a camera control unit, a tablet computer, etc.

[0083] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory device 254 and / or the mass storage device 258. The processor 252 may also include various types of computer-readable media, such as cache memory.

[0084] Memory device 254 includes various computer-readable media such as volatile memory (e.g., random access memory (RAM) 264) and / or nonvolatile memory (e.g., read-only memory (ROM) 266). Memory device 254 may also include a rewritable ROM such as flash memory.

[0085] The mass storage device 258 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., flash memory), etc. Figure 2 As shown in FIG. 2 , the particular mass storage device is a hard drive 274. Various drives may also be included in the mass storage device 258 to enable reading from and / or writing to various computer-readable media. The mass storage device 258 includes removable media 276 and / or non-removable media.

[0086] I / O devices 260 include various devices that allow data and / or other information to be input to or retrieved from computing device 250. Exemplary I / O devices 260 include digital imaging devices, electromagnetic sensors and transmitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCD or other image capture devices, and the like.

[0087] Display device 280 includes any type of device capable of displaying information to one or more users of computing device 250. Examples of display device 280 include a monitor, a display terminal, a video projection device, and the like.

[0088] Interface 256 includes various interfaces that allow computing device 250 to interact with other systems, devices, or computing environments. Exemplary interface 256 may include any number of different network interfaces 270, such as interfaces to connect local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include user interface 268 and peripheral device interface 272. Interface 256 may also include one or more user interface elements 268. Interface 256 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touch pad, etc.), keyboards, etc.

[0089] The bus 262 allows the processor 252, memory device 254, interface 256, mass storage device 258, and I / O device 260 to communicate with each other and with other devices or components coupled to the bus 262. The bus 262 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, etc.

[0090] For exemplary purposes, the programs and other executable program components shown herein are discrete blocks, but it should be understood that such programs and components may reside at various times in different storage components of the computing device 250 and be executed by the processor 252. Alternatively, the systems and programs described herein may be implemented in hardware, or in a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more of the systems and programs described herein.

[0091] Figure 3A An operating cycle of a sensor used in a rolling readout mode or during a sensor readout 300 is shown. A frame readout may begin at and may be represented by a vertical line 310. A readout cycle is represented by a diagonal or slanted line 302. The sensor may be read out row by row, with the top of the downward sloping edge being the sensor top row 312 and the bottom of the downward sloping edge being the sensor bottom row 314. The time between the last row readout and the next readout cycle may be referred to as a blanking period 316. It should be noted that some of the sensor pixel rows may be covered with a light shield (e.g., a metal coating or any other substantially black layer of another material type). These covered pixel rows may be referred to as optical black rows 318 and 320. Optically black rows 318 and 320 may be used as input to a correction algorithm. As shown Figure 3A As shown, these optically black rows 318 and 320 may be located on the top of the pixel array or on the bottom of the pixel array or on both the top and bottom of the pixel array.

[0092] Figure 3B A method for controlling the amount of electromagnetic radiation (e.g., light) exposed to a pixel so that it is integrated or accumulated by the pixel is shown. It should be understood that photons are the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into an electric charge or current. An electronic shutter or rolling shutter (shown by dashed line 322) can be used to start the integration time by resetting the pixel. The light will then be integrated until the next readout phase. The position of the electronic shutter 322 can be moved between two readout cycles 302 to control the pixel saturation for a given amount of light. It should be noted that this technique allows a constant integration time between two different rows, but introduces a delay when moving from the top row to the bottom row.

[0093] Figure 3C The case is shown where the electronic shutter 322 has been removed. In this configuration, integration of incident light may begin during readout 302 and may end at the next readout cycle 302, which also defines the start of the next integration.

[0094] Figure 3DA configuration is shown without the electronic shutter 322 but with controlled and pulsed light 330 during the blanking period 316. This ensures that all rows see the same light as that emitted from the same light pulse 330. In other words, each row will begin its integration in a dark environment, which may be located at the optically black post row 320 of the readout frame (m) to obtain the maximum light pulse width, and will then receive light through and will end its integration in a dark environment, which may be located at the optically black pre row 318 of the next subsequent readout frame (m+1) to obtain the maximum light pulse width. In e.g. Figure 3D In the embodiment of the present invention, the image generated by the light pulse will be available only during the readout of frame (m+1) without interfering with frames (m) and (m+2). It should be noted that the condition for the light pulse to be read out in only one frame and not interfere with adjacent frames is that a given light pulse is fired during the blanking period 316.

[0095] Because the optical black rows 318 , 320 are not sensitive to light, the optical black rear row 320 time of frame (m) and the optical black front row 318 time of frame (m+1) may be added to the blanking period 316 to determine the maximum range of firing times of the light pulses 330 .

[0096] like Figure 3A As shown, the sensor may cycle multiple times to receive data for each pulse color or wavelength (e.g., red, green, blue, or other wavelength on the electromagnetic spectrum). Each cycle may be timed. In one embodiment, the cycle may be timed to operate within an interval of 16.67 ms. In another embodiment, the cycle may be timed to operate within an interval of 8.3 ms. It should be understood that the present disclosure contemplates other timing intervals and is intended to fall within the scope of the present disclosure.

[0097] Figure 4A The operation of an embodiment of an electromagnetic transmitter is graphically illustrated.The transmitter may be timed to correspond to a cycle of the sensor such that electromagnetic radiation is emitted within and / or during a portion of a sensor operating cycle. Figure 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the emitter may pulse during the readout portion 302 of the sensor operating cycle. In one embodiment, the emitter may pulse during the blanking portion 316 of the sensor operating cycle. In one embodiment, the emitter may pulse for a duration that is during a portion of two or more sensor operating cycles. In one embodiment, the emitter may begin pulsing during the blanking portion 316 or during the optically black portion 320 of the readout portion 302, and end pulsing during the readout portion 302 or during the optically black portion 318 of the readout portion 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of the present disclosure as long as the pulsing of the emitter and the cycle of the sensor correspond.

[0098] Figure 4B A graphical representation of varying the duration and magnitude of emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, pulse 3 at 416) to control exposure. An emitter with a fixed output magnitude may be combined with the above Figure 3D and Figure 4A The pulses are pulsed at intervals during any of the cycles described to provide the desired electromagnetic energy to the pixel array. An emitter with a fixed output magnitude may be pulsed at longer intervals to provide more electromagnetic energy to the pixel, or the emitter may be pulsed at shorter intervals to provide less electromagnetic energy. Whether longer or shorter intervals are required depends on the operating conditions.

[0099] Compared to adjusting the time interval of the fixed output magnitude of the emitter pulse, the emission magnitude itself can be increased to provide more electromagnetic energy to the pixel. Similarly, reducing the magnitude of the pulse can provide less electromagnetic energy to the pixel. It should be noted that embodiments of the system can have the ability to adjust both the magnitude and duration if desired. In addition, the sensor can be adjusted to increase its sensitivity and duration as required for optimal image quality. Figure 4B The magnitude and duration of the pulses are shown to be varied. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. In addition, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416, so that the electromagnetic energy provided by the pulse is shown by the area under the pulse shown in the illustration. In the illustration, pulse 2 at 414 has a relatively low magnitude or intensity and a longer duration when compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration when compared to pulse 1 at 412 and pulse 2 at 414.

[0100] Figure 5 According to the principles and teachings of this disclosure, FIG. 3A to FIG. 3D and Figure 4A 302 is a graphical representation of an embodiment of the present disclosure of an operating cycle of an image sensor, an electromagnetic emitter, and an electromagnetic pulse emitted to show an imaging system during operation. As can be seen in the figure, the electromagnetic emitter pulses radiation primarily during the blanking period 316 of the image sensor so that the pixels will be charged and ready to read during the readout period 302 of the image sensor cycle. Figure 5 The dashed lines in the figure represent pulses of electromagnetic radiation (from Figure 4A ). The pulses of electromagnetic radiation are primarily emitted during a blanking period 316 of the image sensor, but may overlap with a readout period 302 of the image sensor.

[0101] The exposure frame includes data read by the pixel array of the image sensor during the readout cycle 302. The exposure frame can be combined with an indication of the type of pulse emitted by the emitter before the readout cycle 302. The combination of the exposure frame and the indication of the pulse type can be referred to as a data set. Multiple exposure frames can be combined to generate a black and white or RGB color image. In addition, hyperspectral, fluorescence and / or laser mapping imaging data can be superimposed on the black and white or RGB image.

[0102] In one embodiment, an RGB image frame is generated based on three exposure frames, including a red exposure frame generated by the image sensor after red emission, a green exposure frame generated by the image sensor after green light emission, and a blue exposure frame generated by the image sensor after blue light emission. Hyperspectral imaging data can be stacked on the RGB image frame. Hyperspectral imaging data can be extracted from one or more hyperspectral exposure frames. The hyperspectral exposure frame includes data generated by the image sensor during the readout cycle 302 after hyperspectral emission of electromagnetic radiation. Hyperspectral emission includes any suitable emission in the electromagnetic spectrum, and may include multiple light emissions across the entire electromagnetic spectrum. In one embodiment, the hyperspectral emission includes emission of electromagnetic radiation having a wavelength from about 513nm to about 545nm, from about 565nm to about 585nm, and / or from about 900nm to about 1000nm. The hyperspectral exposure frame may include multiple hyperspectral exposure frames, each of which is generated by the image sensor after a different type of hyperspectral emission. In one embodiment, the hyperspectral exposure frames include a plurality of hyperspectral exposure frames including a first hyperspectral exposure frame generated by the image sensor after emitting electromagnetic radiation having a wavelength from about 513 nm to about 545 nm, a second hyperspectral exposure frame generated by the image sensor after emitting electromagnetic radiation having a wavelength from about 565 nm to about 585 nm, and a third hyperspectral exposure frame generated by the image sensor after emitting electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm. The hyperspectral exposure frames may include additional additional hyperspectral exposure frames generated by the image sensor after other hyperspectral emissions of light as required based on the imaging application.

[0103] The hyperspectral exposure frame may be generated by the image sensor after emitting multiple different partitions of electromagnetic radiation. For example, a single hyperspectral exposure frame may be sensed by the pixel array after emitting electromagnetic radiation having wavelengths from about 513 nm to about 545 nm, from about 565 nm to about 585 nm, and from about 900 nm to about 1000 nm. The emission of electromagnetic radiation may include a single pulse in which each of the multiple wavelengths is emitted simultaneously, multiple sub-pulses of electromagnetic radiation in which each sub-pulse is a different wavelength, or some combination of the two. The emission of electromagnetic radiation having one or more pulses may occur during a blanking period 316 that occurs before the readout period 302 in which the pixel array senses the exposure frame.

[0104] In one embodiment, the exposure frame is data sensed by the pixel array during a readout period 302 that occurs after a blanking period 316. The emission of electromagnetic radiation is emitted during the blanking period 316. In one embodiment, a portion of the emission of electromagnetic radiation overlaps the readout period 316. The blanking period 316 occurs when optically black pixels of the pixel array are being read, and the readout period 302 occurs when valid pixels of the pixel array are being read. The blanking period 316 may overlap the readout period 302.

[0105] Fig. 6A and Figure 6B A process for recording an image frame is shown. Multiple image frames may be strung together to generate a video stream. A single image frame may include data from multiple exposure frames, where an exposure frame is data sensed by a pixel array after emitting electromagnetic radiation. Fig. 6A A conventional process is shown that is typically implemented with a color image sensor having a color filter array (CFA) to filter out certain wavelengths of light for each pixel. Figure 6B is a process disclosed herein and can be implemented using a monochromatic "color amorphous" image sensor that receives electromagnetic radiation of all wavelengths.

[0106] Fig. 6A The process shown occurs from time t(0) to time t(1). The process begins with white light emission 602 and sensing white light 604. At 606, an image is processed and displayed based on the sensing at 604.

[0107] Figure 6BThe process shown occurs from time t(0) to time t(1). The process begins by emitting green light 612, and sensing reflected electromagnetic radiation 614 after emitting green light 612. The process continues by emitting red light 616, and sensing reflected electromagnetic radiation 618 after emitting red light 616. The process continues by emitting blue light 620, and sensing reflected electromagnetic radiation 622 after emitting blue light 620. The process continues with one or more transmissions of hyperspectral 624 transmissions, and sensing reflected electromagnetic energy 626 after each of the one or more transmissions of hyperspectral 624 transmissions.

[0108] Figure 6B The process shown provides a higher resolution image and provides a means for generating an RGB image that also includes hyperspectral imaging data. Figure 6B ), making the sensor sensitive to electromagnetic energy of all wavelengths. Figure 6B In the process shown, the monochrome pixel array is instructed to sense electromagnetic energy from a predetermined partition of the full spectrum of electromagnetic energy in each cycle. Therefore, to form an image, the sensor only needs to cycle with multiple different partitions from within the full spectrum. The final image is assembled based on the multiple cycles. Because the image from each color partition frame cycle has a higher resolution (compared to the CFA pixel array), the resulting image created when the partitioned light frames are combined also has a higher resolution. In other words, because each pixel within the array (rather than, at most, every other pixel in a sensor with a CFA) senses the magnitude of the energy for a given pulse and a given scene, only a period of time apart, a higher resolution image is produced for each scene.

[0109] As in Fig. 6A and Figure 6B In the embodiment shown in FIG. 1 , it can be seen graphically between time t(0) and t(1) that Figure 6B Sensor pair in the partitioned spectral system Fig. 6A Each of the full spectrum systems in the embodiment is cycled at least four times. In one embodiment, the display device (LCD panel) operates at a speed of 50 to 60 frames per second. In such embodiments, Figure 6B The partitioned optical system in can operate at a speed of 200 to 240 frames per second to maintain the continuity and smoothness of the displayed video. In other embodiments, there may be different capture and display frame rates. In addition, the average capture rate can be any multiple of the display rate.

[0110] In one embodiment, it may be desirable that not all partitions are represented equally within the system frame rate. In other words, not all light sources must be pulsed with the same regularity in order to emphasize and de-emphasize various aspects of the recorded scene as desired by the user. It should also be understood that invisible and visible partitions of the electromagnetic spectrum can be pulsed together within the system, with their corresponding data values ​​stitched into the video output for display to the user.

[0111] Embodiments may include the following pulse cycle patterns:

[0112] i. Green pulse;

[0113] ii. Red pulse;

[0114] iii. Blue pulse;

[0115] iv. Green pulse;

[0116] v.Red pulse;

[0117] vi.Blue pulse;

[0118] vii. Hyperspectral pulses;

[0119] viii.(repeat)

[0120] As can be seen in this example, a hyperspectral partition can be pulsed at a different rate than the other partitions. Doing so can emphasize a certain aspect of a scene, where the hyperspectral data is simply overlaid with other data in the video output to make the desired emphasis. It should be noted that adding a hyperspectral partition on top of the red, green, and blue partitions does not necessarily require a serialized system to operate at four times the rate of a full-spectrum non-serial system, because each partition does not have to be represented equally in the pulse pattern. As seen in this embodiment, adding a hyperspectral partition that pulses less frequently in the pulse pattern will result in an increase in the sensor's cycle speed by less than 20% in order to accommodate irregular partition sampling.

[0121] In various embodiments, the pulse cycle pattern may also include any of the following wavelengths in any suitable order. Such wavelengths may be particularly suitable for exciting fluorescent agents to generate fluorescence imaging data by sensing the relaxation emission of the fluorescent agent based on the relaxation emission of the fluorescent agent:

[0122] i.770±20nm;

[0123] ii.770±10nm;

[0124] iii.770±5nm;

[0125] iv.790±20nm;

[0126] v.790±10nm;

[0127] vi.790±5nm;

[0128] vii.795±20nm;

[0129] viii.795±10nm;

[0130] ix.795±5nm;

[0131] x.815±20nm;

[0132] xi.815±10nm;

[0133] xii.815±5nm;

[0134] xiii. 770nm to 790nm; and / or

[0135] xiv.795nm to 815nm.

[0136] In various embodiments, the pulse cycle may also include any of the following wavelengths in any suitable order. Such wavelengths may be particularly suitable for generating hyperspectral imaging data:

[0137] i.513±545nm;

[0138] ii.565nm to 585nm;

[0139] iii.900nm to 1000nm;

[0140] iv.513±5nm;

[0141] v.513±10nm;

[0142] vi.513±20nm;

[0143] vii.513±30nm;

[0144] viii.513±35nm;

[0145] ix.545±5nm;

[0146] x.545±10nm;

[0147] xi.545±20nm;

[0148] xii.545±30nm;

[0149] xiii.545±35nm;

[0150] xiv.565±5nm;

[0151] xv.565±10nm;

[0152] xvi.565±20nm;

[0153] xvii.565±30nm;

[0154] xviii.565±35nm;

[0155] xix.585±5nm;

[0156] xx.585±10nm;

[0157] xxi.585±20nm;

[0158] xxii.585±30nm;

[0159] xxiii.585±35nm;

[0160] xxiv.900±5nm;

[0161] xxv.900±10nm;

[0162] xxvi.900±20nm;

[0163] xxvii.900±30nm;

[0164] xxviii.900±35nm;

[0165] xxix.1000±5nm;

[0166] xxx.1000±10nm;

[0167] xxxi.1000±20nm;

[0168] xxxii.1000±30nm; or

[0169] xxxiii.1000±35nm.

[0170] The partition loop can be divided into various imaging and video standards to accommodate or approximate. In one embodiment, the partition loop may include the following: 7A to 7D Pulses of electromagnetic energy in the red, green, and blue spectrum are best shown in FIG. Fig. 7A In , different light intensities have been achieved by modulating the light pulse width or duration within the operating range shown by the vertical grey dashed line. Figure 7BIn , different light intensities have been achieved by modulating the power of an optical or electromagnetic emitter (which may be a laser or LED emitter), but keeping the pulse width or duration constant. Figure 7C The case where both the optical power and optical pulse width are modulated for greater flexibility is shown. The zoned cycles can use cyan magenta yellow (CMY), infrared, ultraviolet, hyperspectral and fluorescent, which use invisible pulse sources mixed with visible pulse sources, and any other color space required to produce an image or approximate a desired video standard currently known or yet to be developed. It should also be understood that the system can switch between color spaces on the fly to provide the desired image output quality.

[0171] When using the color space green-blue-green-red (such as Fig.7D In an embodiment (as shown in ), it may be desirable to pulse the luma component more frequently than the chroma component because users are generally more sensitive to differences in light magnitude than to differences in light color. Fig.7D The monochrome sensor shown in Figure utilises this principle. Fig.7D In the scheme, green, which contains the most brightness information, can be pulsed more frequently or with greater intensity to obtain brightness data in the (GBGRGBGR...) scheme. Such a configuration will create a video stream with more perceptible details without creating and transmitting imperceptible data.

[0172] In one embodiment, duplicate pulses of weaker partitions may be used to produce an output that has been adjusted for the weaker pulses. For example, given the weak sensitivity of blue lasers relative to silicon-based pixels, and the difficulty of producing blue lasers compared to red or green light, it may be pulsed more frequently during a frame cycle to compensate for the weakness of the light. These additional pulses may be done continuously over time, or by using multiple lasers to pulse simultaneously to produce the desired compensating effect. It should be noted that by pulsing during blanking periods (time during which the sensor is not reading out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same kind, and simply accumulates the light for the desired output. In another embodiment, the maximum light pulse range may be different from frame to frame. This may be done in a controlled manner. Fig. 7E , where the light pulses are different from frame to frame. The sensor can be constructed to be able to program different blanking periods in a repeating pattern of two frames or three frames or four frames or n frames. Fig. 7E In FIG. 4 , four different light pulses are shown, and pulse 1 may be repeated, for example, after pulse 4, and may have a pattern of four frames with different blanking periods. This technique may be used to place the most powerful partition at the smallest blanking period, and thus allow the weakest partition to have a wider pulse on one of the subsequent frames without increasing the readout speed. The reconstructed frame may still have a regular pattern from frame to frame, since it is made up of many pulse frames.

[0173] Figure 8 A process flow 800 is shown in which a sensor and / or emitter are adjusted to compensate for differences in energy values ​​of a pulse partition spectrum. In process flow 800, data is obtained from a histogram of a previous frame and analyzed at 802. The sensor is adjusted at 804, and the emitter is adjusted at 806. At 808, an image is determined based on the adjusted sample time from the sensor, and / or at 810, an image is determined based on the adjusted (increased or decreased) emitted light.

[0174] Fig. 9 9 is a process flow 900 for adjusting a sensor and recording a frame based on readings from the adjusted sensor. In the process flow 900, a histogram of a previous frame is obtained at 902 and the sensor is adjusted based on sensitivity at 904. At 906, a frame is recorded based on the readings from the adjusted sensor. In one example, the process flows 800, 900 are implemented because the red light spectrum is more easily detected by the sensor within the system than the blue light spectrum. In this example, because the blue partition has a low quantum efficiency relative to silicon (in Fig. 9 ), so the sensor is adjusted to be less sensitive during the red partition cycle and more sensitive during the blue partition cycle. Similarly, the emitter can be adjusted to provide adjusted partitions (e.g., higher or lower intensity and duration).

[0175] Figure 10 to Figure 11 is a schematic diagram of a method for performing color correction on digital imaging. It is common in digital imaging to manipulate values ​​within the image data to correct the output to meet the expectations of a user or to highlight certain aspects of the imaged object. In systems where light is controlled by the user, it is advantageous to provide light emissions that are known to the user and may be only a portion of the electromagnetic spectrum or multiple portions of the full electromagnetic spectrum. Image correction is important to meet the expectations of the user and to detect faults within the system. One calibration method employs a table of expected values ​​for a given imaging condition that can be compared to the data from the image sensor.

[0176] Fig.101 is a schematic diagram of a method 1000 for performing color correction on digital imaging in a light-deficient environment. The method 1000 includes sampling a neutral color scene or palette (see 1010) at startup by running a full cycle of electromagnetic spectrum partitions at 1002. A lookup table 1008 is generated based on the color neutral scene or palette 1010. The lookup table 1008 is used to determine a histogram for a frame cycle at 1008. At 1006, the histogram is compared to known or expected values ​​based on the neutral color scene or palette 1010 and also based on the lookup table 1008. The method includes adjusting settings on an image signal processor (ISP) at 1012 and / or adjusting an emitter at 1014. The adjustment of the emitter at 1014 may include adjustment of any aspect of the emitted light, such as the magnitude, duration (i.e., on-time), or range within the spectrum partition.

[0177] It should be noted that because each partitioned spectrum of light can have different energy values, the sensor and / or light emitter can be adjusted to compensate for the difference in energy values. For example, in one embodiment, because the blue light spectrum has a lower quantum efficiency than the red light spectrum with respect to silicon-based imagers, the responsiveness of the sensor can then be adjusted to respond less during the red cycle and more during the blue cycle. Conversely, because the quantum efficiency of blue light is lower than that of red light, the emitter can emit blue light at a higher intensity than red light to produce a correctly exposed image.

[0178] Fig.11 1 is a schematic diagram of a method 1100 for performing hierarchical adjustments to an image signal processor (ISP) and / or a transmitter to reduce the amount of noise and artifacts within an output image stream or video. The method 1100 includes transmitting and sensing a full cycle of an electromagnetic spectrum partition at 1102. At 1104, the results of the full cycle of the electromagnetic partition are compared with the desired output. Based on the comparison, the image signal processor (ISP) is adjusted at 1106 and / or the transmitter is adjusted at 1108. The adjustments made to the ISP at 1106 and / or the transmitter at 1108 between frame cycles can be limited at 1110. For example, the transmitter can be adjusted by a portion of its operating range at any time between frames. Similarly, the ISP can be adjusted by a portion of its operating range at any time between frames. In one embodiment, both the transmitter and the ISP can be limited so that only a portion of the respective operating ranges of the transmitter and sensor can be adjusted together at any time between frames. The results of these hierarchical adjustments are compared at 1112, and adjustments are completed based on the comparison.

[0179] In an exemplary embodiment, a stepped adjustment of the ISP and / or emitter may be performed at approximately 0.1 dB of the operating range of the component to correct for the exposure of the previous frame. 0.1 dB is merely an example, and it should be noted that in other embodiments, the allowed adjustment of the component may be any portion of its respective operating range. Components of the system may be changed by intensity or duration adjustments that are typically controlled by the number of bits (resolution) output by the component. Component resolution may typically range between approximately 10 bits and 24 bits, but should not be limited to this range as it is intended to include resolutions of components yet to be developed in addition to those currently available. For example, after the first frame, it is determined that the scene is too blue when viewed, then the emitter may be adjusted by a stepped adjustment as described above (such as approximately 0.1 dB) to reduce the pulse magnitude or duration of blue light during the blue cycle of the system.

[0180] In this exemplary embodiment, more than 10% may be needed, but the system has limited itself to an operating range of 0.1 dB adjustment per system cycle. Therefore, the blue light can be adjusted again during the next system cycle if necessary. The stepped adjustment between cycles can have a damping effect on the output image, and will reduce noise and artifacts when operating the emitters and sensors at their operating limits. Any stepped amount that can determine the operating adjustment range of a component can be used as a limiting factor, or it can be determined that certain embodiments of the system may include components that are adjustable throughout their operating range.

[0181] Additionally, the optical black area of ​​any image sensor can be used to aid in image correction and noise reduction. In one embodiment, the values ​​read from the optical black area can be compared with those of the active pixel area of ​​the sensor to establish a reference point to be used for image data processing. Fig.12 The types of sensor correction methods that can be employed in a color pulse system are shown. CMOS image sensors typically have multiple non-idealities such as fixed pattern noise (FPN) and line noise. Fully controlling the illumination has the following benefits: an entire dark data frame can be periodically acquired and used to correct for pixel and column offsets.

[0182] FPN is dispersion in the sensing element offset, which is usually caused by pixel-to-pixel dispersion caused by random variations in dark current from photodiode to photodiode. Column fixed pattern noise is caused by offsets associated with specific columns of pixels in the readout chain and can produce perceived vertical streaks within the image.

[0183] Row noise is the random temporal variation of pixel offsets within each row. Because row noise is temporal, the correction must be recalculated for each row and each frame. To do this, there are typically a number of light-blind (OB) pixels within each row in the array that must first be sampled to estimate the row offset before sampling the light-sensitive pixels. The row offset is then subtracted during the row noise correction process.

[0184] exist Fig.12 In the example of FIG. 1200 , there are other corrections involving obtaining data in the correct order, monitoring and controlling voltage offsets in the analog domain (black clamping), and identifying / correcting individual defective pixels. Process flow 1200 includes cycling the sensor through each of the electromagnetic partitions at a first emission intensity at 1202. Process flow 1200 includes cycling the sensor through each of the electromagnetic partitions at a second emission intensity at 1204. Process flow 1200 includes combining data from the electromagnetic partitions at the first emission intensity and the second emission intensity at 1206.

[0185] Fig.13 13 is a schematic diagram of a process flow 1300 for increasing the dynamic range of a resulting image. The process flow 1300 includes cycling the sensor through each of the electromagnetic sectors at a first emission intensity at 1302. The process flow 1300 includes cycling the sensor through each of the electromagnetic sectors at a second emission intensity at 1304. The process flow 1300 includes cycling the sensor through “n” electromagnetic sectors at an emission intensity of “m” at 1306, and may be repeated any suitable number of times. The process flow 1300 includes cycling the sensor through “n+i” electromagnetic sectors at an emission intensity of “m+j” at 1308. The process flow 1300 includes combining data from each of the cycled emission intensities at 1310.

[0186] In one embodiment, the exposure input may be input at different levels over time and combined to produce a greater dynamic range. A greater dynamic range may be particularly desirable due to the limited spatial environment in which the imaging device is used. In a limited spatial environment that lacks light or is dark except for the light provided by the light source, and where the light source is close to the light emitter, exposure has an exponential relationship with distance. For example, objects close to the light source and the optical opening of the imaging device tend to be overexposed, while objects farther away tend to be extremely underexposed because there is very little (any) ambient light.

[0187] As in Fig.13As can be seen in FIG. 1 , the cycle of a system having electromagnetic energy emission in multiple partitions can be continuously cycled according to the partitions of the electromagnetic spectrum. For example, in an embodiment in which the emitter emits lasers in different red partitions, different blue partitions, different green partitions, and different hyperspectral partitions, the two cycle data sets to be combined can be of the following form:

[0188] i. Red with an intensity of one at 1302;

[0189] ii. Red with an intensity of two at 1304;

[0190] iii. Blue with an intensity of one at 1302;

[0191] iv. Blue with an intensity of two at 1304;

[0192] v. Green with an intensity of one at 1302;

[0193] vi. Green with an intensity of two at 1304;

[0194] vii. A hyperspectral spectrum having an intensity of one at 1302; and

[0195] viii. Hyperspectrum at 1304 with intensity of two.

[0196] Alternatively, the system can loop as follows:

[0197] i. Red with an intensity of one at 1302;

[0198] ii. Blue with an intensity of one at 1302;

[0199] iii. Green with an intensity of one at 1302;

[0200] iv. a hyperspectral image with an intensity of one at 1302;

[0201] v. Red with intensity two at 1304;

[0202] vi. Blue with an intensity of two at 1304;

[0203] vii. Green at an intensity of two at 1304; and

[0204] viii. Hyperspectrum at 1304 with intensity of two.

[0205] In such embodiments, the first image may be derived from intensity one values ​​and the second image may be derived from intensity two values ​​and then combined or processed into a complete image dataset at 1310 rather than at its component parts.

[0206] It is contemplated within the scope of the present invention that any number of transmit partitions may be used in any order. Fig.13 As shown in , "n" is used as a variable to represent any number of electromagnetic partitions, and "m" is used to represent any level of strength of the "n" partitions. Such a system can be looped as follows:

[0207] i. n with intensity m at 1306;

[0208] ii. n+1 with strength m+1;

[0209] iii. n+2 with strength m+2; and

[0210] iv. n+i with intensity m+j at 1308.

[0211] Thus, any sequenced cyclic pattern may be used to produce the desired image correction, where "i" and "j" are additive values ​​within the operating range of the imaging system.

[0212] Fig.14 A process flow 1400 is shown to be implemented by a controller and / or a monochrome image signal processor (ISP) for generating a video stream of RGB images with hyperspectral data superimposed thereon. An image signal processor (ISP) chain may be assembled for the purpose of generating a sequence of sRGB images from raw sensor data produced in the presence of a GRGB-hyperspectral light pulse scheme. In process flow 1400, the first stage involves performing a correction (see Fig.14 1406) to account for any non-ideal factors in the sensor technology that are best suited to work in the raw data domain. In the next stage, multiple frames are buffered (e.g., green frame 1408a, red-blue frame 1408b, and hyperspectral frame 1408c) because each final frame derives data from multiple raw frames. Frame reconstruction at 1264 continues by sampling data from the current frame and two buffered frames (see 1408a, 1408b and / or 1408c). The reconstruction process produces a full-color frame including hyperspectral image data in linear RGB color space. In this example, white balance coefficients at 1268 and a color correction matrix at 1420 are applied before conversion to YCbCr space at 1422 for subsequent edge enhancement at 1424. After edge enhancement at 1424, the image is converted back to linear RGB at 1426 for scaling (if applicable) at 1428. Finally, a gamma transfer function is applied at 1430 to convert the data to the sRGB domain at 1432.

[0213] Fig.151504 and 1505. The color fusion hardware 1500 uses a memory writer 1502 to store the RGBG-hyperspectral video data stream in a memory 1504 and convert the video data stream into a parallel RGB+hyperspectral video data stream at 1505. The bit width on the input side may be, for example, 12 bits per color. The output width of this example will be at least 36 bits per pixel. Other embodiments may have different initial bit widths and a number 3 times the output width. The memory writer 1502 block takes the RGBG-hyperspectral video stream as its input and writes each frame to its correct frame memory 1504 (the memory writer triggers the same pulse generator that runs the laser source (see 1510)). The memory 1504 may store the exposure frame data in a pattern such as that shown, i.e., red, green 1, blue, green 2, hyperspectral, and then start back again with red. The memory reader 1506 reads three frames at a time to construct RGB pixels. Each pixel is three times the bit width of a single color component. At 1510, the memory reader 1506 also triggers the laser pulse generator. In one embodiment, the memory reader 1506 waits until the red frame, green 1 frame, and blue frame have been written, and then continues to read them out in parallel while the writer continues to write green 2, hyperspectral, and begins to return to red. When red is completed, the reader begins reading from blue, green 2, hyperspectral, and red. This pattern continues indefinitely.

[0214] Fig.16 16 is a schematic diagram of a process flow 1600 for a sensor correction process. The process flow 1600 can be used in color and hyperspectral pulse systems as discussed herein. The process flow 1600 can be used to counteract non-ideal factors in CMOS image sensors, such as fixed pattern noise (FPN) and row noise. Fixed pattern noise is the dispersion in the offset of the sensing element. Typically, most of the FPN is inter-pixel dispersion, which originates from random variations in dark current from photodiode to photodiode and other sources. The system disclosed herein maintains full control over the illumination source, and this enables dark data to be collected and used to correct pixel and column offsets. In the illustrated example, a single frame buffer can be used to perform a sliding average over the entire frame in the absence of light using, for example, a simple exponential smoothing filter. During normal operation, this dark average frame can be subtracted from each illumination frame. Row noise is the random temporal variation of pixel offsets within each row. Because row noise is temporal, corrections are calculated for each row and each frame. To this end, there are typically many light-blind (OB) pixels within each row in the pixel array. Before sampling the photosensitive pixels, the OB pixels must first be sampled to evaluate the line offset. The row offset is then subtracted during the row noise correction process.

[0215] The process flow 1600 includes performing top deserialization 1602 and bottom deserialization 1603. The process flow 1600 includes performing line reordering at the top port at 1604 and performing line reordering at the bottom port at 1605. Upon completion of the line reordering, the information may be stored in a separate database 1632, 1634 or other memory device. The process flow 1600 includes performing black clamp calculations on the top ADC at 1606 and performing black clamp calculations on the bottom ADC at 1607. The information exits the process flow 1600 on a first-in, first-out (FIFO) basis. The process flow 1600 includes performing row noise correction at the top ADC at 1608 and performing row noise correction at the bottom ADC at 1609. The process flow 1600 includes performing full row reordering at 1610 and performing dark frame accumulation at 1612. The information may be stored in a database 1630 or other storage device before performing fixed pattern noise (FPN) correction. The process flow includes performing fixed pattern noise (FPN) correction at 1614 and performing pixel defect correction at 1616. The process flow 1600 includes performing programmable digital gain at 1618 before the video stream leaves the process flow 1600 to be provided to a user.

[0216] Fig.17 is a schematic diagram of the pattern reconstruction process. Fig.17 The exemplary pattern shown includes a red pulse, a green pulse, a blue pulse, and a hyperspectral pulse of light each lasting a duration of T1. In various embodiments, the light pulses may have the same duration or different durations. The red exposure frame, the green exposure frame, the blue exposure frame, and the hyperspectral exposure frame are combined to generate an RGB image with hyperspectral data layered thereon. A single image frame including a red exposure frame, a green exposure frame, a blue exposure frame, and a hyperspectral exposure frame requires a time period of 4*T1 to be generated. Fig.17 The durations shown are exemplary only and may vary for different specific implementations. In other embodiments, different pulse schemes may be used. For example, an embodiment may be based on the timing of each color component or frame (T1), and the reconstructed frame has a period that is twice the period of the input color frame (2×T1). Different frames within a sequence may have different frame periods, and the average capture rate may be any multiple of the final frame rate.

[0217] In one embodiment, the dynamic range of the system is increased by changing the pixel sensitivity of pixels within the pixel array of the image sensor. Some pixels may sense reflected electromagnetic radiation at a first sensitivity level, other pixels may sense reflected electromagnetic radiation at a second sensitivity level, and so on. Different pixel sensitivities may be combined to increase the dynamic range provided by the pixel configuration of the image sensor. In one embodiment, adjacent pixels are set with different sensitivities so that each cycle includes data generated by pixels that are more sensitive and less sensitive relative to each other. When multiple sensitivities are recorded in a single cycle of the pixel array, the dynamic range is increased. In one embodiment, a wide dynamic range can be achieved by having multiple global TXs, each TX firing only on different groups of pixels. For example, in global mode, the global TX1 signal is firing pixel group 1, the global TX2 signal is firing pixel group 2, the global TXn signal is firing pixel group n, and so on.

[0218] Fig.18A An example of the timing of two different pixel sensitivities (dual pixel sensitivity) in a pixel array is shown. In this case, the global TX1 signal fires half of the pixels of the array and the global TX2 fires the other half of the pixels. Because global TX1 and global TX2 have different "on" to "off" edge locations, the integrated light is different between TX1 pixels and TX2 pixels.

[0219] Fig.18B Different embodiments of timing for dual pixel sensitivity are shown. In this case, the light pulse is modulated twice (pulse duration and / or amplitude). The TX1 pixel integrates the P1 pulse, and the TX2 pixel integrates the P1+P2 pulse. Separating the global TX signal can be done in a variety of ways, including distinguishing a TX row from each row, and sending multiple TX rows per row, where each TX row addresses a different group of pixels.

[0220] FIG. 19A to FIG. 19C The use of pulsed and / or synchronized or constant white light emission and corresponding color sensors is shown. Fig.19A As can be seen in FIG. 1 , the white light emitter can be configured to emit a light beam during the blanking period of the corresponding sensor to provide a controlled light source in a controlled light environment. The light source can emit a light beam of constant magnitude and vary the duration of the pulse, such as Fig.19A Alternatively, the pulse may be kept constant while varying the magnitude to achieve correctly exposed data, such as Fig.19B shown. Fig.19C Shown in is a graphical representation of a constant light source that can be modulated with a varying current controlled by and synchronized with a sensor.

[0221] In one embodiment, white light or multi-spectral light is emitted as pulses to provide data for use within the system (in FIG. 19A to FIG. 19C ). White light emission combined with partitioning of the electromagnetic spectrum can be used to emphasize and de-emphasize certain aspects within a scene. Such an embodiment may use the following pulse patterns:

[0222] i. Green pulse;

[0223] ii. Red pulse;

[0224] iii. Blue pulse;

[0225] iv. Hyperspectral pulses;

[0226] v.Green pulse;

[0227] vi.Red pulse;

[0228] vii. Blue pulse;

[0229] viii. Hyperspectral pulses;

[0230] ix. White light (multi-spectrum) pulses;

[0231] x.(repeat)

[0232] Any system using an image sensor cycle at least twice faster than white light cycle is intended to fall within the scope of the present disclosure.It should be understood that any combination of partitions of the electromagnetic spectrum are contemplated herein, whether from the visible or invisible spectrum of the full electromagnetic spectrum.

[0233] FIG. 20A to FIG. 20C Each shows a light source 2000 having multiple emitters. The emitters include a first emitter 2002, a second reflector 2004, and a third emitter 2006. Additional emitters may be included, as discussed further below. The emitters 2002, 2004, and 2006 may include one or more laser emitters that emit light having different wavelengths. For example, the first emitter 2002 may emit a wavelength consistent with a blue laser, the second emitter 2004 may emit a wavelength consistent with a green laser, and the third emitter 2006 may emit a wavelength consistent with a red laser. For example, the first emitter 2002 may include one or more blue lasers, the second emitter 2004 may include one or more green lasers, and the third emitter 2006 may include one or more red lasers. The lasers 2002, 2004, 2006 emit laser beams toward a collection area 2008, which may be a waveguide, a lens, or a device for collecting light and / or transmitting light to a waveguide (such as Figure 2 The jumper waveguide 206 or the intracavity waveguide 210) provides a location for other optical components of the light.

[0234] In one specific implementation, the emitters 2002, 2004, and 2006 emit hyperspectral wavelengths of electromagnetic radiation. Certain hyperspectral wavelengths can penetrate tissue and enable a medical practitioner to "see through" tissue in the foreground to identify chemical processes, structures, compounds, biological processes, etc., located behind the tissue in the foreground. Hyperspectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc., which are known to have specific spectral responses.

[0235] In a specific implementation where a reagent or dye has been administered to a patient that aids in identifying certain tissues, structures, chemical reactions, biological processes, etc., the emitters 2002, 2004, and 2006 may emit wavelengths for causing the reagent or dye to fluoresce. Such wavelengths may be determined based on the reagent or dye administered to the patient. In such embodiments, the emitters may need to be highly accurate in order to emit the desired wavelengths to cause certain reagents or dyes to fluoresce or activate.

[0236] In one specific implementation, the transmitters 2002, 2004, and 2006 transmit laser scan patterns for mapping the topology of a scene and / or for calculating the size and distance between objects in the scene. In one embodiment, the endoscopic imaging system is used in conjunction with multiple tools such as scalpels, retractors, clamps, etc. In such embodiments, each of the transmitters 2002, 2004, and 2006 can emit a laser scan pattern such that the laser scan pattern is projected onto each tool individually. In such embodiments, the laser scan data of each of the tools can be analyzed to identify the distance between the tool and other objects in the scene.

[0237] exist Fig. 20B In an embodiment, emitters 2002, 2004, 2006 each deliver laser light to collection area 2008 at different angles. Changes in angle can result in changes in the location of the electromagnetic energy in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at collection area 2008, the changing angles can result in different amounts of light entering different fibers. For example, the angles can result in intensity variations across collection area 2008. In addition, light from different emitters may not be mixed evenly, so some fibers may receive different amounts of different colors of light. Variations in the color or intensity of light in different fibers can result in non-optimal illumination of the scene. For example, variations in the delivered light or light intensity can result in both the scene and the captured image.

[0238] In one embodiment, an intermediate optical element may be placed between the fiber bundle and the transmitter 2002, 2004, 2006 to mix light of different colors (wavelengths) before entering the fiber or other waveguide. Exemplary intermediate optical elements include diffusers, mixing rods, one or more lenses, or other optical components for mixing light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. This mixing can result in the same color in each fiber, but, in some embodiments, can still result in different total brightness delivered to different fibers. In one embodiment, the intermediate optical element can also spread or evenly spread the light over the collection area so that each fiber carries the same total amount of light (e.g., the light may be spread out in a top hat profile). Diffusers or mixing rods can cause light loss.

[0239] Although the collection area in 2008 Fig. 20A 2008 is represented as a physical component, but the collection region 2008 may simply be the region that delivers light from emitters 2002, 2004, and 2006. In some cases, the collection region 2008 may include optical components such as diffusers, mixing rods, lenses, or any other intervening optical components between the emitters 2002, 2004, 2006 and the output waveguides.

[0240] Fig. 20C An embodiment of a light source 2000 is shown having emitters 2002, 2004, 2006 providing light to a collection area 2008 at the same or substantially the same angle. The light is provided at an angle substantially perpendicular to the collection area 2008. The light source 2000 includes a plurality of dichroic mirrors, including a first dichroic mirror 2010, a second dichroic mirror 2012, and a third dichroic mirror 2014. The dichroic mirrors 2010, 2012, 2014 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 2014 may reflect blue laser light provided by a third emitter, while being transparent to red and green light provided by each of the first and second emitters 2002, 2004. The second dichroic mirror 2012 may be transparent to red light from the first emitter 2002, but reflective to green light from the second emitter 2004. If other colors or wavelengths are included, the dichroic mirrors may be selected to reflect light corresponding to at least one emitter and be transparent to other emitters. For example, the third dichroic mirror 2014 reflects light from the third emitter 2006, but is transparent to emitters "behind" it, such as the first emitter 2002 and the second emitter 2004. In embodiments where there are dozens or hundreds of emitters, each dichroic mirror may reflect the corresponding emitter and the emitter in front of it, while being transparent to the emitter behind it. This may allow dozens or hundreds of emitters to emit electromagnetic energy to the collection area 2008 at substantially the same angle.

[0241] Because these dichroic mirrors allow other wavelengths to be transmitted or passed through, each of these wavelengths can arrive at the collection area 2008 from the same angle and / or with the same center point or focus. Providing light from the same angle and / or the same focus / center point can significantly improve reception and color mixing at the collection area 2008. For example, a particular fiber can receive different colors in the same proportions as they are transmitted / reflected by the emitters 2002, 2004, 2006 and the mirrors 2010, 2012, 2014. Fig. 20B Compared to the embodiment of the present invention, the light mixing can be significantly improved at the collection area. In one embodiment, any optical component discussed herein can be used at the collection area 2008 to collect light before providing it to the fiber or fiber bundle.

[0242] Fig. 20C An embodiment of a light source 2000 having emitters 2002, 2004, 2006 that also provide light to a collection area 2008 at the same or approximately the same angle is shown. For example, the light incident on the collection area 2008 is offset from the vertical. Angle 2016 indicates the angle of offset from the vertical. In one embodiment, the laser emitters 2002, 2004, 2006 may have a Gaussian cross-sectional intensity profile. As previously described, the improved distribution of light energy between fibers can be achieved by forming a flatter or top-hat-shaped intensity profile. In one embodiment, as the angle 2016 increases, the intensity across the collection area 2008 approaches a top-hat profile. For example, by increasing the angle 2016 until the profile is sufficiently flat, the top-hat profile can even approximate a non-flat output beam. The top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intermediate optical components between the emitters 2002, 2004, 2006 and the output waveguide, fiber or fiber bundle.

[0243] Fig.21 21 is a schematic diagram showing a single optical fiber 2102 outputted via a diffuser 2104 at the output. In one embodiment, the optical fiber 2102 has a diameter of 500 microns, a numerical aperture of 0.65, and emits a light cone 2106 of about 70 or 80 degrees without the diffuser 2104. With the diffuser 2104, the light cone 2106 may have an angle of about 110 or 120 degrees. The light cone 2106 may be the bulk of where all the light arrives and is evenly distributed. The diffuser 2104 may allow for a more even distribution of the electromagnetic energy of the scene observed by the image sensor.

[0244] In one embodiment, the intracavity waveguide 210 comprises a single plastic or glass optical fiber of about 500 microns. Plastic fibers are relatively low in cost, but their width may allow the fiber to carry a sufficient amount of light to the scene through coupling, diffusion, or other losses. For example, a smaller fiber may not be able to carry as much light or power as a larger fiber. The intracavity waveguide 210 may comprise a single or multiple optical fibers. The intracavity waveguide 210 may receive light directly from a light source or via a jumper waveguide. A diffuser may be used to widen the light output 206 to obtain a desired field of view for an image sensor 214 or other optical component.

[0245] Although in FIG. 20A to FIG. 20C Three emitters are shown, but in some embodiments, a number ranging from one to hundreds or more emitters may be used. The emitters may have different wavelengths or spectra of light that they emit, and these lights may be used to continuously cover the desired portion of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra). The emitters may be configured to be able to emit visible light such as red, green, and blue light, and may also be configured to be able to emit hyperspectral emissions of electromagnetic radiation, fluorescence excitation wavelengths for causing agents to fluoresce, and / or laser mapping patterns for calculating parameters and distances between objects in a scene.

[0246] Fig. 22 A portion of the electromagnetic spectrum 2200 is shown divided into twenty different sub-spectra. The number of sub-spectra is exemplary only. In at least one embodiment, the spectrum 2200 can be divided into hundreds of sub-spectra, each with a small band. The spectrum can extend from the infrared spectrum 2202, through the visible spectrum 2204, and into the ultraviolet spectrum 2206. The sub-spectra each have a band 2208 covering a portion of the spectrum 2200. Each band can be defined by an upper wavelength and a lower wavelength.

[0247] Hyperspectral imaging includes imaging information from across the electromagnetic spectrum 2200. A hyperspectral pulse of electromagnetic radiation may include multiple sub-pulses spanning one or more portions of the electromagnetic spectrum 2200 or the entire electromagnetic spectrum 2200. A hyperspectral pulse of electromagnetic radiation may include a single wavelength partition of the electromagnetic radiation. The resulting hyperspectral exposure frame includes information sensed by the pixel array after the hyperspectral pulse of electromagnetic radiation. Thus, the hyperspectral exposure frame may include data for any suitable partition of the electromagnetic spectrum 2200, and may include multiple exposure frames for multiple partitions of the electromagnetic spectrum 2200. In one embodiment, the hyperspectral exposure frame includes multiple hyperspectral exposure frames such that the combined hyperspectral exposure frame includes data for the entire electromagnetic spectrum 2200.

[0248] In one embodiment, at least one emitter (such as a laser emitter) is included in a light source (such as light source 202, 2000) to provide complete and continuous coverage of the entire spectrum 2200. For example, a light source for providing coverage of the sub-spectrum shown may include at least 20 different emitters, at least one emitter for each sub-spectrum. In one embodiment, each emitter covers a spectrum covering 40 nanometers. For example, one emitter may emit light in a band from 500nm to 540nm, while another emitter may emit light in a band from 540nm to 580nm. In another embodiment, the emitters may cover bands of other sizes, depending on the type of available emitters or imaging needs. For example, a plurality of emitters may include a first emitter covering a band from 500nm to 540nm, a second emitter covering a band from 540nm to 640nm, and a third emitter covering a band from 640nm to 650nm. Each emitter may cover a different segment of the electromagnetic spectrum ranging from far infrared, mid-infrared, near infrared, visible light, near ultraviolet, and / or far ultraviolet. In some cases, multiple emitters of the same type or wavelength may be included to provide sufficient output power for imaging. The number of emitters required for a particular wavelength band may depend on the sensitivity of the monochromatic sensor to the wavelength band and / or the power output capability of the emitters in that wavelength band.

[0249] The bandwidth and coverage provided by the emitter can be selected to provide any desired combination of spectra. For example, continuous coverage of spectra using very small bandwidths (e.g., 10 nm or less) can allow highly selective hyperspectral and / or fluorescence imaging. The bandwidth can allow selective emission of excitation wavelengths of one or more specific fluorescent agents. In addition, the bandwidth can allow selective emission of certain portions of hyperspectral electromagnetic radiation for identification of specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths come from emitters that can be selectively activated, extreme flexibility in causing one or more specific fluorescent agents to fluoresce during an inspection can be achieved. In addition, extreme flexibility in identifying one or more objects or processes through hyperspectral imaging can be achieved. Therefore, more fluorescence and / or hyperspectral information can be achieved in less time and within a single inspection, which would otherwise require multiple inspections, delays due to the application of dyes or stains, etc.

[0250] Fig.232300 is a schematic diagram showing the timing diagram 2300 for generating the emission and readout of the image. The solid line represents the readout (peak 2302) and blanking period (valley) for capturing a series of exposure frames 2304-2314. A series of exposure frames 2304-2314 may include a series of repeated exposure frames that can be used to generate laser scanning, hyperspectral and / or fluorescence data that can be stacked on an RGB video stream. In one embodiment, a single image frame includes information from a plurality of exposure frames, wherein one exposure frame includes red image data, another exposure frame includes green image data, and another exposure frame includes blue image data. In addition, a single image frame may include one or more of hyperspectral image data, fluorescence image data, and laser scanning data. The plurality of exposure frames are combined to produce a single image frame. A single image frame is an RGB image with hyperspectral imaging data. The series of exposure frames includes a first exposure frame 2304, a second exposure frame 2306, a third exposure frame 2308, a fourth exposure frame 2310, a fifth exposure frame 2312, and an Nth exposure frame 2326.

[0251] In addition, hyperspectral image data, fluorescent image data, and laser scanning data can be used in combination to identify key tissues or structures and also measure the size of those key tissues or structures. For example, hyperspectral image data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, cancerous tissues, etc. The location and identification of the key structure can be received from the corresponding system and can also be used to generate the topology of the key structure using laser scanning data. For example, the corresponding system determines the location of a cancerous tumor based on hyperspectral imaging data. Because the location of the cancerous tumor is known based on the hyperspectral imaging data, the topology and distance of the cancerous tumor can be calculated based on the laser scanning data. This example can also be applicable when identifying a cancerous tumor or other structure based on fluorescent imaging data.

[0252] In one embodiment, each exposure frame is generated based on at least one pulse of electromagnetic energy. The electromagnetic energy pulses are reflected and detected by the image sensor and then read out in a subsequent readout (2302). Thus, each blanking period and readout results in an exposure frame for a particular spectrum of electromagnetic energy. For example, a first exposure frame 2304 may be generated based on the spectrum of the first one or more pulses 2316, a second exposure frame 2306 may be generated based on the spectrum of the second one or more pulses 2318, a third exposure frame 2308 may be generated based on the spectrum of the third one or more pulses 2320, a fourth exposure frame 2310 may be generated based on the spectrum of the fourth one or more pulses 2322, a fifth exposure frame 2312 may be generated based on the spectrum of the fifth one or more pulses 2324, and an Nth exposure frame 2326 may be generated based on the spectrum of the Nth one or more pulses 2326.

[0253] Pulses 2316 to 2326 may include energy from a single emitter or from a combination of two or more emitters. For example, a spectrum included in a single readout cycle or within multiple exposure frames 2304-2314 may be selected for the desired inspection or detection of a specific tissue or condition. According to one embodiment, one or more pulses may include visible spectrum light for generating an RGB or black and white image, while emitting one or more additional pulses to sense the spectral response to the hyperspectral wavelength of electromagnetic radiation. For example, pulse 2316 may include red light, pulse 2318 may include blue light, and pulse 2320 may include green light, while the remaining pulses 2322-2326 may include wavelengths and spectra for detecting a specific tissue type, causing an agent to fluoresce, and / or mapping the topology of a scene. As another example, the pulses of a single readout cycle include spectra (e.g., different fragments of the electromagnetic spectrum) generated by multiple different emitters that can be used to detect a specific tissue type. For example, if a combination of wavelengths causes a pixel to have a value that exceeds or is below a threshold, the pixel may be classified as corresponding to a specific type of tissue. Each frame can also be used to narrow down the type of tissue present at that pixel (e.g., and every pixel in the image) to provide a very specific classification of the tissue and / or the state of the tissue (disease / healthy) based on the spectral response of the tissue and / or the presence or absence of fluorescent agents at the tissue.

[0254] Multiple frames 2304 to 2314 are shown with readout periods of varying lengths and pulses of varying lengths or intensities. Blanking periods, pulse lengths or intensities, etc. may be selected based on the sensitivity of the monochromatic sensor to a particular wavelength, the power output capability of the transmitter, and / or the carrying capacity of the waveguide.

[0255] In one embodiment, dual image sensors may be used to obtain a three-dimensional image or video feed.Three-dimensional examination may allow for an improved understanding of the three-dimensional structure of the examination region and mapping of different tissue or material types within the region.

[0256] In an exemplary implementation, a fluorescent agent is provided to a patient, and the fluorescent agent is configured to be able to attach to cancer cells. It is known that the fluorescent agent fluoresces when irradiated by a specific partition of electromagnetic radiation. The relaxation wavelength of the fluorescent agent is also known. In an exemplary implementation, the patient is imaged using an endoscopic imaging system as discussed herein. The endoscopic imaging system pulses partitions of light of red, green, and blue wavelengths to generate an RGB video stream of the inside of the patient's body. In addition, the endoscopic imaging system pulses the electromagnetic radiation excitation wavelength of the fluorescent agent applied to the patient. In the example, the patient has cancer cells, and the fluorescent agent attaches to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength of the fluorescent agent, the fluorescent agent will fluoresce and emit a relaxation wavelength. If cancer cells are present in a scene imaged by the endoscopic imaging system, the fluorescent agent will also be present in the scene and emit its relaxation wavelength after fluorescing due to the emission of the excitation wavelength. The endoscopic imaging system senses the relaxation wavelength of the fluorescent agent, thereby sensing the presence of the fluorescent agent in the scene. Because the fluorescent agent is known to attach to cancer cells, the presence of the fluorescent agent also indicates the presence of cancer cells in the scene. The endoscopic imaging system thereby identifies the location of cancer cells within the scene. The endoscopic imaging system may also emit a laser scanning pulse scheme for generating the topology of the scene and calculating the size of objects within the scene. The location of the cancer cells (as identified by the fluorescence imaging data) may be combined with the topological and size information calculated based on the laser scanning data. Thus, the precise location, size, dimensions, and topology of the cancer cells may be identified. This information may be provided to a medical practitioner to aid in the removal of cancer cells. Additionally, this information may be provided to a robotic surgical system to enable the surgical system to remove cancer cells.

[0257] In another exemplary implementation, an endoscopic imaging system is used to image a patient to identify quantitative diagnostic information about the patient's tissue pathology. In this example, the patient is suspected or known to have a disease that can be tracked using hyperspectral imaging to observe the progression of the disease in the patient's tissue. The endoscopic imaging system pulses partitions of red, green, and blue wavelengths of light to generate an RGB video stream inside the patient's body. In addition, the endoscopic imaging system pulses light of one or more hyperspectral wavelengths, which allows the system to "see through" some tissues and generate imaging of tissues affected by the disease. The endoscopic imaging system senses the reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue, thereby identifying the location of the diseased tissue in the patient's body. The endoscopic imaging system can also emit a laser scanning pulse scheme for generating the topology of the scene and calculating the size of the object in the scene. The location of the diseased tissue (identified by the hyperspectral imaging data) can be combined with the topological and size information calculated using the laser scanning data. Therefore, the precise location, size, size, and topology of the diseased tissue can be identified. This information can be provided to a practicing physician to help remove, image, or study the diseased tissue. Additionally, this information may be provided to a robotic surgical system to enable the surgical system to resect diseased tissue.

[0258] Fig.24A and Fig. 24B A perspective view and a side view, respectively, of an implementation of a monolithic sensor 2400 having multiple pixel arrays for producing a three-dimensional image in accordance with the teachings and principles of the present disclosure are shown. Such an implementation may be desirable for three-dimensional image capture, where the two pixel arrays 2402 and 2404 may be offset during use. In another implementation, the first pixel array 2402 and the second pixel array 2404 may be dedicated to receiving electromagnetic radiation of a predetermined wavelength range, where the first pixel array is dedicated to electromagnetic radiation of a different wavelength range than the second pixel array.

[0259] Fig.25A and Fig.25BA perspective view and a side view of a specific implementation of an imaging sensor 2500 built on multiple substrates are shown respectively. As shown, multiple pixel columns 2504 forming the pixel array are located on a first substrate 2502 and multiple circuit columns 2508 are located on a second substrate 2506. The figure also shows the electrical connection and communication between a pixel column and its associated or corresponding circuit column. In a specific implementation, the image sensor may have a pixel array separated from all or most of the supporting circuits, and it may be otherwise manufactured so that its pixel array and supporting circuits are on a single, monolithic substrate / chip. The present disclosure may use at least two substrates / chips, which will be stacked together using three-dimensional stacking technology. The first 2502 of the two substrates / chips may be processed using an image CMOS process. The first substrate / chip 2502 may consist of only a pixel array, or may consist of a pixel array surrounded by a limited circuit. The second or subsequent substrate / chip 2506 may be processed using any process, rather than necessarily from an image CMOS process. The second substrate / chip 2506 may be, but is not limited to, a high density digital process for integrating various and multiple functions into a very limited space or area on the substrate / chip, or a mixed mode or analog process for integrating, for example, precise analog functions, or an RF process for implementing wireless capabilities, or a MEMS (micro-electro-mechanical system) for integrating MEMS devices. The image CMOS substrate / chip 2502 may be stacked with the second or subsequent substrate / chip 2506 using any three-dimensional technology. The second substrate / chip 2506 may support a substantial portion or a large portion of the circuitry that would otherwise be implemented in the first image CMOS chip 2502 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increase the overall system area while keeping the pixel array size constant and optimized as much as possible. The electrical connection between the two substrates / chips may be accomplished through interconnects, which may be bonding wires, bumps, and / or TSVs (through silicon vias).

[0260] Fig.26A and Fig.26B A perspective view and a side view of a specific implementation of an imaging sensor 2600 having multiple pixel arrays for producing a three-dimensional image are shown, respectively. The three-dimensional image sensor can be constructed on multiple substrates and can include multiple pixel arrays and other associated circuits, wherein multiple pixel columns 2604a forming a first pixel array and multiple pixel columns 2604b forming a second pixel array are located on respective substrates 2602a and 2602b, respectively, and multiple circuit columns 2608a and 2608b are located on a separate substrate 2606. Electrical connections and communications between pixel columns and associated or corresponding circuit columns are also shown.

[0261] The multiple pixel arrays can sense information at the same time, and the information from the multiple pixel arrays can be combined to generate a three-dimensional image. In one embodiment, the endoscopic imaging system includes two or more pixel arrays that can be deployed to generate three-dimensional imaging. The endoscopic imaging system may include a transmitter for emitting electromagnetic radiation pulses during the blanking period of the pixel array. The pixel arrays can be synchronized so that for the two or more pixel arrays, optical black pixels are read at the same time (i.e., a blanking period occurs). The transmitter can emit electromagnetic radiation pulses for charging each of the two or more pixel arrays. The two or more pixel arrays can read their corresponding charged pixels at the same time, so that the readout cycles of the two or more pixel arrays occur simultaneously or approximately simultaneously. In one embodiment, the endoscopic imaging system includes a plurality of transmitters, each transmitter being individually synchronized with one or more pixel arrays in the plurality of pixel arrays. Information from the plurality of pixel arrays can be combined to generate a three-dimensional image frame and a video stream.

[0262] It should be understood that the teachings and principles of the present disclosure can be used for reusable device platforms, limited-use device platforms, reconfigurable device platforms, or single-use / disposable device platforms without departing from the scope of the present disclosure. It should be understood that in a reusable device platform, the end user is responsible for the cleaning and disinfection of the device. In a limited-use device platform, the device can be used a specified number of times before becoming inoperable. A typical new device has been sterilized before delivery and should be cleaned and disinfected by the end user before other uses if it is to be used for other purposes. In a reconfigurable device platform, a third party can reprocess the device (e.g., clean, package, and disinfect) a single-use device for additional use at a lower cost than a new unit. In a single-use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0263] Example

[0264] The following examples relate to preferred features of further embodiments:

[0265] Embodiment 1 is a system for imaging in a light-deficient environment. The system includes an emitter and an image sensor, the emitter is used to emit electromagnetic radiation pulses, and the image sensor includes a pixel array for sensing reflected electromagnetic radiation. The system includes a controller, the controller includes a processor in electrical communication with the image sensor and the emitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor. The system is such that at least a portion of the electromagnetic radiation pulses emitted by the emitter includes one or more of the following: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm, electromagnetic radiation having a wavelength from about 565 nm to about 585 nm, or electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm.

[0266] Embodiment 2 is a system according to embodiment 1, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to an electromagnetic radiation pulse emitted by the transmitter, and a data set corresponding in time to each electromagnetic radiation pulse is generated to generate a plurality of data sets corresponding to the plurality of exposure frames.

[0267] Embodiment 3 is a system according to any one of Embodiments 1 to 2, wherein the image sensor is configured to be capable of reading each data set corresponding to each of the multiple exposure frames from the pixel array during a readout frame, the readout frame corresponding to the duration required to read each pixel in the pixel array, and wherein at least the first exposure frame overlaps with the readout frame.

[0268] Embodiment 4 is the system of any one of embodiments 1 to 3, wherein the plurality of exposure frames and the plurality of data sets are combined to form an image frame.

[0269] Embodiment 5 is a system according to any one of embodiments 1 to 4, wherein at least a portion of the electromagnetic radiation pulses emitted by the transmitter includes electromagnetic radiation of a green wavelength, electromagnetic radiation of a red wavelength, and electromagnetic radiation of a blue wavelength.

[0270] Embodiment 6 is a system according to any one of embodiments 1 to 5, wherein the transmitter is configured to be capable of transmitting a plurality of electromagnetic radiation sub-pulses having sub-durations shorter than the pulse duration during a pulse duration.

[0271] Embodiment 7 is a system according to any one of embodiments 1 to 6, wherein one or more of the electromagnetic radiation pulses emitted by the transmitter includes electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

[0272] Embodiment 8 is a system according to any one of embodiments 1 to 7, wherein pulses in the electromagnetic radiation pulses emitted by the transmitter produce exposure frames generated by the image sensor, and wherein the system also includes a display for displaying two or more exposure frames as a stacked image.

[0273] Embodiment 9 is a system according to any one of embodiments 1 to 8, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter produces a hyperspectral exposure frame generated by the image sensor, and wherein the controller is also configured to provide the hyperspectral exposure frame to a corresponding system, and the corresponding system determines the position of the key tissue structure within the scene based on the hyperspectral exposure frame.

[0274] Embodiment 10 is a system according to any one of embodiments 1 to 9, wherein the controller is further configured to be capable of: receiving the position of the key tissue structure from the corresponding system; generating an overlay frame including the position of the key tissue structure; and combining the overlay frame with a color image frame depicting the scene to indicate the position of the key tissue structure within the scene.

[0275] Embodiment 11 is a system according to any one of embodiments 1 to 10, wherein the blanking period of the image sensor corresponds to the time between the readout of the last row of the pixel array and the start of the next readout cycle of the pixel array.

[0276] Embodiment 12 is a system according to any one of embodiments 1 to 11, wherein the controller is further configured to be able to adjust the sequence of electromagnetic radiation pulses emitted by the emitter based on a threshold, wherein the threshold determines the appropriate illumination of a scene in a light-deficient environment.

[0277] Embodiment 13 is a system according to any one of embodiments 1 to 12, wherein two or more pulses of electromagnetic radiation emitted by the transmitter produce two or more instances of reflected electromagnetic radiation sensed by the pixel array to generate an image frame.

[0278] Embodiment 14 is a system according to any one of embodiments 1 to 13, wherein the image frame is assigned a visible color for use on a display, wherein the visible color is 8 bits or 16 bits or n bits.

[0279] Embodiment 15 is a system according to any one of embodiments 1 to 14, wherein at least one of the electromagnetic radiation pulses emitted by the transmitter produces a hyperspectral exposure frame generated by the image sensor, and wherein the controller is also configured to provide the hyperspectral exposure frame to a corresponding system, and the corresponding system identifies one or more key structures in the body based on the hyperspectral exposure frame.

[0280] Embodiment 16 is a system according to any one of embodiments 1 to 15, wherein the one or more critical structures in the body include one or more of nerves, ureters, blood vessels, arteries, blood flow, or tumors.

[0281] Embodiment 17 is a system according to any one of embodiments 1 to 16, wherein the image sensor includes a first image sensor and a second image sensor, so that the image sensor can generate a three-dimensional image.

[0282] Embodiment 18 is a system according to any one of embodiments 1 to 17, wherein the transmitter is configured to repeatedly transmit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, each exposure frame corresponding to an electromagnetic radiation pulse.

[0283] Embodiment 19 is a system according to any one of embodiments 1 to 18, wherein the reflected electromagnetic radiation sensed by the image sensor produces multiple exposure frames that are combined to generate an image frame, wherein the image frame includes: a luminance frame including luminance data, a chrominance frame including chrominance data, and a hyperspectral frame including hyperspectral data.

[0284] Embodiment 20 is a system according to any one of embodiments 1 to 19, wherein the pixel array includes pixels having multiple pixel sensitivities, wherein the multiple pixel sensitivities include long exposure and short exposure.

[0285] Embodiment 21 is a system according to any one of embodiments 1 to 20, wherein the image sensor is configured to generate a hyperspectral frame of long exposure pixel data and short exposure pixel data.

[0286] Embodiment 22 is a system according to any one of embodiments 1 to 21, wherein the transmitter includes multiple light sources, each of which emits pulses of a portion of the electromagnetic spectrum.

[0287] Embodiment 23 is a system according to any one of embodiments 1 to 22, wherein the electromagnetic radiation pulses are emitted in a pattern of electromagnetic radiation of different wavelengths, and wherein the emitter repeats the pattern of electromagnetic radiation of different wavelengths.

[0288] Embodiment 24 is a system according to any one of embodiments 1 to 23, wherein the electromagnetic radiation pulses include: electromagnetic radiation of a red wavelength; electromagnetic radiation of a green wavelength; electromagnetic radiation of a blue wavelength; electromagnetic radiation having a wavelength from about 513nm to about 545nm; electromagnetic radiation having a wavelength from about 565nm to about 585nm; and electromagnetic radiation having a wavelength from about 900nm to about 1000nm.

[0289] Embodiment 25 is a system according to any one of embodiments 1 to 24, wherein at least a portion of the electromagnetic radiation pulses include red wavelengths, green wavelengths, blue wavelengths, and hyperspectral wavelengths, so that the reflected electromagnetic radiation corresponding to each of the red wavelengths, the green wavelengths, the blue wavelengths, and the hyperspectral wavelengths sensed by the pixel array can be processed to generate a red, green, and blue (RGB) image, and the red, green, and blue (RGB) image includes a stack of hyperspectral imaging data.

[0290] It should be appreciated that the various features disclosed herein provide significant advantages and advances in the art. The following claims are examples of some of those features.

[0291] In the above-mentioned detailed description of the present disclosure, for the purpose of simplifying the present disclosure, the various features of the present disclosure are concentrated in a single embodiment. The method of the present disclosure should not be understood as embodying such an intention: the disclosure protected by the claims requires more features than those explicitly listed in each claim. On the contrary, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0292] It should be understood that any features of the above-described arrangements, embodiments and implementations may be combined in a single embodiment comprising a combination of features obtained from any of the disclosed arrangements, embodiments and implementations.

[0293] It should be understood that the above configuration is only an exemplary application of the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, those skilled in the art may design many modifications and alternative configurations, and the appended claims are intended to cover these modifications and configurations.

[0294] Thus, while the present disclosure is shown in the drawings and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications may be made without departing from the principles and concepts described herein, including but not limited to changes in size, material, shape, form, function and mode of operation, assembly and use.

[0295] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0296] For the purpose of illustration and description, the above-mentioned specific embodiments have been provided. These specific embodiments are not intended to be exhaustive or to limit the present disclosure to the disclosed specific forms. According to the above-mentioned teachings, many modifications and changes can be made to the present disclosure. In addition, it should be noted that any or all of the aforementioned alternative specific implementations can be used in any desired combination to form other mixed specific implementations of the present disclosure.

[0297] In addition, although specific implementations of the present disclosure have been described and illustrated, the present disclosure is not limited to the specific forms or arrangements of parts as described and illustrated. The scope of the present disclosure will be defined by the claims appended hereto, any future claims filed here and in different applications, and their equivalents.

Claims

1. A system comprising: An emitter for emitting electromagnetic radiation pulses, wherein the emitter comprises a plurality of electromagnetic sources for hyperspectral imaging, wherein the plurality of electromagnetic radiation pulses emitted by the plurality of electromagnetic sources comprises hyperspectral emission, the hyperspectral emission comprising wavelengths from 900 nm to 1000 nm and one or more of: Wavelengths from 513nm to 545nm; and Wavelengths from 565nm to 585nm; an image sensor comprising an array of pixels for sensing reflected electromagnetic radiation; and a controller including a processor in electrical communication with the image sensor and the transmitter; Wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor.

2. The system according to claim 1, wherein: The image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to an electromagnetic radiation pulse emitted by the emitter, and to generate a data set corresponding in time to each electromagnetic radiation pulse to generate a plurality of data sets corresponding to the plurality of exposure frames.

3. The system according to claim 2, wherein: The image sensor is configured to be capable of sensing each of the multiple exposure frames during a readout cycle of the pixel array, wherein the readout cycle is a duration for reading valid pixels in the pixel array, and wherein a portion of the blanking period overlaps with a portion of a next subsequent readout cycle.

4. The system according to claim 3, wherein: The plurality of exposure frames are combined to form an image frame.

5. The system according to claim 1, wherein: At least a portion of the electromagnetic radiation pulses emitted by the emitter includes green wavelength electromagnetic radiation, red wavelength electromagnetic radiation, and blue wavelength electromagnetic radiation.

6. The system according to claim 1, wherein: The transmitter is configured to be capable of transmitting, during a pulse duration, a plurality of electromagnetic radiation sub-pulses having sub-durations shorter than the pulse duration.

7. The system according to claim 1, wherein: One or more of the electromagnetic radiation pulses emitted by the emitter comprises electromagnetic radiation emitted simultaneously as a single pulse or a single sub-pulse at two or more wavelengths.

8. The system according to claim 1, wherein: At least one of the electromagnetic radiation pulses emitted by the emitter produces an exposure frame generated by the image sensor, wherein the system further comprises a display for displaying two or more exposure frames as an image frame.

9. The system according to claim 1, wherein: At least a portion of the electromagnetic radiation pulses emitted by the emitter produces a hyperspectral exposure frame generated by the image sensor, and wherein the controller is further configured to provide the hyperspectral exposure frame to a corresponding system that determines the location of a critical tissue structure within a scene based on the hyperspectral exposure frame.

10. The system according to claim 9, wherein: The controller is also configured to: receiving the location of the key organizational structure from the corresponding system; generating a stacked frame including the location of the critical tissue structure; as well as The overlay frame is combined with a color image frame depicting the scene to indicate the location of the key tissue structure within the scene.

11. The system according to claim 1, wherein: The blanking period of the image sensor corresponds to the time between a readout of a last row of valid pixels in the pixel array and the start of a next subsequent readout of valid pixels in the pixel array.

12. The system of claim 1, wherein: The controller is also configured to be capable of adjusting the sequence of electromagnetic radiation pulses emitted by the emitter based on a threshold value, wherein the threshold value determines appropriate illumination of a scene in a light-deficient environment.

13. The system of claim 1, wherein: Two or more pulses of electromagnetic radiation emitted by the emitter produce two or more instances of reflected electromagnetic radiation, which are sensed by the pixel array to generate two or more exposure frames, which are combined to form an image frame.

14. The system according to claim 13, wherein: The image frame is assigned visible colors for use on a display, where the visible colors are 8 bits or 16 bits or n bits.

15. The system of claim 1, wherein: At least one of the plurality of electromagnetic radiation pulses emitted by the emitter produces a hyperspectral exposure frame generated by the image sensor, and wherein the controller is further configured to provide the hyperspectral exposure frame to a corresponding system that identifies one or more key structures in a body based on the hyperspectral exposure frame.

16. The system of claim 15, wherein: The one or more critical structures in the body include one or more of nerves, ureters, blood vessels, arteries, blood flow, or tumors.

17. The system of claim 1, wherein: The image sensor includes a first image sensor and a second image sensor, such that the image sensor can generate a three-dimensional image.

18. The system of claim 1, wherein: The transmitter is configured to repeatedly transmit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, wherein each of the exposure frames corresponds to an electromagnetic radiation pulse.

19. The system of claim 1, wherein: The reflected electromagnetic radiation sensed by the image sensor produces a plurality of exposure frames that are combined to generate an image frame, wherein the image frame includes a luma frame including luma data, a chroma frame including chroma data, and a hyperspectral frame including hyperspectral data.

20. The system of claim 1, wherein: The pixel array includes pixels having a plurality of pixel sensitivities, wherein the plurality of pixel sensitivities include a long exposure and a short exposure.

21. The system of claim 20, wherein: The image sensor is configured to generate a hyperspectral frame of long exposure pixel data and short exposure pixel data.

22. The system of claim 1, wherein: The plurality of electromagnetic sources of the emitter further includes a red light source for pulsed red light, a green light source for pulsed green light, and a blue light source for pulsed blue light.

23. The system of claim 1, wherein: The pulses of electromagnetic radiation are emitted in a pattern of electromagnetic radiation at different wavelengths, and wherein the emitter repeats the pattern of electromagnetic radiation at different wavelengths.

24. The system of claim 1, wherein: The electromagnetic radiation pulse comprises: electromagnetic radiation of red wavelengths; electromagnetic radiation of green wavelengths; electromagnetic radiation of blue wavelengths; Electromagnetic radiation having a wavelength from 513 nm to 545 nm; Electromagnetic radiation having a wavelength from 565 nm to 585 nm; and Electromagnetic radiation having a wavelength from 900nm to 1000nm.

25. The system of claim 1, wherein: At least a portion of the electromagnetic radiation pulses include red wavelengths, green wavelengths, blue wavelengths, and hyperspectral wavelengths, so that reflected electromagnetic radiation corresponding to each of the red wavelengths, the green wavelengths, the blue wavelengths, and the hyperspectral wavelengths sensed by the pixel array can be processed to generate a red, green, and blue (RGB) image, which includes a stack of hyperspectral imaging data.

Images