Hyperspectral imaging in light deficient environment

Abstract

To provide a system for generating an image in a light deficient environment, a related structure and a method.SOLUTION: An endoscopic imaging system in a light deficient environment includes an imaging device having a tube, one or more image sensors, and a lens assembly including at least one optical element that corresponds to the one or more image sensors. The endoscopic system includes a display for a user to visualize a scene and an image signal processing controller. The endoscopic system includes a light engine having an illumination source generating one or more pulses of electromagnetic radiation and a lumen transmitting one or more pulses of electromagnetic radiation to a distal tip of an endoscope.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Advances in technology have led to advances in medical imaging capabilities. Endoscopes can be used to see inside the body and to examine the inside of organs or cavities of the body. Endoscopes can be used to investigate patient symptoms, confirm diagnoses, or provide medical treatment. Medical endoscopes can be used to view various body tissues and parts, such as the digestive tract, airways, urinary tract, abdominal cavity, etc., through small incisions. Endoscopes can also be used for surgical procedures, such as orthopedic procedures, procedures performed on joints or bones, procedures performed on the nervous system, procedures performed within the abdominal cavity, etc.

[0002] Endoscopes are also used in non-medical fields to view and inspect spaces that may be inaccessible or difficult to see. For example, endoscopes can be used by planners or architects to visualize scale models of proposed buildings or cities. Endoscopes can be used to visualize the interior spaces of complex systems such as computers. Endoscopes can even be used by law enforcement or military personnel to perform surveillance of tight spaces or to inspect explosive devices.

[0003] Among the various applications of endoscopy, it may be beneficial to view a space in color. A digital color image may include at least three layers or "color channels" for each pixel of the image. Each of the color channels measures the intensity and chrominance of light in a spectral band. Typically, a digital color image includes color channels for the red, green, and blue spectral bands of light (this may be referred to as an RGB image). Each of the red, green, and blue color channels includes luminance information for the red, green, or blue spectral bands of light. The luminance information of the separate red, green, and blue layers can be combined to create a digital color image. Because color images are composed of separate layers, digital camera image sensors typically include a color filter array that allows red, green, and blue visible light wavelengths to impinge on selected pixel sensors. Each individual pixel sensor element is sensitive to red, green, or blue wavelengths and returns image data for only those wavelengths. Image data from the entire array of pixel sensors is combined to generate an RGB image.

[0004] In the case of endoscopic imaging for medical diagnosis or medical treatment, it may be beneficial or even necessary to view a body cavity with a color image. For example, when viewing the abdominal cavity of a body using an endoscope, a color image may provide useful information to aid in identifying different organs or tissues within the abdomen, or identifying a particular condition or disease within the space. As discussed above, a digital camera capable of capturing color images may have at least three different types of pixel sensors to capture the red, green, and blue layers of the color image individually. At least three different types of pixel sensors may consume a relatively large physical space (when compared to a pixel array that cannot recognize color), and thus the entire pixel array cannot fit into the small distal end of an endoscope that is inserted into the body. Because a color digital camera may include at least three different types of pixel sensors, the entire pixel array (i.e., the image sensor) is typically located in the handpiece unit of the endoscope, which is held by the endoscope operator and not placed in the body cavity. In such an endoscope, light is transmitted along the length of the endoscope from the handpiece unit to the distal end of the endoscope that is placed in the body cavity. This endoscope configuration has significant limitations: endoscopes having this configuration are delicate and can be easily displaced or damaged when bumped or impacted during normal use. This can significantly degrade the quality of the images produced by the endoscope and can require frequent repair or replacement of the endoscope.

[0005] In some cases, especially in the case of medical imaging or medical procedures, it may be more beneficial to see than to have a color image. A color image reflects what the human eye detects when viewed in an environment. However, the human eye is limited to seeing only visible light and cannot detect other wavelengths of the electromagnetic spectrum. Other wavelengths of the electromagnetic spectrum beyond the "visible light" wavelengths can provide additional information about the environment. One way to detect additional information about the environment beyond what the human eye can detect is hyperspectral imaging. In the case of imaging for medical purposes, hyperspectral imaging can provide a unique view of a body cavity that provides additional information beyond what the human eye or a digital color camera can detect.

[0006] Hyperspectral imaging can be used to distinguish between different materials or objects, and to distinguish between different processes. The human eye sees visible light colors primarily in three bands of wavelengths, sensed as a combination of red, green, and blue. A typical human eye responds to wavelengths of electromagnetic radiation from about 350 nm to about 750 nm, and this band of wavelengths is sometimes referred to as the "visible spectrum." Hyperspectral imaging differs in that it separates the spectrum of light wavelengths into many more bands based on the reflectance of tissues, objects, or materials. This allows hyperspectral imaging to provide information beyond what the human eye can see. Hyperspectral imaging can provide useful information in a variety of applications where it is difficult or impossible to identify certain materials, objects, or processes with the human eye or a computer. Hyperspectral imaging can be particularly effective in certain cases because some objects leave unique "fingerprints" in the electromagnetic spectrum, sometimes referred to as spectral signatures. These spectral signatures allow for the identification of certain materials, objects, or processes that cannot be seen by the human eye or a computer.

[0007] Hyperspectral imaging was originally developed for mining and geology applications. Unlike regular camera images that provide limited information to the human eye, hyperspectral imaging can identify unique minerals based on the spectral signatures of different minerals. Hyperspectral imaging can be useful even when taking aerial images and can provide information about, for example, oil or gas leaks from pipelines or natural wells and their impact on nearby vegetation. This information is gathered based on the spectral signatures of specific materials, objects, or processes that can be identified by hyperspectral imaging.

[0008] However, hyperspectral imaging can be quite complex and require fast computer processing power, sensitive detectors, and large data storage capacity. In addition, when a hyperspectral image is overlaid on a black and white or color image to provide a context to the practitioner, the camera (or cameras) capable of generating the overlaid image may have many different types of pixel sensors sensitive to unique electromagnetic radiation ranges. This may include three separate types of pixel sensors for generating RGB color images by conventional methods, along with additional pixel sensors for generating hyperspectral image data at different wavelengths of the electromagnetic spectrum. This may consume a relatively large physical space and require a large pixel array to ensure that the image resolution is sufficient. For endoscopic imaging, since placing multiple wavelength-sensitive pixel sensors at the distal end of the endoscope within the body cavity would require too much physical space and too large a pixel array, one or more cameras may be placed in the endoscope hand unit or robot unit. This may result in the same disadvantages as mentioned above and may also make the endoscope very delicate, resulting in significant degradation of image quality when bumped or impacted during use of the endoscope. Summary of the Invention [Means for solving the problem]

[0009] The present disclosure relates generally to electromagnetic sensing and sensors that can be applied to endoscopic imaging. The present disclosure also relates to low energy electromagnetic input conditions, as well as low energy electromagnetic throughput conditions. The present disclosure more specifically, but not necessarily generally, relates to systems and related structures, methods, and features for generating images in low light environments, the methods including controlling a light source through duration, intensity, or both, pulsing a content-controlled light source during blanking periods of an image sensor, maximizing blanking periods to allow optimal light, and maintaining color balance.

[0010] The features and advantages of the present disclosure will be set forth in the description which follows, and in part will be obvious from the description, and / or may be learned by the practice of the disclosure without undue experimentation. The features and advantages of the present disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. [Brief description of the drawings]

[0011] Non-limiting and non-exhaustive implementations of the present disclosure are described with reference to the following figures, in which like reference numerals refer to like parts throughout the various views unless otherwise specified. Advantages of the present disclosure will be better understood with reference to the following description and the accompanying drawings. [Figure 1] 1 is a schematic diagram of an embodiment of a system of a pair of sensors and electromagnetic emitters in operation for use in generating images in low light environments, according to one embodiment. [Diagram 2] FIG. 2 is a schematic diagram of complementary system hardware. [Figure 2A] FIG. 2 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Figure 2B] FIG. 2 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Figure 2C]FIG. 2 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Figure 2D] FIG. 2 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Diagram 3] 1 is a graphical representation of the operation of an embodiment of an electromagnetic emitter, according to one embodiment. [Figure 4] 1 is a graphical representation of varying the duration and amplitude of an emitted electromagnetic pulse to provide exposure control, according to one embodiment. [Diagram 5] 5 is a graphical representation of an embodiment of the present disclosure combining the operational cycles of the sensors, electromagnetic emitters, and emitted electromagnetic pulses of FIGS. 2A-4 showing an imaging system in operation, according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of two different processes for recording video frames of full spectrum and split spectrum light over the period t(0) to t(1), according to one embodiment. [Figure 7A] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7B] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7C] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7D] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7E] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 8] FIG. 1 illustrates the tuning of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 9] FIG. 1 illustrates the tuning of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 10] FIG. 1 illustrates the tuning of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 11] FIG. 1 illustrates the tuning of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 12] FIG. 1 illustrates the tuning of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 13] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 14] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 15] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 16] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 17] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 18] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 19]1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 20] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 21] 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 22] FIG. 1 is a schematic diagram of a method and hardware for increasing dynamic range in a closed or limited light environment, according to an embodiment of the present disclosure. [Diagram 23] FIG. 1 is a schematic diagram of a method and hardware for increasing dynamic range in a closed or limited light environment, according to an embodiment of the present disclosure. [Figure 24] FIG. 1 illustrates the effect on the signal-to-noise ratio of color correction of a typical Bayer-based sensor compared to one without color correction. [Diagram 25] FIG. 1 illustrates the chromaticity of three monochrome lasers compared to the sRGB color gamut. [Figure 26] FIG. 1 is a schematic diagram of a method and hardware for increasing dynamic range in a closed or limited light environment, according to an embodiment of the present disclosure. [Figure 27A] FIG. 1 is a schematic diagram of a method and hardware for increasing dynamic range in a closed or limited light environment, according to an embodiment of the present disclosure. [Figure 27B] FIG. 1 is a schematic diagram of a method and hardware for increasing dynamic range in a closed or limited light environment, according to an embodiment of the present disclosure. [Figure 28A] FIG. 13 illustrates the use of white light emission pulsed and / or synchronized with a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 28B] FIG. 13 illustrates the use of white light emission pulsed and / or synchronized with a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 28C]FIG. 13 illustrates the use of white light emission pulsed and / or synchronized with a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 29A] FIG. 1 illustrates an implementation having multiple pixel arrays for generating a three-dimensional image, according to an embodiment of the present disclosure. [Figure 29B] FIG. 1 illustrates an implementation having multiple pixel arrays for generating a three-dimensional image, according to an embodiment of the present disclosure. [Figure 30A] 1A and 1B are perspective and side views, respectively, of an implementation of an image sensor constructed on multiple substrates, with multiple pixel columns forming a pixel array located on a first substrate and multiple circuit columns located on a second substrate, showing electrical connections and communication between one pixel column and its associated or corresponding circuit columns. [Figure 30B] 1A and 1B are perspective and side views, respectively, of an implementation of an image sensor constructed on multiple substrates, with multiple pixel columns forming a pixel array located on a first substrate and multiple circuit columns located on a second substrate, showing electrical connections and communication between one pixel column and its associated or corresponding circuit columns. [Figure 31A] 1A and 1B are perspective and side views, respectively, of an implementation of an imaging sensor having multiple pixel arrays for generating a three-dimensional image, the multiple pixel arrays and image sensor being constructed on multiple substrates. [Figure 31B] 1A and 1B are perspective and side views, respectively, of an implementation of an imaging sensor having multiple pixel arrays for generating a three-dimensional image, the multiple pixel arrays and image sensor being constructed on multiple substrates. [Diagram 32] 1A-1C illustrate embodiments of emitters with various mechanical filter and shutter configurations in accordance with embodiments of the present disclosure. [Diagram 33] 1A-1C illustrate embodiments of emitters with various mechanical filter and shutter configurations in accordance with embodiments of the present disclosure. [Diagram 34] 1A-1C illustrate embodiments of emitters with various mechanical filter and shutter configurations in accordance with embodiments of the present disclosure. [Diagram 35]1A-1C illustrate embodiments of emitters with various mechanical filter and shutter configurations in accordance with embodiments of the present disclosure. [Diagram 36] 1A-1C illustrate embodiments of emitters with various mechanical filter and shutter configurations in accordance with embodiments of the present disclosure. [Figure 37] FIG. 1 is a schematic diagram illustrating a system for providing illumination to a low-light environment, according to one embodiment. [Figure 38] 1 is a schematic block diagram illustrating a light source having multiple emitters, according to one embodiment. [Figure 39] 1 is a schematic block diagram illustrating a light source having multiple emitters, according to another embodiment. [Diagram 40] 1 is a schematic block diagram illustrating a light source having multiple emitters, according to yet another embodiment. [Diagram 41] FIG. 1 is a schematic diagram illustrating a single optical fiber outputting through a diffuser at the output to illuminate a scene, according to one embodiment. [Diagram 42] FIG. 2 is a block diagram illustrating the generation of a filtered image using a filter, according to one embodiment. [Diagram 43] FIG. 2 illustrates a portion of the electromagnetic spectrum separated into multiple different sub-spectra that can be emitted by an emitter of a light source, according to one embodiment. [Diagram 44] FIG. 2 is a schematic diagram illustrating an emission and readout timing diagram for generating a multispectral or hyperspectral image according to one embodiment. [Diagram 45] FIG. 2 is a block diagram illustrating the generation of a filtered image using a filter, according to one embodiment. [Diagram 46] FIG. 2 is a block diagram illustrating the generation of a filtered image using multiple filters, according to one embodiment. [Figure 47] 1 is a schematic diagram illustrating a grid array for tracking objects and / or surfaces, according to one embodiment. [Figure 48]FIG. 1 is a schematic flow chart diagram illustrating a method of emission and readout for generating a multispectral or hyperspectral image, according to one embodiment. [Figure 49] FIG. 1 is a schematic flow chart diagram illustrating a method of emission and readout for generating a fluorescent image, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present disclosure extends to methods, systems, and computer-based products for digital imaging that may be primarily suitable for medical applications, such as medical endoscopic imaging. Such methods, systems, and computer-based products disclosed herein may provide imaging or diagnostic capabilities for use in medical robotics applications, such as the use of robotics to perform imaging procedures, surgical procedures, and the like. In the following description of the present disclosure, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which embodiments of the present invention may be practiced. It is understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure.

[0013] Endoscopes have a wide variety of uses and can provide significant benefits in the medical field. Endoscopy is used in medicine to see inside the body and can provide imaging that, in some cases, is otherwise not visible or would require invasive surgical procedures. Endoscopes can be used for medical diagnosis, investigation, or research, and can also be used to perform medical procedures in a minimally invasive manner. Medical endoscopes can provide significant benefits to patients and medical personnel by negating the need for painful and invasive corrective or exploratory surgery.

[0014] As disclosed herein, an endoscopic system for use in low light environments, such as a body cavity, can include an imaging device and a light engine. The light engine can include an illumination source for generating pulses of electromagnetic radiation and can further include a lumen at a distal tip of the endoscope for transmitting the pulses of electromagnetic radiation. The lumen can transmit the pulses of electromagnetic radiation at a particular wavelength or band of wavelengths in the electromagnetic spectrum. The lumen can transmit such pulses in a timed sequence, and imaging data can be captured by a sensor during each pulse. The imaging data associated with the pulses of different wavelengths can be used to generate a red green blue (RGB) image and / or a multispectral or hyperspectral image. In an embodiment, the multispectral or hyperspectral imaging can be overlaid on a black and white or RGB image.

[0015] As disclosed herein, systems, methods, and devices for endoscopic imaging systems can provide specialized image data of low-light environments. The specialized image data can be used to generate multispectral or hyperspectral imaging and / or to identify specific materials, tissues, components, or processes within the low-light environment. In certain embodiments, the multispectral or hyperspectral imaging can be provided to a practitioner or a computer-implemented program to enable identification of specific structures or tissues within the body. Such multispectral or hyperspectral imaging data can be overlaid on black and white or RGB images to provide additional information and context.

[0016] Furthermore, such systems, methods, and devices of endoscopic imaging systems can be used in concert with specific reagents or dyes. In medical imaging implementations, a patient can be administered a specific reagent or dye that can fluoresce or react to electromagnetic radiation of a specific wavelength. The endoscopic imaging systems disclosed herein can transmit electromagnetic radiation of a specified wavelength to cause the reagent or dye to fluoresce. The fluorescence of the reagent or dye can be captured by an image sensor to generate an image to aid in identifying tissues or structures and / or aid in diagnosis or investigation. In one implementation, a patient can be administered multiple reagents or dyes, each configured to fluoresce at a different wavelength and / or provide an indication of a different structure, tissue, chemical reaction, biological process, or the like. In such implementations, the endoscopic systems disclosed herein can cause the applicable reagent or dye to fluoresce by emitting each of the applicable wavelengths. This can negate the past need to perform individual imaging procedures for each of the multiple reagents or dyes.

[0017] A medical endoscope can provide a continuous digital image stream of the interior space of the body into which the distal end of the endoscope is inserted. In various implementations, it may be beneficial or even necessary for the digital image stream to provide full color imaging so that medical personnel can better distinguish between tissues and structures within the body. In further implementations, it may be beneficial to provide hyperspectral imaging data to distinguish between structures, tissues, processes, and conditions with increased accuracy. Additionally, hyperspectral imaging can enable medical personnel or computer programs to receive information about conditions of the human body that cannot be seen by the human eye or discerned in an RGB color image.

[0018] Disclosed herein are systems, methods, and apparatus for generating color image data and / or hyperspectral image data by an endoscope. The disclosed system includes an imaging device having a tube, one or more image sensors, and a lens assembly. The lens assembly can include at least one optical element corresponding to at least one of the one or more image sensors. The system further includes a display for visualizing the scene, and an image signal processing controller. The system can further include a light engine. The light engine includes an illumination source configured to generate one or more pulses of electromagnetic radiation, and a lumen for transmitting the one or more pulses of electromagnetic radiation to a distal tip of the endoscope. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation includes electromagnetic radiation between 513 nm and 545 nm. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation includes electromagnetic radiation between 565 nm and 585 nm. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation include electromagnetic radiation between 900 nm and 1000 nm. In implementations of the present disclosure, pulsing by the light engine can be of very short duration and can have on / off times of less than 1 microsecond (μs).

[0019] In an embodiment of the present disclosure, an endoscopic system illuminates an illumination source for spectral or hyperspectral imaging and pulses electromagnetic radiation. Spectral imaging uses multiple bands across the electromagnetic spectrum. This differs from conventional cameras that only capture light across three wavelengths based on the visible spectrum that are distinguishable by the human eye, including red, green, and blue wavelengths to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, visible spectrum, ultraviolet spectrum, x-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can overlay imaging generated based on invisible bands (e.g., infrared) on top of imaging based on visible bands (e.g., standard RGB images) to provide additional information that can be easily identified by humans or computer algorithms.

[0020] Hyperspectral imaging is a subcategory of spectral imaging. Hyperspectral imaging includes spectroscopy and digital photography. In one embodiment of hyperspectral imaging, a complete spectrum or some spectral information is collected at every pixel in the image plane. Using specialized hardware, a hyperspectral camera can capture any suitable number of wavelength bands per pixel, which can be interpreted as a complete spectrum. The goal of hyperspectral imaging can vary for different applications. In one application, the goal of hyperspectral imaging is to obtain the entire electromagnetic range of each pixel in the image scene. This can make it possible to find specific objects that may otherwise be indistinguishable under visible light wavelength bands. This can make it possible to accurately identify specific materials or tissues when they may be indistinguishable under visible light wavelength bands. Furthermore, this can make it possible to detect specific processes by capturing images across all wavelengths of the electromagnetic spectrum.

[0021] Hyperspectral imaging can provide certain advantages over conventional imaging in medical applications. Information obtained by hyperspectral imaging can enable medical personnel and / or computer-implemented programs to accurately identify specific tissues or conditions that may lead to diagnoses that may not be possible or may be less accurate when using conventional imaging, such as RGB imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, which can enable medical personnel to, for example, view tissues located behind specific tissues or fluids, see atypical cancer cells as opposed to typical healthy cells, identify specific tissues or conditions, identify important structures, etc. Hyperspectral imaging can provide specialized diagnostic information regarding tissue physiology, morphology, and composition that cannot be generated by conventional imaging.

[0022] Endoscopic hyperspectral imaging can offer advantages over conventional imaging in various applications and implementations of the present disclosure. In medical implementations, endoscopic hyperspectral imaging can allow a practitioner or computer-implemented program to identify, for example, nerve tissue, muscle tissue, various vasculature, blood flow direction, and the like. Hyperspectral imaging can precisely distinguish atypical cancerous tissue from typical healthy tissue, thus allowing a practitioner or computer-implemented program to identify the borders of cancerous tumors during operational or investigative imaging. Additionally, hyperspectral imaging in low light environments as disclosed herein can be combined with the use of reagents or dyes to allow further differentiation of specific tissues or materials. In one such embodiment, a reagent or dye can fluoresce with a specific wavelength band within the electromagnetic spectrum, thus providing specific information for the purpose of that reagent or dye. The systems, methods, and devices disclosed herein can allow for pulsing of any number of wavelength bands such that one or more reagents or dyes can fluoresce at different times. In certain embodiments, this may allow for the identification or investigation of several medical conditions during a single imaging procedure.

[0023] Medical endoscopes can pulse electromagnetic radiation in wavelength bands outside the visible light spectrum to enable the production of hyperspectral images. Endoscopic hyperspectral imaging is a non-contact, non-invasive means of medical imaging that does not require the patient to undergo harmful radiation exposure common in other imaging methods.

[0024] Conventional endoscopes used in robotic endoscopic procedures, such as arthroscopy and laparoscopy, are designed such that the image sensor is typically located in a handpiece unit that is held by the endoscope operator and not inserted into the cavity. In such a configuration, the endoscope unit transmits incident light along the length of the endoscope tube toward the sensor with minimal loss and distortion through a complex set of precisely articulated optical components. The optical components are expensive and the manufacturing process for the optical components is labor intensive, so the cost of the endoscope unit is dominated by the optical components. Furthermore, this type of endoscope is mechanically delicate and relatively light impacts can easily damage the components or upset their relative alignment. Even slight misalignment of the endoscope components (such as the precisely articulated optical components) can cause a substantial degradation in image quality or render the endoscope unusable. If the components are misaligned, less incident light travels along the length of the endoscope, so there is little or no light at the distal end of the endoscope, rendering the endoscope unusable. Because conventional endoscopes require such precise and complex optical components, and because such components can easily become misaligned, such conventional endoscopes require frequent and expensive repair cycles to maintain image quality.

[0025] One solution to this challenge is to place the image sensor at the distal end within the endoscope itself. Such a solution can eliminate the need for complex and precise collection of linked optical components that can easily become misaligned and / or damaged. This solution potentially approaches the optical simplicity, robustness, and economy commonly realized in, for example, cell phone cameras. However, it should be understood that many of the advantages offered by endoscopes arise from the compactness of the distal end of the endoscope. If the distal end of the endoscope were enlarged to accommodate the multiple different wavelength-sensitive pixel sensors traditionally used for color or hyperspectral imaging, the pixel array may become too large and the endoscope may no longer fit into tight spaces or may be obstructive or invasive when used in medical implementations. Because the distal end of the endoscope must remain very small, it is difficult to place one or more image sensors at the distal end. An acceptable solution to this method is by no means trivial and brings its own set of engineering challenges, not least of which is the fact that the sensors for color imaging and / or hyperspectral imaging must be fitted into a very limited area. This is particularly difficult when the pixel array of a conventional camera includes separate pixel sensors for each of the red, green, and blue visible light bands, along with additional pixel sensors for the other wavelength bands used for hyperspectral imaging. The area of ​​the distal tip of the endoscope may be limited, particularly from side to side in the X and Y dimensions, but there is more space along the length of the endoscope tube in the Z dimension.

[0026] Since many of the advantages of endoscopes come from the miniaturization of the distal end of the endoscope, invasive constraints must be imposed on the image sensor area when the image sensor is located at the distal end. Such invasive constraints imposed on the sensor area necessarily result in fewer and / or smaller pixels in the pixel array. Reducing the number of pixels may directly affect the spatial resolution, while reducing the pixel area may reduce the available signal capacity and thus the pixel sensitivity, as well as the minimum pixel resolution using the optimization of the number of pixels, maximum pixel quality and pitch to maximize the image quality, so that the resolution becomes unimportant, and the signal to noise ratio (SNR) of each pixel is reduced. Reducing the signal capacity reduces the dynamic range, i.e., the ability of the imager or camera to simultaneously capture all useful information from a scene with a wide range of luminosity. There are various methods to extend the dynamic range of an imaging system beyond the dynamic range of its own pixels. However, they all have some kind of loss (e.g. in resolution or frame rate) and can introduce undesirable artifacts that become problematic in extreme cases. Reducing sensitivity results in darker areas of the scene requiring more light power to bring them to an acceptable signal level. Lowering the F-number (widening the aperture) can compensate for the loss of sensitivity, but at the expense of spatial distortion and a shallower depth of focus.

[0027] In the sensor industry, complementary metal-oxide-semiconductor ("CMOS") image sensors have largely replaced traditional charge-coupled device ("CCD") image sensors in modern camera applications. CMOS image sensors are easier to integrate and operate, have better or comparable image quality, are more versatile, and are less expensive than CCD image sensors. Typically, CMOS image sensors can include the circuitry required to convert image information into digital data, and can have various levels of digital processing subsequently incorporated. This can range from basic algorithms to correct for non-idealities, e.g., resulting from variations in amplifier behavior, to a complete image signal processing (ISP) chain providing video data in a standard red-green-blue (RGB) color space, e.g., a camera on-chip.

[0028] The control unit of the endoscope or image sensor may be located remotely from the image sensor and may be at a significant physical distance from the image sensor. If the control unit is remote from the sensor, it may be desirable to transmit the data in the digital domain, as this is less susceptible to interference noise and signal degradation when compared to transmitting an analog data stream. It will be appreciated that various electrical digital signaling standards may be used, such as low voltage differential signaling (LVDS), sub-LVDS, scalable low voltage signaling (SLVS), or other electrical digital signaling standards.

[0029] There may be a strong desire to minimize the number of electrical conductors to reduce the number of space consuming pads on the sensor, as well as to reduce the complexity and cost of sensor manufacturing. The addition of analog-to-digital conversion to the sensor may be advantageous, but the additional area occupied by the conversion circuitry is offset by the significant reduction in analog buffering power required by the initial conversion to a digital signal.

[0030] With respect to area consumption, given the typical feature sizes available in CMOS image sensor (CIS) technology, in some implementations it may be preferable to have all internal logic signals generated on the same chip as the pixel array via a set of control registers and a simple command interface.

[0031] Some implementations of the present disclosure can include combined sensor and system design aspects that enable high resolution imaging with reduced pixel count in highly controlled lighting environments. This can be achieved by pulsing a single color wavelength on a frame-by-frame basis, and by switching or alternating each frame between a single different color wavelength using a controlled light source in conjunction with a high frame capture rate and a corresponding specially designed monochrome sensor. In addition, electromagnetic radiation outside the visible light spectrum can be pulsed to enable the generation of hyperspectral images. The pixels can be color blind, and thus each pixel can generate data for each pulse of electromagnetic radiation, including pulses of red, green, and blue visible light wavelengths along with other wavelengths that can be used for hyperspectral imaging.

[0032] As used herein, a monochrome sensor refers to an unfiltered imaging sensor. Because the pixels are color agnostic, the effective spatial resolution is much higher than their color (usually filtered with a Bayer pattern) counterparts in conventional single-sensor cameras. The pixels may also have a higher quantum efficiency, since much less incident light is wasted between individual pixels. Furthermore, Bayer-based spatial color modulation requires that the modulation transfer function (MTF) of the accompanying optics be degraded compared to the monochromatic case in order to blur the color artifacts associated with the Bayer pattern. This has a detrimental effect on the actual spatial resolution that can be achieved with a color sensor.

[0033] This disclosure also relates to a system solution for endoscopic applications where the image sensor resides at the distal end of the endoscope. In the pursuit of the smallest area sensor-based system, there are other design aspects that can be exploited besides reducing the number of pixels. The area of ​​the digital part of the chip can be minimized. In addition, the number of connections to the chip (pads) can also be minimized. This disclosure describes a novel way of achieving these goals to realize such a system. This involves the design of a full custom CMOS image sensor with several novel features.

[0034] For the purpose of promoting an understanding of the principles according to the present disclosure, reference will now be made to the embodiments shown in the drawings, and specific language will be used to describe the drawings. It will nevertheless be understood that no limitation of the scope of the present disclosure is intended thereby. Any alterations and further modifications of the features of the present invention exemplified herein, and any additional applications of the principles of the present disclosure exemplified herein, will normally occur to those skilled in the art, and possession of the present disclosure is deemed to be within the scope of the disclosure as claimed.

[0035] Before disclosing and describing structures, systems, and methods for generating images in low light environments, it is to be understood that the present disclosure is not limited to the particular structures, configurations, process steps, and materials disclosed herein, as such structures, configurations, process steps, and materials may vary somewhat. It is also to be understood that the terminology used herein is used for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present disclosure will be limited only by the appended claims and equivalents thereof.

[0036] In describing and claiming the subject matter of this disclosure, the following terminology will be used in accordance with the definitions set out below.

[0037] It must be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0038] As used herein, the terms "comprising," "including," "containing," "characterized by," and their grammatical equivalents are inclusive or broadly interpreted terms that do not exclude additional, unrecited elements or method steps.

[0039] As used herein, the word "consisting of" and its grammatical equivalents exclude any element or step not specified in the claim.

[0040] As used herein, the phrase "consisting essentially of" and their grammatical equivalents limit the scope of a claim to the materials or steps specified and that do not materially affect the basic and novel feature or features of the claimed disclosure.

[0041] As used herein, the term "proximal" refers broadly to the concept of the part closest to the origin.

[0042] As used herein, the term "distal" refers generally to the opposite of proximal and thus the notion of being further or furthest from the origin, depending on the context.

[0043] As used herein, a color sensor or multispectral sensor is a sensor known to have a color filter array (CFA) thereon to filter incident electromagnetic radiation into its separate components. In the electromagnetic spectrum, such a CFA may be built on the Bayer pattern, or a modification thereof, to separate the green, red, and blue spectral components of light.

[0044] A system and method for generating images in low light environments will now be described with reference to Figures 1-5. Figure 1 illustrates a schematic diagram of a pair of sensors and electromagnetic emitters in operation for use in generating images in low light environments. Such a configuration allows for enhanced functionality in light controlled or low ambient lighting environments.

[0045] As used herein, the term "light" refers to electromagnetic radiation, both particles and wavelengths, that can be detected by the pixel array, and it should be noted that this may include wavelengths in the visible and non-visible spectrum of electromagnetic radiation. The term "septum" is used herein to refer to a range of wavelengths in the electromagnetic spectrum that is less than the entire spectrum, in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that may be controllable with respect to the physical properties of its components, the intensity of emission, or the duration of emission, or the portion of the electromagnetic spectrum that is emitted or can operate with respect to all of the above. Emitters may emit light in any dithered, diffused, or collimated emission type, and may be controlled digitally or via analog methods or systems. As used herein, an electromagnetic emitter is a source of electromagnetic energy and includes light sources such as lasers, LEDs, incandescent lights, or any light source that can be digitally controlled.

[0046] The pixel array of the image sensor may be electrically paired with the emitter so that they are synchronized during operation to receive both emissions and to adjustments made within the system. As can be seen in FIG. 1, the emitter 100 may be arranged to emit electromagnetic radiation in the form of a laser, which may be pulsed to illuminate the object 110. The emitter 100 may be pulsed at intervals corresponding to the operation and function of the pixel array 122. The emitter 100 may pulse light at multiple electromagnetic partitions 105, so that the pixel array receives electromagnetic energy and generates a data set corresponding (in time) to each particular electromagnetic partition 105. For example, FIG. 1 shows a system having a monochrome sensor 120 with a pixel array (black and white) 122 and supporting circuitry, where the pixel array 122 is sensitive to electromagnetic radiation of any wavelength. The light emitter 100 illustrated in the figure may be a laser emitter capable of emitting red electromagnetic septa 105a, blue electromagnetic septa 105b, and green electromagnetic septa 105c in any desired sequence. In an embodiment capable of producing a hyperspectral image, the light emitter 100 may pulse electromagnetic radiation at any wavelength in the electromagnetic spectrum such that a hyperspectral image may be produced. It will be understood that other light emitters 100, such as digital or analog-based emitters, may be used in FIG. 1 without departing from the scope of the present disclosure.

[0047] During operation, the data produced by the monochromatic sensor 120 for any individual pulse can be assigned a particular color or wavelength partition, based on the timing of the color or wavelength partition pulsed from the emitter 100. Even if a pixel 122 is not color-dedicated, such pixel can be assigned a color for any given data set based on a priori information about the emitter.

[0048] In an exemplary embodiment of the present disclosure, emitter 100 pulses electromagnetic radiation at specialized wavelengths. Such pulses can enable the generation of specialized hyperspectral images that are particularly suited for certain medical or diagnostic applications. In an exemplary embodiment, emitter 100 pulses electromagnetic radiation at 513 nm to 545 nm, 565 nm to 585 nm, and 900 nm to 1000 nm.

[0049] In one embodiment, three data sets representing red, green, and blue electromagnetic pulses may be combined to form a single image frame. One or more additional data sets representing other wavelength divisions may be overlaid onto the single image frame based on the red, green, and blue pulses. The one or more additional data sets may represent pulses from 513 nm to 545 nm, 565 nm to 585 nm, and 900 nm to 1000 nm, for example. The one or more additional data sets may represent fluorescent imaging and / or hyperspectral, which may be overlaid onto the single image frame based on the red, green, and blue pulses.

[0050] It will be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition, and that any color combination or any electromagnetic partition may use cyan, magenta, and yellow, ultraviolet, infrared, any combination of the foregoing, or any other color combination, including all visible and non-visible wavelengths, instead of red, green, and blue, without departing from the scope of the present disclosure. In the figure, the object 110 being imaged includes a red portion 110a, a green portion 110b, and a blue portion 110c. As shown, the reflected light from the electromagnetic pulse contains data only for the portion of the object that has a particular color corresponding to the pulsed color partition. These separate color (or color interval) data sets can then be used to reconstruct an image at 130 by combining the data sets.

[0051] In one embodiment, multiple data sets representing red, green, and blue electromagnetic pulses, along with additional wavelength partitions along the electromagnetic spectrum, can be combined to form a single image frame having an RGB image with hyperspectral image data overlaid on the RGB image. Depending on the application or situation, different combinations of wavelength data sets may be desirable. For example, in some implementations, a data set representing a particular wavelength partition may be used to generate a specialized hyperspectral image for diagnosing a particular medical condition, investigating a particular body tissue, etc.

[0052] As illustrated in FIG. 2, embodiments of the present disclosure may comprise or utilize a special purpose or general purpose computer including computer hardware such as one or more processors and system memory, as discussed in detail below. Embodiments within the scope of the present disclosure may also include physical media and other computer readable media that support or store computer executable instructions and / or data structures. Such computer readable media may be any available media that can be accessed by a general purpose or special purpose computer system. Computer readable media that store computer executable instructions are computer storage media (devices). Computer readable media that support computer executable instructions are transmission media. Thus, by way of non-limiting example, embodiments of the present disclosure may include at least two distinctly different types of computer readable media: computer storage media (devices) and transmission media.

[0053] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSD") (e.g., RAM-compatible), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computer.

[0054] A "network" is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules and / or other electronic devices. In one implementation, the sensors and camera control units can be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transferred or provided to a computer over a network or another communication connection (either hardwired, wireless, or a combination of hardwired or wireless), the computer properly regards the connection as a transmission medium. A transmission medium can include a network and / or data links that can be used to convey desired program code means in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0055] Furthermore, program code means in the form of computer executable instructions or data structures that can be automatically transferred from transmission media to computer storage media (devices) (or vice versa) upon reaching various computer system components. For example, computer executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., "NIC") and then eventually transferred to computer system RAM and / or less volatile computer storage media (devices) in the computer system. RAM can also include solid state drives (SSD or PCIx-compatible real-time memory hierarchical storage, such as FusionIO). It should therefore be understood that computer storage media (devices) can be included in computer system components that also (or may primarily) utilize transmission media.

[0056] Computer-executable instructions include, for example, instructions and data that, when executed on a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or source code. Although the subject matter has been described in language specific to structural features and / or method acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as exemplary forms of carrying out the disclosure.

[0057] Those skilled in the art will appreciate that the present disclosure may be practiced in a network computing environment having many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, control units, camera control units, handheld devices, handpieces, multiprocessor systems, microprocessor-based or programmable consumer electronics products, network PCs, minicomputers, mainframe computers, cell phones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the computing devices described above may be provided by or located within a brick-and-mortar location. The present disclosure may also be practiced in a distributed system environment in which both local and remote computer systems, linked via a network (either by hardwired data links, wireless data links, or a combination of hardwired and wireless data links), perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0058] Further, where appropriate, the functions described herein may be implemented by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) may be programmed to execute one or more of the systems and procedures described herein. Certain terms are used throughout the following description and claims to refer to particular system components. As will be appreciated by those skilled in the art, components may be referred to by different names. This specification does not intend to distinguish between components that differ in name but not function.

[0059] 2 is a block diagram illustrating an exemplary computing device 150. Computing device 150 can be used to perform various procedures, such as those discussed herein. Computing device 150 can function as a server, a client, or any other computing entity. Computing device 150 can perform various monitoring functions as discussed herein and can execute one or more application programs, such as those described herein. Computing device 150 can be any of a wide variety of types of computing devices, such as a desktop computer, a notebook computer, a server computer, a portable computer, a camera control unit, a tablet computer, etc.

[0060] Computing device 150 includes one or more processors 152, one or more memory devices 154, one or more interfaces 156, one or more mass storage devices 158, one or more input / output (I / O) devices 160, and a display device 180, all coupled to a bus 162. Processor(s) 152 include one or more processors or controllers that execute instructions stored in memory(s) 154 and / or mass storage device(s) 158. Processor(s) 152 may also include various types of computer-readable media, such as cache memory.

[0061] The storage device(s) 154 may include a variety of computer-readable media, such as volatile memory (e.g., random access memory (RAM) 164) and / or non-volatile memory (e.g., read-only memory (ROM) 166). The storage device(s) 154 may also include re-writable ROM, such as flash memory.

[0062] Mass storage device(s) 158 include various computer readable media, such as magnetic tapes, magnetic disks, optical disks, solid state memory (e.g., flash memory), etc. As shown in Figure 2, a particular mass storage device is a hard disk drive 174. Various drives may also be included in mass storage device(s) 158 to enable reading from and / or writing to the various computer readable media. Mass storage device(s) 158 include removable media 176 and / or non-removable media.

[0063] I / O device(s) 160 include a variety of devices that enable the input or retrieval of data and / or other information to computing device 150. Exemplary I / O device(s) 160 include digital imaging devices, electromagnetic sensors and emitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.

[0064] Display device 180 includes any type of device capable of displaying information to one or more users of computing device 150. Examples of display device 180 include a monitor, a display terminal, a video projection device, etc.

[0065] The interface(s) 106 include a variety of interfaces that allow the computing device 150 to interact with other systems, devices, or computing environments. Exemplary interface(s) 156 can include any number of different network interfaces 170, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet. Other interface(s) include a user interface 168 and a peripheral interface 172. The interface(s) 156 can also include one or more user interface elements 168. The interface(s) 156 can also include one or more peripheral interfaces, such as a printer, a pointing device (mouse, trackpad, etc.), a keyboard, etc.

[0066] Bus 162 allows processor(s) 152, storage device(s) 154, interface(s) 156, mass storage device(s) 158, and I / O device(s) 160 to communicate with each other and with other devices or components coupled to bus 162. Bus 162 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, etc.

[0067] For purposes of illustration, programs and other executable program components are illustrated herein as discrete blocks, with the understanding that such programs and components may reside at various times in different storage components of computing device 150 and be executed by processor(s) 152. Alternatively, the systems and procedures described herein may be implemented in hardware or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) may be programmed to execute one or more of the systems and procedures described herein.

[0068] FIG. 2A illustrates an operational cycle of a sensor used in a rolling readout mode or during sensor readout 200. A frame readout may be represented by a vertical line 210 and may begin there. A readout period is represented by a diagonal or diagonal line 202. The sensor may be read out row by row, with the top of the downward diagonal edge being the top row of sensors 212 and the bottom of the downward diagonal edge being the bottom row of sensors 214. The time between the last row readout and the next readout cycle may be referred to as a blanking time 216. Note that some of the sensor pixel rows may be covered with a light shield (e.g., a substantially black layer of a metal coating or any other type of material). These covered pixel rows may be referred to as optical black rows 218 and 220. The optical black rows 218 and 220 may be used as input for a correction algorithm. As shown in FIG. 2A, these optical black rows 218 and 220 can be located at the top of the pixel array, or at the bottom of the pixel array, or at the top and bottom of the pixel array. FIG. 2B illustrates a process of controlling the amount of electromagnetic radiation, e.g., light, that is exposed to a pixel and thereby integrated or stored by the pixel. It will be understood that photons are elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or stored by each pixel and converted into an electric charge or current. An electronic shutter or rolling shutter (shown by dashed line 222) can be used to start the integration time by resetting the pixel. The light can then be integrated until the next readout stage. The position of the electronic shutter 222 can be moved between the two readout cycles 202 to control pixel saturation for a given amount of light. Note that this technique allows a constant integration time between two different lines, but creates a delay when moving from the top row to the bottom row. FIG. 2C illustrates the case where the electronic shutter 222 is removed. In this configuration, integration of the incident light can begin during readout 202 and can end with the next readout cycle 202, which also defines the start of the next integration. Figure 2D shows a configuration without an electronic shutter 222, but with a controlled and pulsed light 230 during the blanking time 216.This ensures that all rows see the same light emitted from the same light pulse 230. In other words, each row can start its integration in a dark environment, which may be at the optical black trailing row 220 of readout frame (m) for maximum light pulse width, then receive a light strobe, and end its integration in a dark environment, which may be at the optical black leading row 218 of the next subsequent readout frame (m+1) for maximum light pulse width. In the example of FIG. 2D, the image generated from the light pulse is only available during the readout of frame (m+1) without any interference with frames (m) and (m+2). Note that a condition to have a light pulse that is read out only in one frame and does not interfere with adjacent frames is the emission of a given light pulse during blanking time 216. Since the optical black columns 218, 220 are insensitive to light, the trailing optical black row 220 of frame (m) and the leading optical black row 218 of frame (m+1) can be added to the blanking time 216 to determine the maximum range of emission times of the light pulses 230. As illustrated in FIG. 2A, the sensor can be cycled multiple times to receive data for each pulsed color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum). Each cycle can be timed. In one embodiment, the cycles can be timed to operate within an interval of 16.67 ms. In another embodiment, the cycles can be timed to operate within an interval of 8.3 ms. It will be understood that other time intervals are contemplated and intended to be within the scope of this disclosure.

[0069] FIG. 3 illustrates in graphical form the operation of an embodiment of an electromagnetic emitter. The emitter can be timed to correspond to a cycle of the sensor so that electromagnetic radiation is emitted within and / or during a portion of the sensor operating cycle. FIG. 3 illustrates pulse 1 at 302, pulse 2 at 304, and pulse 3 at 306. In one embodiment, the emitter can be pulsed during the readout portion 202 of the sensor operating cycle. In one embodiment, the emitter can be pulsed during the blanking portion 216 of the sensor operating cycle. In one embodiment, the emitter can be pulsed for a duration during a portion of two or more sensor operating cycles. In one embodiment, the emitter can begin pulsing during the blanking portion 216 or during the optical black portion 220 of the readout portion 202 and end pulsing during the readout portion 202 or during the optical black portion 218 of the readout portion 202 of the next subsequent cycle. It will be understood that any combination of the above is intended to fall within the scope of the present disclosure so long as the emitter pulses and sensor cycles are matched.

[0070] 4 graphically illustrates the variation of duration and amplitude of emitted electromagnetic pulses (e.g., pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406) to control exposure. An emitter with a constant output amplitude may be pulsed during any of the cycles described above with reference to FIG. 2D and FIG. 3 at intervals to provide the required electromagnetic energy to the pixel array. An emitter with a constant output amplitude may be pulsed at a longer interval to provide more electromagnetic energy to the pixel, or the emitter may be pulsed at a shorter interval to provide less electromagnetic energy. Whether a longer or shorter interval is required depends on the operating conditions.

[0071] In contrast to adjusting the time interval at which the emitter pulses a constant output amplitude, the amplitude of the emission itself can be increased to provide more electromagnetic energy to the pixel. Similarly, decreasing the amplitude of the pulse provides less electromagnetic energy to the pixel. It should be noted that an embodiment of the system can have the ability to simultaneously adjust both the amplitude and duration if desired. Also, the sensor can be adjusted to increase the desired sensitivity and duration for optimal image quality. FIG. 4 illustrates varying the amplitude and duration of the pulses. In the figure, pulse 1 at 402 has a greater amplitude or intensity than either pulse 2 at 404 or pulse 3 at 406. In addition, pulse 1 at 402 has a shorter duration than pulse 2 at 404 or pulse 3 at 406, such that the electromagnetic energy provided by the pulses is illustrated by the area under the pulses shown in the figure. In the figure, pulse 2 at 404 has a relatively smaller amplitude or intensity and a longer duration when compared to either pulse 1 at 402 or pulse 3 at 406. Finally, in the figure, pulse 3 at 406 has an intermediate amplitude or intensity and duration when compared to pulse 1 at 402 and pulse 2 at 404 .

[0072] 5 shows a graphical representation of an embodiment of the present disclosure that combines the operating cycle, electromagnetic emitter, and emitted electromagnetic pulses of FIGS. 2-4 to demonstrate an imaging system in operation in accordance with the principles and teachings of the present disclosure. As can be seen, the electromagnetic emitter pulses emission primarily during the blanking period 216 of the sensor, which causes the pixels to be charged and ready to be read during the readout portion 202 of the sensor cycle. The dashed portions of the pulses (from FIG. 3) illustrate the potential or ability to emit electromagnetic energy during the optical black portions 220 and 218 of the readout cycle (sensor cycle) 200 when additional time is needed or desired to pulse the electromagnetic energy.

[0073] Referring now to FIG. 6-FIG. 9A, FIG. 6 shows a schematic diagram of two different processes over a period t(0)-t(1) for recording a video frame of full spectrum light and split spectrum light. It is noted that the color sensor has a color filter array (CFA) for filtering a specific wavelength of light per pixel that is typically used for full spectrum light reception. An example of a CFA is the Bayer pattern. Since the color sensor may include pixels in the array made sensitive to a single color from the full spectrum, a low resolution image results because the pixel array has pixel space dedicated to only a single light in the full spectrum. Typically, such an arrangement is formed in a checkerboard type pattern across the array.

[0074] In contrast, when a split spectrum of light is used, a sensor can be made that can be sensitive or responsive to all light energy amplitudes because the pixel array is instructed to sense electromagnetic energy from a predetermined partition of the full spectrum of electromagnetic energy in each cycle. Thus, to form an image, the sensor needs to cycle through only different partitions from within the full spectrum of light and then reassemble the image to display a predetermined mix of color values ​​for all pixels in the array. Thus, a higher resolution image is also provided because the distance is reduced compared to a Bayer sensor between pixel centers of the same color sensitivity for each color pulse. As a result, the color image formed has a higher modulation transfer function (MTF). Because the image from each color split frame cycle has a higher resolution, the resulting image produced when the split light frames are combined into a full color frame also has a higher resolution. In other words, because every pixel in the array (at most every second pixel in the sensor with a color filter) senses the energy amplitude for a given pulse and given scene, a high resolution image is created for each scene using less acquired (less accurate) data that needs to be introduced only a few individual times.

[0075] For example, white light or full spectrum visible light is a combination of red light, green light, and blue light. In the embodiment shown in FIG. 6, it can be seen that in both split spectrum process 620 and full spectrum process 610, the time of capturing an image is t(0) to t(1). In full spectrum process 610, white light or full spectrum electromagnetic energy is emitted at 612. The white or full spectrum electromagnetic energy is sensed at 614. The image is processed and displayed at 616. Thus, between times t(0) and t(1), the image is processed and displayed. Conversely, in split spectrum process 620, a first septum is emitted at 622 and sensed at 624. A second septum is emitted at 626 and then sensed at 628. A third septum is emitted at 630 and then sensed at 632. The image is processed and displayed at 634. It will be understood that any system that uses an image sensor cycle that is at least twice as fast as the white light cycle is intended to fall within the scope of this disclosure.

[0076] 6, as can be seen in graphical format between times t(0) and t(1), the sensors of split spectrum system 620 cycle three times for each full spectrum system. In split spectrum system 620, the first of the three sensor cycles is for green spectrum 622 and 624, the second of the three is for red spectrum 626 and 628, and the third is for blue spectrum 630 and 632. Thus, in one embodiment where the display device (LCD panel) operates at 50-60 frames per second, the split light system should operate at 150-180 frames per second to maintain continuity and smoothness of the displayed image.

[0077] In other embodiments, there may be different capture and display frame rates. Additionally, the average capture rate may be any multiple of the display rate.

[0078] In one embodiment, it may be desirable for all partitions not to be equally represented within the system frame rate. In other words, all light sources must be pulsed with the same regularity to accentuate and de-emphasize aspects of the recorded scene as desired by the user. It should also be understood that non-visible and visible partitions of the electromagnetic spectrum may be pulsed together within the system, with their respective data values ​​stitched into the video output as desired for display to the user.

[0079] One embodiment may include a pulse cycle pattern as follows: i. green pulse, ii. red pulse; iii. Blue pulse; iv. green pulse; v. Red pulse, vi. Blue pulse, vii. Infra-red (IR) pulses; viii.(Repeat)

[0080] As can be seen in the examples, infrared septa or special wavelength septa (e.g., 513-545 nm, 565-585 nm, and / or 900-100 nm) can be pulsed at a different rate than the rate of other septa pulses. This may be done to highlight certain aspects of a scene, and the IR data can simply be overlaid with other data in the video output to provide the desired emphasis. Note that the addition of electromagnetic septa on top of the red, green, and blue septa does not necessarily require the serialized system to operate at four times the rate of a full spectrum non-serial system, since not all septa need to be equally represented in the pulse pattern. As can be seen in this embodiment, the addition of septa pulses that are less represented in the pulse pattern (infrared in the above example) can result in less than a 20% increase in the sensor cycle rate to accommodate the irregular septa sampling.

[0081] In various embodiments, the pulse cycle pattern can include any of the following wavelengths in any suitable order, which may be particularly suitable for determining multispectral or hyperspectral image data, or for determining image data based on fluorescent reagent relaxation emissions: i.465±5nm, ii. 533±4 nm, iii.638±5nm, iv.780±5nm, v.805±5nm, vi.975±5nm, vii. 577±2 nm, or viii.523±4 nm.

[0082] In one embodiment, an electromagnetic septum sensitive to a dye or material used to highlight an aspect of the scene may be emitted. In this embodiment, high resolution may not be required, which may be sufficient to highlight the location of the dye or material. In such an embodiment, the dye-sensitive electromagnetic septum may be cycled much less frequently than other septums in the system to contain the highlighted data.

[0083] The partition cycle may be divided to correspond to or approximate various imaging and video standards. In one embodiment, as best illustrated in Figures 7A-7D below, the partition cycle may include pulses of electromagnetic energy in the red, green, and blue spectrum. In Figure 7A, different light intensities are achieved by adjusting the width or duration of the light pulse within the operating range shown by the grey vertical dashed lines. In Figure 7B, different light intensities are achieved by adjusting the light power or the power of the electromagnetic emitter, which may be a laser emitter or an LED emitter, but keeping the pulse width or duration constant. Figure 7C shows the case where both the light power and the light pulse width are adjusted, which leads to greater flexibility. The partition cycle may use CMY, IR, and ultraviolet using invisible pulse sources mixed with visible pulse sources and any other color spaces required to generate images or to approximate desired video standards now known or further developed. It should also be understood that the system may be capable of switching between color spaces on the fly to provide the desired image output quality.

[0084] In an embodiment using the color space Green-Blue-Red (as seen in FIG. 7D), a user may wish to pulse the luminance component more frequently than the chrominance components, since they are generally more sensitive to differences in the amplitude of light than to differences in the color of light. This principle can be exploited using a monochrome sensor, as illustrated in FIG. 7D, where green, which contains the most luminance information, can be pulsed more frequently or with greater intensity in a (GBGRGBGR...) scheme to obtain the luminance data. Such a configuration creates a video stream with more perceptible detail without creating and transmitting imperceptible data.

[0085] In one embodiment, duplicating the pulse of a weaker partition may be used to generate an output adjusted for the weaker pulse. For example, blue laser light is considered weak compared to the sensitivity of silicon-based pixels and is also difficult to generate compared to red or green light, and therefore may be pulsed more frequently during the frame cycle to compensate for the weaker light. These additional pulses may be done sequentially over time, or by using multiple lasers pulsing simultaneously to generate the desired compensation effect. Note that by pulsing during blanking periods (times when the sensor is not reading out the pixel array), the sensor is insensitive to differences / mismatches of the same type of laser and simply accumulates light for the desired output. In another embodiment, the maximum light pulse range may differ between frames. This is shown in FIG. 7E, where the light pulses differ between frames. The sensor may be constructed such that different blanking times can be programmed with a repeating pattern of 2, or 3, or 4, or n frames. In FIG. 7E, four different light pulses are illustrated, pulse 1 can be repeated, for example, after pulse 4, and have a pattern of four frames with different blanking times. Using this technique, the strongest partitions can be placed for the smallest blanking time, thus allowing the weakest partitions to have a wider pulse in one of the next frames without the need to increase the readout speed. The reconstructed frame, when composed of many pulsed frames, can still have a regular pattern between the frames.

[0086] As can be seen in FIG. 8, since the light in each divided range can have different energy values, the sensor and / or light emitter can be adjusted to compensate for the difference in energy values. At 810, the data obtained from the histogram from the previous frame can be analyzed. At 820, the sensor can be adjusted as described below. In addition, at 830, the emitter can be adjusted. At 840, an image can be obtained from the sensor at an adjusted sampling time, or the image can be obtained with an adjusted (either increased or decreased) emitted light, or a combination of the above. For example, since a red light spectrum is more easily detected by a sensor in the system than a blue light spectrum, the sensor can be adjusted to be less sensitive during a red barrier cycle and more sensitive during a blue barrier cycle since the blue barrier has a lower quantum efficiency with respect to silicon (best illustrated in FIG. 9). Similarly, the emitter can be adjusted to provide an adjusted barrier (e.g., higher or lower intensity and duration). Furthermore, adjustments can be made at both the sensor level and the emitter level. Emitters can also be designed to emit at one particular frequency, or can be modified to emit multiple frequencies of a particular magnitude to broaden the spectrum of light being emitted if desired for a particular application.

[0087] Figure 10 shows a schematic diagram of a non-shared 4T pixel. The TX signal is used to transfer the accumulated charge from the photodiode (PPD) to the floating diffusion (FD). The reset signal is used to reset the FD to the reset bus. If the reset and TX signals are "on" at the same time, the PPD is always reset (each photocharge generated in the PPD is collected directly in the reset bus) and the PPD is always empty. A typical pixel array implementation includes a horizontal reset line that attaches the reset signal of all pixels in a row, and a horizontal TX line that attaches the TX signal of all pixels in a row.

[0088] In one embodiment, the timing of sensor sensitivity adjustment is illustrated and can be achieved using a global reset mechanism (i.e., a means of firing all pixel array reset signals at the same time) and a global TX mechanism (i.e., a means of firing all pixel array TX signals at the same time). This is shown in Figure 11. In this case, the light pulse is constant in duration and amplitude, but the light integrated within all pixels starts with the global TX "on", transitions to "off", and ends with the light pulse. Thus, modulation is achieved by moving the falling edge of the global TX pulse.

[0089] Conversely, the emitter may emit red light at a lesser intensity than the blue light to produce a properly exposed image (best illustrated in FIG. 12). At 1210, data obtained from a histogram from a previous frame may be analyzed. At 1220, the emitter may be adjusted. At 1230, an image may be obtained from the adjusted emitted light. Additionally, in one embodiment, the emitter and sensor may be adjusted simultaneously.

[0090] In some embodiments, reconstructing the split spectrum frame into a full spectrum frame for later output can be as simple as blending the sensed values ​​for each pixel in the array. Additionally, the blending and mixing of values ​​can be a simple average, or can be tailored to a predefined lookup table (LUT) of values ​​for the desired output. In one embodiment of a system using a split light spectrum, the sensed values ​​can be pre-processed or further refined remotely from the sensor by an image or secondary processor and just before being output to a display.

[0091] FIG. 13 illustrates a basic implementation of a monochrome ISP at 1300 and how the ISP chain can be assembled for the purpose of generating an sRGB image sequence from raw sensor data obtained with a GRGB light pulsing scheme.

[0092] The first stage involves making corrections (see 1302, 1304, and 1306 in FIG. 13) to compensate for any non-idealities in the sensor technology that is best suited to function in the raw data domain (see FIG. 21).

[0093] In the next step, each final frame derives data from three raw frames and then buffers two frames (see 1308 and 1310 in FIG. 13). At 1314, we begin to reconstruct the frame with sampled data from the current frame and the two buffered frames (1308 and / or 1310). The reconstruction process results in a full color frame in linear RGB color space.

[0094] In this embodiment, white balance factors at 1318 and a color correction matrix at 1320 are applied before converting to YCbCr space at 1322 for subsequent edge enhancement at 1324. After edge enhancement at 1324, if applicable, the image is converted back to linear RGB at 1326 for scaling at 1328.

[0095] Finally, at 1330, a gamma transfer function is applied to translate the data into the sRGB domain at 1332.

[0096] FIG. 14 is an example of one embodiment of the color fusion hardware. At 1402, the color fusion hardware takes an RGBGRGBGRGBG video data stream and converts it to a parallel RGB video data stream at 1405. The bit width on the input side can be, for example, 12 bits per color. The output width in this example is 36 bits per pixel. Other embodiments can have different initial bit widths and three times the number of output widths. At 1402, the memory writer block takes the RGBG video stream as its input and writes each frame to its correct frame memory buffer at 1404 (the memory writer triggers the same pulse generator 1410 that runs the laser light source). As illustrated at 1404, the write to memory follows the pattern of red, green 1, blue, green 2, and then starting red again. At 1406, the memory reader reads out the three frames simultaneously to build an RGB pixel. Each pixel is three times the bit width of an individual color component. The reader also triggers the laser pulse generator at 1410. The reader waits until the Red, Green 1, and Blue frames have been written and then proceeds to read them while the writer continues writing Green 2 and starts Red again. Once Red is complete, the reader begins reading from Blue, Green 2, and Red. This pattern continues indefinitely.

[0097] 15 and 16, in one embodiment, reconstruction of the RG1BG2RG1BG2 pattern illustrated in FIG. 16 allows for a 60 fps output with a 120 fps input. Each successive frame includes either the red or blue component from the previous frame. In FIG. 16, each color component is available for 8.3 ms, and the resulting reconstructed frame has a duration of 16.67 ms. In general, for this pulsing scheme, the reconstructed frame has a duration twice that of the incident color frame, as shown in FIG. 15. In other embodiments, different pulsing schemes can be employed. For example, an embodiment can be based on the timing of each color component or frame (T1), and the reconstructed frame having a duration twice that of the incident color frame (2×T1). Different frames in the sequence can have different frame durations, and the average capture rate can be a multiple of the final frame rate.

[0098] 17-20 illustrate schematic diagrams of color correction methods and hardware for use with split light systems. In digital imaging, it is common to manipulate values ​​in image data to correct the output to meet user expectations or to emphasize certain aspects of the imaged object. Most commonly, this is done in satellite imagery, which is tuned and adjusted to emphasize one data type over another. Often, satellite acquired data has no controlled light source, i.e., the sun is the light source, so there is the full spectrum of electromagnetic energy available. In contrast, there are imaging conditions where the light is controlled and even provided by the user. In such situations, calibration of the image data is still desirable, since without calibration, inappropriate emphasis may be given to certain data over other data. In systems where the light is controlled by the user, it is advantageous to provide an emission of light that is known to the user and may be only a portion of the electromagnetic spectrum or multiple portions of the entire electromagnetic spectrum. Calibration remains important to meet user expectations and to check for faults in the system. One method of calibration could be a table of expected values ​​for a given imaging condition that can be compared to data from the sensor. One embodiment could include a color-neutral scene with known values ​​to be output by the imaging device, and the device could be adjusted to meet those known values ​​when it samples the color-neutral scene.

[0099] In use, and upon start-up, the system can sample a color-neutral scene at 1710 (as illustrated in FIG. 17) by running a full cycle of a number of electromagnetic spectrum partitions at 1702. At 1704, a table of values ​​1708 can be formed to generate a histogram of the frame. At 1706, the values ​​of the frame can be compared to known or expected values ​​from a color-neutral scene. The imager can then be adjusted at 1712 to meet a desired output. In one embodiment illustrated in FIG. 17, the system can include an image signal processor (ISP) that can adjust the imager to color correct.

[0100] It should be noted that because each split spectrum of light may have a different energy value, the sensor and / or light emitter may be adjusted to compensate for the difference in energy values. For example, in one embodiment, for a silicon-based imager, the blue light spectrum has a lower quantum efficiency than the red light spectrum, so the sensor's response may be adjusted to be less responsive during the red cycle and more responsive to the blue cycle. Conversely, the emitter may emit blue light at a higher intensity, since blue light has a lower quantum efficiency than red light to produce a properly exposed image.

[0101] In one embodiment illustrated in Figure 18 where light source emissions are provided and controllable by the system, adjustments of such light emissions can be made at 1800 to provide color correction of the image. Adjustments can be made to any aspect of the emitted light, such as amplitude, duration (i.e., time on), or range within the spectral divide. Additionally, in some embodiments such as shown in Figure 19, the emitter and sensor can be adjusted simultaneously.

[0102] As can be seen in FIG. 20, fine-grained adjustments can be made to sensors or emitters in the system to reduce the amount of noise and artifacts in the output image stream or video. While FIG. 20 illustrates a system 2000 in which both the emitter 2006 and the sensor 2008 can be adjusted, imaging devices in which either the emitter or the sensor are adjusted during use or over a portion of use are also contemplated and within the scope of this disclosure. It may be advantageous to adjust only the emitter during one portion of use and only the sensor during another portion of use, while also adjusting both simultaneously during a portion of use. In any of the above embodiments, improved image quality can be obtained by limiting the overall adjustments that the system can make during a frame cycle. In other words, an embodiment can limit the emitter to only be able to adjust a portion of its operating range at any time during a frame. Similarly, the sensor can be limited to only be able to adjust a portion of its operating range at any time during a frame. Additionally, an embodiment can limit both the emitter and the sensor to only be able to adjust a portion of their respective operating ranges together at any time during a frame.

[0103] In an exemplary embodiment, the adjustment of a portion of the components in the system may be performed at about 0.1 dB of the component's operating range, for example, to correct for the exposure of the previous frame. Note that this 0.1 dB is merely an example, and that in other embodiments, the possible adjustment of the components may be any portion of their respective operating range. The components of the system may be modified by adjustments in intensity or duration, generally governed by the number of bits (resolution) output by the components. The resolution of the components may typically range from about 10 to 24 bits, but should not be limited to this range, as it is intended to include resolutions of components that are developed in addition to those currently available. For example, if after the first frame, the scene is determined to be too blue when viewed, the emitter may be adjusted to reduce the amplitude or duration of the pulse of blue light during the blue cycle of the system, by a portion adjustment as discussed above, such as about 0.1 dB.

[0104] In this exemplary embodiment, the system limited itself to an adjustment of 0.1 dB of the operating range per system cycle, although more than 10 percent may have been necessary. Thus, the blue light can then be adjusted again during the next system cycle, as necessary. The fine adjustment between cycles can have a damping effect on the output image, and also reduces noise and artifacts when operating the emitter and sensor at their operating extremes. It may be determined that any fractional amount of the adjustment range of the component's operation can be used as a limiting factor, or that a particular embodiment of the system can include components that can be adjusted over the entirety of their operating range.

[0105] In addition, the optical black areas of any image sensor can be used to assist in image correction and noise removal. In one embodiment, values ​​read out from the optical black areas can be compared to values ​​from the active pixel regions of the sensor to establish a reference point for use in processing the image data. FIG. 21 illustrates a type of sensor correction process that can be employed in a color pulsing system. CMOS image sensors typically have several non-idealities that adversely affect image quality, especially in low light. The main ones are fixed pattern noise and line noise. Fixed pattern noise (FPN) is the variation in the offset of the sensing elements. Typically, most FPN is pixel-to-pixel variation caused by random variations in dark current between photodiodes, among other sources. This appears very unnatural to the viewer. Even more egregious is column FPN, which is caused by the offset of the readout chain associated with a particular column of pixels. This results in the perception of vertical stripes in the image.

[0106] Global illumination control has the advantage that an entire frame of dark data can be periodically acquired and used to correct for pixel and column offsets. In the illustrated embodiment, a single frame buffer can be used to perform a running average of the entire frame without illumination, for example using simple exponential smoothing. This dark average frame is subtracted from every illuminated frame during normal operation.

[0107] Line noise is the random time variation of the offset of the pixels in each row. Because it is transient, the correction must be calculated anew for each line and each frame. For this purpose, there are usually a number of optically blind (OB) pixels in each row in the array, which must be sampled first to estimate the line offset before sampling the light-sensitive pixels. Then, during the line noise correction process, the line offset is simply subtracted.

[0108] In the embodiment of FIG. 21, other corrections are made with respect to acquiring the data in the proper order, monitoring and controlling the voltage in the analog domain (black clamp), and identifying / correcting individual bad pixels.

[0109] 22 and 23 illustrate schematics of methods and hardware for enhancing dynamic range in closed or limited light environments. In an embodiment, the exposure input can be at different levels over time and can be combined to generate a larger dynamic range. As can be seen in FIG. 22, the imaging system can be cycled at a first intensity for a first cycle, at 2202, then cycled at a second intensity for a second cycle, at 2204, and then cycled by combining the first and second cycles into a single frame, at 2206, so that a larger dynamic range can be achieved. A larger dynamic range can be particularly desirable because the spatial environment in which the imaging device is used is limited. In a limited spatial environment that is light-poor or dark except for the light provided by the light source, and when the light source is close to the light emitter, the exposure has an exponential relationship with the distance. For example, objects near the light source and the optical aperture of the imaging device tend to be overexposed, while objects far away tend to be very underexposed since there is little (if any) ambient light.

[0110] 23, at 2300, the cycle of a system having emission of electromagnetic energy in multiple partitions can be cycled continuously according to the partitions of the electromagnetic spectrum. For example, in an embodiment in which an emitter emits lasers with different red partitions, different blue partitions, and different green partitions, the two sets of cycle data that are combined can be in the following form: i.2302 red with intensity 1, ii.2304 red with intensity 2; iii.2302 with blue intensity 1; iv.2304 is blue with intensity 2, v.2302 green with intensity 1, vi.2304 green intensity 2

[0111] Alternatively, the system can be cycled in the following manner. i.2302 red with intensity 1, ii.2302 with blue intensity 1; iii.2302 with green intensity 1; iv.2304 is red with intensity 2, v.2304 blue with intensity 2, vi.2304 green intensity 2

[0112] In one such embodiment, a first image may be derived from one intensity value and a second image may be derived from two intensity values, which may then be combined or processed at 2310 as a complete image data set rather than their constituent parts.

[0113] It is contemplated within the scope of this disclosure that any number of emission septa may be used in any order. As seen in Figure 23, "n" is used as a variable to represent any number of electromagnetic septa and "m" is used to represent any level of strength of the "n" septa. Such a system may be cycled as follows: In i.2306, n of intensity m, ii. n+1 of intensity m+1; iii. n+2 with intensity m+2; In iv.2308, n+i with intensity m+j.

[0114] Thus, any pattern of serialized cycles can be used to produce the desired image correction, where "i" and "j" are additional values ​​within the operating range of the imaging system.

[0115] Digital color cameras incorporate an image processing stage to maximize the fidelity of color reproduction, which is achieved by a 3 × 3 matrix known as the Color Correction Matrix (CCM).

[0116]

number

[0117] The terms in the CCM are adjusted using a set of reference colors (e.g., from the Macbeth chart) to provide the best overall match to the sRGB standard color space. The diagonal terms, a, e, and i, are effective white balance gains. However, typically white balance is applied separately and the sum of the horizontal rows is constrained to be unity so that no net gain is applied by the CCM itself. The off-diagonal terms effectively address color crosstalk within the input channels. Thus, Bayer sensors have higher off-diagonal gains than 3-chip cameras have channels because color filer arrays have a lot of response overlap between channels.

[0118] There is a signal-to-noise ratio penalty for color correction that depends on the amplitude of the off-diagonal terms. A hypothetical sensor with channels perfectly matched to the sRGB components has the identity matrix CCM:

[0119]

number

[0120] The signal-to-noise ratio estimated in the green channel, for a perfect white light signal of 10,000 e / pixel (neglecting readout noise) in this case, is:

[0121]

number

[0122] Any deviation from this will degrade the SNR. For example, consider the CCM with values ​​that are not unusual for a Bayer CMOS sensor:

[0123]

number

[0124] In this case, the green SNR is:

[0125]

number

[0126] Figure 24 shows the overall SNR simulation results using D65 illumination for the CCM of a typical Bayer sensor when using the identity matrix versus the adjusted CCM. The SNR evaluated for the luminance component is approximately 6 dB worse as a result of the color correction.

[0127] Since the system described in this disclosure uses monochromatic illumination of multiple discrete wavelengths, there is naturally no color crosstalk. The X's in Figure 25 indicate the location of the three wavelengths available via the laser diode source (465, 532, and 639 nm) compared to the sRGB range shown by the triangle.

[0128] In this case, the off-diagonal terms of the CCM are significantly reduced compared to the Bayer sensor, providing a significant SNR advantage.

[0129] 26 illustrates an imaging system with increased dynamic range provided by the pixel configuration of the pixel array of the image sensor. As can be seen, adjacent pixels 2602 and 2604 can be set to different sensitivities, such that each cycle contains data generated by pixels that are more and less sensitive to each other. Because multiple sensitivities can be recorded in a single cycle of the array, as opposed to the time-dependent sequential nature of other embodiments when recorded in parallel, the dynamic range can be increased.

[0130] In one embodiment, the array may comprise rows of pixels that may be arranged in the rows based on their sensitivity. In one embodiment, pixels of different sensitivity may be staggered in rows or columns relative to their nearest neighbors based on their sensitivity throughout the array to form a checkerboard pattern. This may be accomplished through any shared pixel circuit arrangement or with any independent pixel circuit arrangement.

[0131] A wide dynamic range can be achieved by having multiple global TXs, each firing only to a different set of pixels. For example, in global mode, the global TX1 signal lights the first set of pixels, the global TX2 signal lights the second set of pixels, ..., the global TXn signal lights the nth set of pixels.

[0132] Based on FIG. 11, FIG. 27A shows an example of timing for two different pixel sensitivities (dual pixel sensitivity) in a pixel array. In this case, the global TX1 signal lights up half of the pixels of the array and the global TX2 lights up the other half. The integrated light is different between the TX1 and TX2 pixels because the global TX1 and global TX2 have different edge positions from "on" to "off". FIG. 27B shows a different embodiment of timing for dual pixel sensitivity. In this case, the light pulse is modulated twice (pulse duration and / or amplitude). The TX1 pixel integrates the P1 pulse and the TX2 pixel integrates the P1+P2 pulse. The separation of the global TX signal can be done in a number of ways. Below are examples: i. Distinguish TX lines from each row, ii. Send multiple TX lines per row, each addressing a different set of pixels.

[0133] In one implementation, a means of providing wide dynamic range video is described that utilizes the color pulsing system described in this disclosure. The basis of this means is to have multiple flavors of pixels, or pixels that can be adjusted differently, in the same monochrome array that can integrate incident light for different durations in the same frame. One example of pixel arrangement in the array of such a sensor is a uniform checkerboard pattern throughout, with two independent variable integration times. In such a case, both red and blue information can be provided in the same frame. In fact, since the two integration times can be adjusted on a frame-by-frame basis, this can be done simultaneously to extend the dynamic range of the green frame when it is most needed. The advantage is that color motion artifacts are less of an issue when all data is derived from two frames versus three. Of course, there is a subsequent loss of spatial resolution of the red and blue data, but this does not affect image quality as much as compared to green, since the green data accounts for the luminance component.

[0134] An inherent characteristic of monochrome wide-dynamic range (WDR) arrays is that pixels with long integration times must integrate a superset of the light seen by pixels with short integration times. This is desirable for standard wide-dynamic range operation in green frames. For red and blue frames, pulsing must be controlled in conjunction with the exposure period to provide blue light from the beginning of the long exposure, for example, and switch to red when the short exposure pixels are turned on (both pixel types have their charge transferred at the same time).

[0135] In the color blending stage, the two flavors of a pixel are separated into two buffers. Empty pixels are then filled, for example using linear interpolation. At this point, one buffer contains the blue data, and the other contains the full red+blue image. The blue buffer can be subtracted from the second buffer to provide pure red data.

[0136] 28A-C illustrate the use of white light emission that is pulsed and / or synchronized with a corresponding color sensor or held constant. As can be seen in FIG. 28A, the white light emitter can be configured to emit a beam of light during the blanking period of the corresponding sensor to provide a controlled light source in a controlled light environment. The light source can emit a beam with a constant amplitude and vary the duration of the pulse as seen in FIG. 28A, or the pulse can be kept constant while varying the amplitude to achieve the correct exposure data as illustrated in FIG. 28B. FIG. 28C illustrates a graphical representation of a constant light source that can be adjusted with varying current controlled and synchronized with a sensor.

[0137] In one embodiment, white light or multi-spectral light can be emitted as pulses, as desired, to provide data for use within the system (best illustrated in Figures 28A-C). White light emission combined with a range of electromagnetic spectrum partitions can be useful for highlighting and de-emphasizing certain aspects within a scene. Such an embodiment can use the following pulsing patterns: i. green pulse, ii. red pulse; iii. Blue pulse; iv. green pulse; v. Red pulse, vi. Blue pulse, vii. White light (multispectral) pulses; viii.(Repeat)

[0138] Any system that uses an image sensor cycle that is at least twice as fast as the white light cycle is intended to fall within the scope of this disclosure. It will be understood that any combination of partitions of the electromagnetic spectrum, whether from the visible or invisible spectrum of the entire electromagnetic spectrum, is contemplated herein.

[0139] 29A and 29B illustrate perspective and side views, respectively, of an implementation of a monolithic sensor 2900 having multiple pixel arrays for generating a three-dimensional image in accordance with the teachings and principles of the present disclosure. Such an implementation may be desirable for three-dimensional image capture, where the two pixel arrays 2902 and 2904 may be offset during use. In another implementation, the first pixel array 2902 and the second pixel array 2904 may be dedicated to receiving electromagnetic radiation in a predetermined range of wavelengths, with the first pixel array dedicated to a different range of wavelengths of electromagnetic radiation than the second pixel array.

[0140] 30A and 30B respectively illustrate perspective and side views of an implementation of an image sensor 3000 built on multiple substrates. As illustrated, a number of pixel columns 3004 forming a pixel array are located on a first substrate 3002, and a number of circuit columns 3008 are located on a second substrate 3006. Also illustrated in the figure are electrical connections and communications between one pixel column and its associated or corresponding circuit columns. In one implementation, an image sensor that may otherwise be fabricated with the pixel array and supporting circuitry on a single monolithic substrate / chip may have the pixel array separated from all or most of the supporting circuitry. The present disclosure may use at least two substrates / chips that are stacked on top of each other using three-dimensional stacking techniques. The first 3002 of the two substrates / chips may be processed using an imaging CMOS process. The first substrate / chip 3002 may be composed of either exclusively a pixel array or a pixel array surrounded by limited circuitry. The second or subsequent substrate / chip 3006 can be processed using any process and does not have to be by imaging CMOS process. The second substrate / chip 3006 can be a high density digital process to integrate various and several functions in a very limited space or area on the substrate / chip, or a mixed mode or analog process, for example to integrate precise analog functions, or an RF process to implement wireless capabilities, or a MEMS (Micro-Electro-Mechanical Systems) device, but not limited to these. The imaging CMOS substrate / chip 3002 can be stacked with the second or subsequent substrate / chip 3006 using any three-dimensional technique. The second substrate / chip 3006 can support most or most of the circuits that would otherwise be implemented in the first imaging CMOS chip 3002 as peripheral circuits (if implemented on a monolithic substrate / chip), thus increasing the overall system area while keeping the pixel array size constant and optimized to the greatest extent possible.Electrical connections between the two substrates / chips can be made through interconnects 3003 and 3005, which can be wirebonds, bumps and / or TSVs (Through Silicon Vias).

[0141] 31A and 31B respectively illustrate perspective and side views of an implementation of an imaging sensor 3100 having multiple pixel arrays for generating a three-dimensional image. The three-dimensional image sensor can be constructed on multiple substrates and can include multiple pixel arrays and other associated circuitry, with a plurality of pixel columns 3104a forming a first pixel array and a plurality of pixel columns 3104b forming a second pixel array located on respective substrates 3102a and 3102b, respectively, and a plurality of circuit columns 3108a and 3108b located on a separate substrate 3106. Also shown are electrical connections and communications between the pixel columns and the associated or corresponding circuitry columns.

[0142] It will be appreciated that the teachings and principles of the present disclosure may be used in a reusable device platform, a limited use device platform, a relocatable use device platform, or a single use / disposable device platform without departing from the scope of the present disclosure. It will be appreciated that in a reusable device platform, the end user is responsible for cleaning and sterilizing the device. In a limited use device platform, the device can be used a specified number of times before it becomes inoperable. A typical new device is supplied sterile with further use requiring the end user to clean and sterilize. In a relocatable use device platform, a third party can reprocess (e.g., clean, package, and sterilize) the device into a single use device for further use at a lower cost than a new unit. In a single use / disposable device platform, the device is provided sterile to the operating room and is used only once before disposal.

[0143] One embodiment of the emitter can employ the use of mechanical shutters and filters to create pulsed color light. As illustrated in FIG. 32, an alternative method for generating pulsed color light uses a system 3200 of white light sources, mechanical color filters, and shutters. The wheel can include a pattern of semi-transparent color filter windows and opaque sections for shuttering. The opaque sections do not allow light to pass through and create dark periods during which the sensor readout occurs. The white light source can be based on any technology, laser, LED, xenon, halogen, metal halide, or other. The white light can be projected through a series of color filters 3207, 3209, and 3211 of colored light pulses in the desired pattern. The pattern in one embodiment can be red filter 3207, green filter 3209, blue filter 3211, green filter 3209. The filter and shutter system 3200 can be arranged on a wheel that rotates at the required frequency to synchronize with the sensor, such that knowledge of the arch length and rotational speed of the system of mechanical color filters 3207, 3209, and 3211, and shutter 3205 provides timing information for the operation of a corresponding monochrome image sensor.

[0144] The embodiment illustrated in FIG. 33 may include a pattern of only semi-transparent color filters 3307, 3309, and 3311 on the filter wheel 3300. Different shutters may be used in this configuration. The shutters may be mechanical and may dynamically adjust the "pulse" duration by varying the size. Alternatively, the shutters may be electronic and may be incorporated into the sensor design. The motor that rotates the filter wheel 3300 needs to communicate with or be controlled in conjunction with the sensor, such that knowledge of the arch length and rotational speed of the system of mechanical color filters 3307, 3309, and 3311 provides timing information for the operation of the corresponding monochrome image sensor. The control system will need to know the appropriate color filter for each frame captured by the sensor so that a full color image can be properly reconstructed in the ISP. Although an RGBG color pattern is shown, other colors and / or patterns may be used if advantageous. The relative sizes of the color sections are shown as equal, but may be adjusted if advantageous. Although the filter mechanical structure is shown as a rotationally moving circle, it can be a rectangle with linear movement, or a different shape with a different movement pattern.

[0145] As illustrated in FIG. 34, an embodiment for pulsing colored light can consist of a mechanical wheel or barrel that holds electronics and heat sinks for red, green, blue, or white LEDs. The LEDs are spaced at a distance related to the rotational or twisting speed of the barrel or wheel to allow for timed light pulsing consistent with other embodiments in this patent. The wheel or barrel is rotated using an electric motor and a mechanical bracket that attaches the wheel or barrel to the electric motor. The motor is controlled using a microcontroller, FPGA, DSP, or other programmable device that includes a control algorithm for proper timing as described in this patent. On one side there is a mechanical opening that is optically coupled to the optical fiber to transport the fiber to the end of the scope by the methods described in this patent. This coupling can also have a mechanical opening that can be opened and closed to control the amount of light that is allowed down the fiber optic cable. This is a mechanical shutter device, alternatively an electronic shutter designed into a CMOS or CCD type sensor can be used. This device is difficult to control and calibrate during manufacturing, but is another way that pulsed light can be passed through the system.

[0146] 35 illustrates one embodiment of an emitter 3502 with a linear filter 3504 and shuttering mechanism to provide pulsed electromagnetic radiation. The linear filter 3504 and shuttering mechanism move horizontally at the required frequency to filter out the appropriate wavelength of light.

[0147] 36 illustrates one embodiment of an emitter 3602 with a prism filter 3604 and shutter mechanism to provide pulsed electromagnetic radiation. The prism filter 3604 filters the light and delivers it to an output that may include a shutter. The prism filter 3604 shifts at the required frequency to provide the correct color output pattern.

[0148] Additionally, the teachings and principles of the present disclosure may include any and all wavelengths of electronic energy including the visible and non-visible spectrum, such as infrared (IR), ultraviolet (UV), and x-ray.

[0149] FIG. 37 is a schematic diagram illustrating a system 3700 for providing illumination to a low light environment, such as for endoscopic imaging. The system 3700 can be used in combination with any of the systems, methods, or devices disclosed herein. The system 3700 includes a light source 3702, a controller 3704, a jumper waveguide 3706, a waveguide connector 3708, a lumen waveguide 3710, a lumen 3712, and an image sensor 3714 with associated optical components (such as lenses). The light source 3702 generates light that travels through the jumper waveguide 3706 and the lumen waveguide 3710 to illuminate a scene at the distal end of the lumen 3712. The light source 3700 can be used to emit electromagnetic energy of any wavelength, including visible, infrared, ultraviolet, and other wavelengths. The lumen 3712 can be inserted into a patient's body for imaging, such as during a procedure or examination. The light is output as illustrated by dashed line 3716. The scene illuminated by the light can be captured using image sensor 3714 and displayed to a physician or some other medical personnel. Controller 3704 can provide control signals to light source 3702 to control when illumination is provided to the scene. In one embodiment, light source 3702 and controller 3704 are located in a camera control unit (CCU) or an external console to which the endoscope is connected. If image sensor 3714 includes a CMOS sensor, light can be provided to the scene periodically in a series of illumination pulses during a readout period of image sensor 3714 during what are known as blanking periods. Thus, the light can be pulsed in a controlled manner to avoid overlapping of image pixels in the pixel array of image sensor 3714 onto the readout period.

[0150] In one embodiment, the lumen waveguide 3710 includes one or more optical fibers. The optical fibers can be made of low-cost materials, such as plastic, to allow for disposal of the lumen waveguide 3710 and / or other parts of the endoscope. In one embodiment, a single glass fiber with a diameter of 500 microns can be used. The jumper waveguide 3706 can be permanently attached to the light source 3702. For example, the jumper waveguide 3706 can receive light from an emitter in the light source 3702 and provide the light to the lumen waveguide 3710 at the connector 3708. In one embodiment, the jumper waveguide 106 can include one or more glass fibers. The jumper waveguide can include any other type of waveguide for guiding light to the lumen waveguide 3710. The connector 3708 selectively couples the jumper waveguide 3706 to the lumen waveguide 3710 and allows light in the jumper waveguide 3706 to pass to the lumen waveguide 3710. In one embodiment, the lumen waveguide 3710 can be directly coupled to a light source without any intervening jumper waveguide 3706.

[0151] 38-40 are schematic block diagrams illustrating a light source 3800 having multiple emitters. With reference to FIG. 38, the emitters include a first emitter 3802, a second emitter 3804, and a third emitter 3806. Additional emitters may be included as discussed further below. The emitters 3802, 3804, and 3806 may include one or more laser emitters that emit light having different wavelengths. For example, the first emitter 3802 may emit a wavelength corresponding to a blue laser, the second emitter 3804 may emit a wavelength corresponding to a green laser, and the third emitter 3806 may emit a wavelength corresponding to a red laser. For example, a first emitter 3802 can include one or more blue lasers, a second emitter 3804 can include one or more green lasers, and a third emitter 3806 can include one or more red lasers. The emitters 3802, 3804, 3806 emit laser beams toward a collection region 3808, which can be at the location of a waveguide, lens, or other optical component to collect and / or provide light to a waveguide, such as the jumper waveguide 3706 or lumen waveguide 3710 of FIG.

[0152] In one implementation where a patient has been administered a reagent or dye to aid in the identification of a particular tissue, structure, chemical reaction, biological process, etc., emitters 3802, 3804, and 3806 can emit wavelength(s) to cause the reagent or dye to fluoresce. Such wavelength(s) can be determined based on the reagent or dye administered to the patient. In such an embodiment, the emitters may need to be very precise to emit the desired wavelength(s) to cause or activate the particular reagent or dye to fluoresce.

[0153] In the embodiment of FIG. 38, emitters 3802, 3804, 3806 each deliver laser light at a different angle to the collection region 3808. Variations in the angle can result in variations when the electromagnetic energy is located in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at the collection region 3808, variations in the angle can cause different amounts of light to enter different fibers. For example, the angles can result in variations in intensity across the collection region 3808. Furthermore, since the light from different emitters cannot mix uniformly, some fibers may receive different amounts of light of different colors. Variations in the color or intensity of the light in the different fibers can lead to less than optimal illumination of the scene. For example, variations in the delivered light or light intensity can occur in the scene and the captured image.

[0154] In one embodiment, intervening optics can be placed between the fiber bundles and the emitters 3802, 3804, 3806 to mix different colors (wavelengths) of light before entering the fibers or other wave guides. Exemplary intervening optics include diffusers, mixing rods, one or more lenses, or other optical components that mix the light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in a fiber bundle can have the same color. This mixing can result in the same color for each fiber, but in some embodiments, can still result in different total brightness being delivered to different fibers. In one embodiment, the intervening optics can also spread or even out the light over the collection area (e.g., the light can be diffused with a top hat profile) so that each fiber carries the same total amount of light. The diffuser or mixing rod can lead to light loss.

[0155] 38 as a physical component, the collection region 3808 may simply be the area through which light from the emitters 3802, 3804, and 3806 is delivered. In some cases, the collection region 3808 may include optical components, such as a diffuser, mixing rod, lens, or any other intervening optical components between the emitters 3802, 3804, 3806 and the output waveguide.

[0156] 39 illustrates an embodiment of a light source 3800 having emitters 3802, 3804, 3806 that provide light to a collection area 3808 at the same or substantially the same angle. The light is provided at an angle that is substantially perpendicular to the collection area 3808. The light source 3800 includes multiple dichroic mirrors, including a first dichroic mirror 3902, a second dichroic mirror 3904, and a third dichroic mirror 3906. The dichroic mirrors 3902, 3904, 3906 include mirrors that reflect light of a first wavelength but transmit (or are transmissive) light of a second wavelength. For example, the third dichroic mirror 3906 can reflect blue laser light provided by the third emitter while transmitting red and green light provided by the first and second emitters 3802 and 3804, respectively. The second dichroic mirror 3904 may transmit red light from the first emitter 3802 but reflect green light from the second emitter 3804. If other colors or wavelengths are included, the dichroic mirror may reflect light corresponding to at least one emitter and transmit light corresponding to other emitters. For example, the third dichroic mirror 3906 may reflect light from the third emitter 3806 but reflect the light to "back" emitters such as the first emitter 3802 and the second emitter 3804. In embodiments where there are tens or hundreds of emitters, each dichroic mirror may reflect to a corresponding emitter or emitters in front of it while transmitting to emitters behind it. This allows tens or hundreds of emitters to emit electromagnetic energy into the collection area 3808 at substantially the same angle.

[0157] The dichroic mirror allows other wavelengths to be transmitted or passed so that each of the wavelengths can arrive at the collection area 3808 from the same angle and / or at the same center or focus. Providing light from the same angle and / or the same focus / center point can significantly improve light reception and color mixing in the collection area 3808. For example, a particular fiber can receive different colors in the same ratio as transmitted / reflected by the emitters 3802, 3804, 3806 and mirrors 3902, 3904, 3906. Light mixing can be significantly improved in the collection area compared to the embodiment of FIG. 38. In an embodiment, any of the optical components discussed herein can be used in the collection area 3808 to collect light before providing it to the fiber or fiber bundle.

[0158] FIG. 40 also illustrates an embodiment of a light source 3800 having emitters 3802, 3804, 3806 that provide light to a collection region 3808 at the same or substantially the same angle. However, the light incident on the collection region 3808 is offset from normal. Angle 4002 indicates the angular offset from normal. In one embodiment, the laser emitters 3802, 3804, 3806 can have a cross-sectional intensity profile that is Gaussian. As previously discussed, improved distribution of optical energy between fibers can be achieved by creating a flatter or top-hat shaped intensity profile. In one embodiment, as angle 4002 increases, the intensity across the collection region 3808 approaches a top-hat profile. For example, a non-flat output beam can be made to approximate a top-hat profile by increasing angle 4002 until the profile is sufficiently flat.

[0159] A top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intervening optical components between the emitters 3802, 3804, 3806 and the output waveguides or fiber optic bundles.

[0160] 41 is a schematic diagram illustrating a single optical fiber 4102 outputting through a diffuser 4104 at the output. In one embodiment, the optical fiber 4102 may have a diameter of 500 microns and a numerical aperture of 0.65, emitting a light cone 4106 of about 70 degrees or 80 degrees without the diffuser 4104. With the diffuser 4104, the light cone 4106 may have an angle of about 110 degrees or 120 degrees. The light cone 4106 may be the majority of where all the light travels and is evenly distributed. The diffuser 4104 may allow for a more even distribution of electromagnetic energy of the scene observed by the image sensor.

[0161] In one embodiment, the lumen waveguide 4102 may include a single plastic or glass optical fiber of about 500 microns. Although plastic fibers may be low cost, their width may allow the fiber to carry a sufficient amount of light to the scene due to couplers, diffusers, or other losses. For example, smaller fibers may not be able to carry as much light or power as larger fibers. The lumen waveguide 3710 may include a single or multiple optical fibers. The lumen waveguide 3702 may receive light from a light source directly or through a jumper waveguide (see, for example, jumper waveguide 3706 in FIG. 37). A diffuser may be used to widen the light output 3706 for a desired field of view of the image sensor 3714 or other optical components.

[0162] Although three emitters are shown in Figures 38-40, in some embodiments anywhere from one to hundreds or more of emitters can be used. The emitters can have different wavelengths or spectrums of light that they emit that can be used to continuously cover desired portions of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectrum).

[0163] In one embodiment, a light source with multiple emitters can be used for multispectral or hyperspectral imaging in low light environments. For example, different chemicals, materials, or tissues may have different responses to different colors or wavelengths of electromagnetic energy. Some tissues have their own spectral signatures (how the tissue reacts or varies in reflected wavelengths of electromagnetic radiation). In one embodiment, a particular type of tissue can be detected based on how the tissue reacts to a particular wavelength or a particular combination of wavelengths. For example, vascular tissue can absorb and reflect electromagnetic energy of different wavelengths or spectra in a unique way that distinguishes it from muscle, fat, bone, nerve, ureter, or other tissues or materials in the body. Furthermore, a particular type of muscle or other type of tissue can be distinguished based on their spectral response. Disease status of tissues can also be determined based on spectral information. See U.S. Pat. No. 8,289,503. See also U.S. Pat. No. 8,158,957.

[0164] In one embodiment, multispectral or hyperspectral image data can be obtained by filtering all light or electromagnetic energy except that at desired wavelengths or spectra using one or more filters. Figure 42 is a block diagram illustrating a filter 4202 for removing unwanted wavelengths before the light 4208 (or other electromagnetic radiation) encounters an imaging sensor 4204 or other imaging medium (e.g., film). In one embodiment, white light 4208 passes through the filter 4202 and the filtered light 4210 passes through a lens 4206 that focuses it onto the imaging sensor 4204 for image capture and readout. The filter can be located anywhere in the system or can be an attachment to the lens 4206 or image sensor 4204.

[0165] In a low light environment, the light 4208 can include white light emitted by an emitter in a low light environment. The filter 4202 can be selected for the desired inspection. For example, if it is desired to detect or highlight a particular tissue, the filter 4202 can be selected to pass wavelengths corresponding to the spectral response of the particular tissue. The image sensor 4204, which can include a monochrome image sensor, can generate an image. Pixels of the captured image that are above or below a threshold can then be characterized as corresponding to the particular tissue. This data can then be used to generate an image that indicates the location of the particular tissue.

[0166] In another embodiment, a fluorescent dye can be used to image a particular tissue type, pathway, etc. within the body. For example, a fluorescent dye can be administered to a patient and then an image of the dye can be captured. In one embodiment, causing the dye to fluoresce can be triggered using electromagnetic energy of a particular wavelength. For example, the dye can fluoresce only in the presence of electromagnetic energy.

[0167] However, both filters and fluorescent dyes greatly limit the examination. For example, when using filters, the desired spectral response that can be detected, and therefore the materials or tissues that can be detected, are limited by the available filters. Furthermore, filters may need to be replaced or exchanged. With dyes, the dye must be administered before imaging, and there may be competition between the administration of different dyes for different purposes during the same examination. Thus, examinations using filters and dyes may take a long time and may require many different examinations to obtain the desired information.

[0168] In one embodiment, multispectral or hyperspectral imaging in low light environments can be achieved using monochrome image sensors and emitters that emit electromagnetic energy at multiple different wavelengths or spectra. In one embodiment, a light source (such as light source 3800 of any of FIGS. 38-40) or other electromagnetic source can include multiple emitters to cover a desired spectrum.

[0169] FIG. 43 illustrates a portion of the electromagnetic spectrum 4300 divided into 20 different subspectra. The number of subspectra is exemplary only. In at least one embodiment, the spectrum 4300 can be divided into hundreds of subspectra, each having a small wavelength band. The spectrum can range from an infrared spectrum 4302, through a visible spectrum 4304, to an ultraviolet spectrum 4306. The subspectra have wavelength bands 4308 that each cover a portion of the spectrum 4300. Each wavelength band can be defined by an upper wavelength and a lower wavelength.

[0170] In one embodiment, at least one emitter (such as a laser emitter) can be included in a light source (such as light sources 3702, 3800 in Figures 37-40) for each subspectrum to provide complete and continuous coverage of the entire spectrum 4300. For example, a light source for providing the illustrated subspectral coverage can include at least 20 different emitters, at least one for each subspectrum. In one embodiment, each emitter can cover a spectrum covering 40 nanometers. For example, one emitter can emit light in a wavelength range of 500 nm to 540 nm, while another emitter can emit light in a wavelength range of 540 nm to 580 nm. In another embodiment, the emitters can cover other size wavelength ranges depending on the type of emitters available or the imaging needs. For example, the multiple emitters may include a first emitter covering a wavelength range of 500-540 nm, a second emitter covering a wavelength range of 540 nm-640 nm, and a third emitter covering a wavelength range of 640 nm-650 nm. Each emitter may cover a different slice of the electromagnetic spectrum across the far infrared, mid infrared, near infrared, visible, near ultraviolet, and / or extreme ultraviolet. In some cases, multiple emitters of the same type or wavelength may be included to provide sufficient power for imaging. The number of emitters required for a particular wavelength range may depend on the sensitivity of the monochromatic sensor to that wavelength range and / or the power output capabilities of the emitters in that wavelength range.

[0171] The width and coverage of the wavelength bands provided by the emitters can be selected to provide any desired combination of spectra. For example, continuous coverage of the spectrum using very narrow wavelength band widths (e.g., 10 nm or less) can enable highly selective hyperspectral imaging. Because the wavelengths come from emitters that can be selectively activated, maximum flexibility can be achieved in determining the spectral response of materials during an examination. Thus, more information about the spectral response that would require multiple examinations, delayed by the administration of dyes or stains, etc. can be achieved in less time and within a single examination. In one embodiment, the system can capture hyperspectral image data and process the data to identify what type of tissue is present at each pixel.

[0172] 44 is a schematic diagram illustrating a timing diagram 4400 for emission and readout to generate a multispectral or hyperspectral image, according to one embodiment. The solid lines represent readouts (peaks 4402) and blanking periods (valleys) for capturing a series of frames 4404-4414. The series of frames 4404-4414 may include a repeating series of frames that can be used to generate hyperspectral data for a video feed. The series of frames includes a first frame 404, a second frame 4406, a third frame 4408, a fourth frame 4410, a fifth frame 4412, and an nth frame 4426.

[0173] In one embodiment, each frame is generated based on at least one pulse of electromagnetic energy. The pulse of electromagnetic energy is reflected and detected by an image sensor, and then read out in a subsequent readout (4402). Thus, each blanking period and readout results in an image frame of a particular range of electromagnetic energy. For example, the first frame 404 can be generated based on the spectrum of the first one or more pulses 4416, the second frame 4406 can be generated based on the spectrum of the second one or more pulses 4418, the third frame 4408 can be generated based on the spectrum of the third one or more pulses 4420, the fourth frame 4410 can be generated based on the spectrum of the fourth one or more pulses 4422, the fifth frame 4412 can be generated based on the spectrum of the fifth one or more pulses 4424, and the nth frame 4426 can be generated based on the spectrum of the nth one or more pulses 4426.

[0174] Pulses 4416-4426 may include energy from a single emitter or from a combination of two or more emitters. For example, the spectra included within a single readout period or within multiple frames 4404-4414 may be selected for the desired examination or detection of a particular tissue or condition. According to one embodiment, one or more pulses may include visible spectrum light to generate a color or black and white image, while one or more additional pulses are used to obtain a spectral response to classify a tissue type. For example, pulse 4416 may include red light, pulse 4418 may include blue light, and pulse 4420 may include green light, while the remaining pulses 4422-4426 may include wavelengths and spectra to detect a particular tissue type. As a further example, the pulses of a single readout period may include spectra (e.g., different slices of the electromagnetic spectrum) generated from multiple different emitters that may be used to detect a particular tissue type. For example, if a combination of wavelengths results in a pixel having a value above or below a threshold, the pixel may be classified as corresponding to a particular type of tissue. Each frame can be used to further refine the type of tissue present at that pixel (e.g., each pixel in an image) to provide a very specific classification of tissue and / or tissue status (diseased / healthy) based on the spectral response.

[0175] The multiple frames 4404-4414 are shown having readout periods of varying length, and pulses having different lengths or intensities. The blanking periods, pulse lengths or intensities, etc., can be selected based on the sensitivity of the monochromatic sensor to a particular wavelength, the power output capabilities of the emitter(s), and / or the carrying capabilities of the waveguide.

[0176] A hyperspectral image or hyperspectral image data acquired in the manner illustrated in Figure 44 can result in multiple frames, each based on a different spectrum or combination of spectra. In some cases, tens or hundreds of different frames can be acquired. In other cases, such as for video streams, some frames may be limited to provide a viewable frame rate. Because a combination of different spectra can be provided within a single readout period, useful and dynamic spectral information can still be acquired, even for video streams.

[0177] In one embodiment, the video or other image may include a black and white or color image overlaid with information derived from the spectral response of each pixel. For example, pixels corresponding to a particular tissue or condition may be shown in a bright green or other color to assist a physician or other medical professional during an examination.

[0178] In one embodiment, dual image sensors can be used to obtain three-dimensional images or video feeds. Three-dimensional inspection can allow for improved understanding of the three-dimensional structure of the examined area as well as mapping of different tissue or material types within the area.

[0179] In one embodiment, multispectral or hyperspectral imaging can be used to see through materials or substances. For example, infrared wavelengths can pass through some tissues, such as muscle or fat, while reflecting off blood vessels. In one embodiment, infrared waves can penetrate 5, 8, or 10 mm or more into tissue. Acquiring a series of frames, including at least one infrared frame, can allow the inspection to provide information regarding the location of blood vessels below the surface. This can be extremely useful in surgical procedures where it may be desirable to perform an incision that avoids blood vessels. In one embodiment, a color or grayscale image can be overlaid with a green color indicating the location of blood vessels below the surface. Similarly, the known spectral response of blood can be used to see through the blood of a subject under inspection to see tissues or structures.

[0180] Assembly of the sub-frames into a single frame for display on a monitor or other display device can occur after capturing the sequence of frames 4404-4414. A color or grayscale image can be generated from one or more of the frames, and pixel overlay information can be determined based on all or the remaining frames. The color or grayscale image can be combined with the overlay information to generate a single frame. The single frame can be displayed as a single image or as an image in a video stream.

[0181] In one embodiment, the hyperspectral data acquired as illustrated in FIG. 44 can be provided to a third party algorithm for analysis to classify the tissue or material captured in the image. In one embodiment, the third party algorithm can be used to select the spectrum or wavelength bands used during imaging so that a desired spectral response analysis can be performed. In one embodiment, the spectral response analysis can be performed in real time during a medical imaging procedure or other medical procedure. The spectral data can be overlaid on an RGB or black and white image so that a user can easily distinguish between specific types of tissue, organs, chemical processes, diseases, etc. In one embodiment, the spectral data can be provided to a computer operated system such as a robotic system for automating medical imaging or medical procedures.

[0182] 45 is a schematic diagram of an imaging system 4500 with a single cut filter. The system 4500 includes an endoscope 4506 or other suitable imaging device having a light source 4508 for use in low light environments. The endoscope 4506 includes an image sensor 4504 and a filter 4502 for filtering unwanted wavelengths of light or other electromagnetic radiation before reaching the image sensor 4504. The light source 4508 transmits light that can illuminate a surface 4512 in a low light environment, such as a body cavity. The light 4510 reflects off the surface 4512 and passes through the filter 4502 before hitting the image sensor 4504.

[0183] Filter 4502 may be used in an implementation where a fluorescent reagent or dye is administered. In one such embodiment, filter 4502 is configured to filter all light or other electromagnetic radiation except light of one or more desired wavelengths or spectral bands. In one embodiment, filter 4502 is configured to filter electromagnetic radiation at an excitation wavelength that causes the reagent or dye to fluoresce, such that only the expected relaxation wavelengths of the fluorescent reagent or dye are allowed to pass through filter 4502 and reach image sensor 4504. In one embodiment, filter 4502 filters at least fluorescent reagent excitation wavelengths between 770 nm and 790 nm. In one embodiment, filter 4502 filters at least fluorescent reagent excitation wavelengths between 795 nm and 815 nm. In one embodiment, filter 4502 filters at least fluorescent reagent excitation wavelengths between 770 nm and 790 nm and between 795 nm and 815 nm. In these embodiments, the filter 4502 filters out the excitation wavelengths of the reagent, allowing only the relaxation wavelengths of the fluorescent reagent to be read out by the image sensor 4504. The image sensor 4504 can be a wavelength-blind image sensor, and the filter 4502 can be configured to allow the image sensor 4504 to receive only the relaxation wavelengths of the fluorescent reagent and not the emission excitation wavelengths of the reagent. The data determined by the image sensor 4504 can then indicate the presence of important body structures, tissues, biological processes, or chemical processes as determined by the location of the reagent or dye.

[0184] A filter 4502 can also be used in an implementation where no fluorescent reagent or dye is administered. The filter 4502 can be selected to allow wavelengths corresponding to a desired spectral response to pass and be read out by the image sensor 4504. The image sensor 4504 can be a monochromatic image sensor such that pixels of a captured image above or below a threshold can be characterized as corresponding to a particular spectral response or fluorescent emission. The spectral response or fluorescent emission determined by the pixels captured by the image sensor 4504 can indicate the presence of a particular body tissue or structure, a particular condition, a particular chemical process, etc.

[0185] In one embodiment, the light source 4508 transmits white light that contacts the surface 4512 and reflects back where it is filtered by the filter 4502 before hitting the image sensor 4504. In one embodiment, the light source 4508 transmits white light such that filtered light of only one or more desired wavelengths that pass through the filter 4502 emerges from the filter 4502 to be reflected back to the surface 4512 and read by the image sensor 4504. For example, in one embodiment, the filter 4502 allows only light having a wavelength of 795 nm to pass through the filter 4502 and contact the image sensor 4504. In a further embodiment, the filter 4502 allows only certain wavelengths of light to be reflected back to the image sensor 4504 of the endoscope 4506 or other imaging device. The filter 4502 can be located anywhere in the system 4500 or can be an attachment to the lens or image sensor 4504. The filter 4502 can be located in front of and / or behind the image sensor 4504. In one embodiment, the light emitted by the light source 4508 is filtered before it reaches the surface 4512, and the reflected light is filtered by an additional filter before it is usable by the image sensor 4504.

[0186] The light source 4508 can be an emitter that can be configured to emit white light or electromagnetic radiation of one or more specific wavelengths. The light source 4508 can include multiple lasers configured to emit or pulse light of specified wavelengths. In one embodiment, the light source 4508 emits white light and the filter 4502 is configured to filter out all unwanted light or other electromagnetic radiation except for the light of one or more desired wavelengths. The filter 4502 can be selected for a particular examination or purpose, for example, to highlight a type of body tissue or structure, or to highlight a particular condition or chemical process.

[0187] FIG. 46 is a schematic diagram of an imaging system 4600 having multiple cut filters. The system 4600 includes an endoscope 4606 or other suitable imaging device having a light source 4608 for use in low light environments. The endoscope 4606 includes an image sensor 4604 and two filters 4602a, 4602b. It should be understood that in alternative embodiments, the system 4600 can include any number of filters, and the number of filters and the type of filters can be selected for a particular purpose, such as to collect imaging information of a particular body tissue, body condition, chemical process, etc. The filters 4602a, 4602b are configured to filter unwanted wavelengths of light or other electromagnetic radiation. The filters 4602a, 4602b can be configured to filter unwanted wavelengths from white light or other electromagnetic radiation that can be emitted by the light source 4608. The filtered light can strike a surface 4612 (e.g., body tissue) and reflect back to the image sensor 4604.

[0188] Continuing with the disclosure with respect to FIG. 45, the filters 4602a, 4602b can be used in an implementation where a fluorescent reagent or dye is administered. The filters 4602a, 4602b can be configured to block the emission excitation wavelength of the reagent or dye and allow the image sensor 4604 to read only the relaxation wavelength of the reagent or dye. Additionally, the filters 4602a, 4602b can be used in an implementation where a fluorescent reagent or dye is not administered. In such an implementation, the filters 4602a, 4602b can be selected to allow wavelengths corresponding to a desired spectral response to pass and be read by the image sensor 4604.

[0189] The multiple filters 4602a, 4602b can each be configured to filter a different wavelength range of the electromagnetic spectrum. For example, one filter can be configured to filter wavelengths above a desired wavelength range and an additional filter can be configured to filter wavelengths below the desired wavelength range. The combination of two or more filters can result in only a particular wavelength or band of wavelengths being read out by the image sensor 4604.

[0190] In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 513 nm and 545 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 565 nm and 585 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 900 nm and 1000 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 425 nm and 475 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 520 nm and 545 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 625 nm and 645 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 760 nm and 795 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 795 nm and 815 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 370 nm and 420 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b can be customized to allow electromagnetic radiation between 600 nm and 670 nm to contact image sensor 4604. In one embodiment, the filters 4602 a , 4602 b are configured to allow only certain fluorescent relaxation emissions to pass through the filters 4602 a , 4602 b and contact the image sensor 4604 .

[0191] In one embodiment, the system 4600 includes multiple image sensors 4604, and in particular may include two image sensors for use in generating a three-dimensional image. The image sensor(s) 4604 may be color / wavelength blind and may be configured to read any wavelength of electromagnetic radiation reflected off the surface 4612. In one embodiment, the image sensors 4604 are each color-dependent or wavelength-dependent and configured to read a particular wavelength of electromagnetic radiation that reflects off the surface 4612 and back to the image sensor 4604. Alternatively, the image sensor 4604 may include a single image sensor having multiple different pixel sensors configured to read different wavelengths or colors of light, such as a Bayer filter color filter array. Alternatively, the image sensor 4604 may include one or more color blind image sensors that may be configured to read different wavelengths of electromagnetic radiation according to a pulsing schedule, such as those illustrated in Figures 5-7E and 15-16.

[0192] FIG. 47 is a schematic diagram illustrating a system 4700 for mapping a surface and / or tracking an object in a low light environment. In one embodiment, an endoscope 4702 in a low light environment pulses a grid array 4706 (which may also be referred to as a laser map pattern) of a surface 4704. The grid array 4706 may include vertical hashing 4708 and horizontal hashing 4710 in one embodiment illustrated in FIG. 47. It should be understood that the grid array 4706 may include any suitable array for mapping the surface 4704 including, for example, a raster grid of discrete locations, an occupancy grid map, a dot array, etc. Additionally, the endoscope 4702 may pulse multiple grid arrays 4706 and may pulse one or more individual grid arrays for each of multiple objects or structures in the low light environment, for example.

[0193] In one embodiment, the system 4700 can pulse the grid array 4706, which can be used to determine a three-dimensional surface and / or track the position of an object, such as a tool or another device, in a low-light environment. In one embodiment, the system 4700 can provide data to a third party system or computer algorithm to determine the dimensions and configuration of a surface by light detection and ranging (LIDAR) mapping. The system 4700 can pulse any suitable wavelength of light or electromagnetic radiation to the grid array 4706, including, for example, ultraviolet, visible, and / or infrared or near-infrared light. Surfaces 4704 and / or objects in the environment can be mapped and tracked with very high resolution and with very high accuracy and precision.

[0194] In one embodiment, the system 4700 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. The system 4700 can include a light engine having an illumination source that generates one or more pulses of electromagnetic radiation and a lumen that transmits the one or more pulses of electromagnetic radiation to a distal tip of an endoscope within a light-deficient environment, such as a body cavity. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation includes a laser map pattern that is emitted to a surface within the light-deficient environment, such as a surface of a body tissue and / or a surface of a tool or other device within the body cavity. The endoscope 4702 can include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations within the light-deficient environment.

[0195] In one embodiment, the system 4700 includes a processor for determining a distance from an object, such as the surface 4704, to the endoscope or tool. The processor can further determine an angle between the endoscope or tool and the object. The processor can further determine surface area information about the object, including, for example, a size of the surgical tool, a size of the structure, a size of the anatomical structure, position information, and other position data and metrics. The system 4700 can include one or more image sensors that provide image data that is output to a control system for determining a distance from the endoscope or tool to an object, such as the surface 4704. The image sensor can output information to the control system for determining an angle between the endoscope or tool and the object. Additionally, the image sensor can output information to the control system for determining surface area information about the object, a size of the surgical tool, a size of the structure, a size of the anatomical structure, position information, and other position data and metrics.

[0196] In one embodiment, the grid array 4706 is pulsed by an illumination source of the endoscope 4702 at a sufficient rate such that the grid array 4706 is not visible to a user. In various implementations, a user may be distracted by viewing the grid array 4706 during an endoscopic imaging procedure and / or an endoscopic surgical procedure. The grid array 4706 may be pulsed for a period of time that is short enough such that the grid array 4706 is not detectable by the human eye. In an alternative embodiment, the endoscope 4702 pulses the grid array 4706 at a sufficient repetition rate such that the grid array 4706 is visible to a user. In one such embodiment, the grid array 4706 may be superimposed on an image of the surface 4704 on a display. The grid array 4706 may be superimposed on a black and white or RGB image of the surface 4704 such that the grid array 4706 is visible to a user while using the system 4700. A user of the system 4700 can indicate whether the grid array 4706 should be overlaid on the image of the surface 4704 and / or whether the grid array 4706 should be visible to the user. The system 4700 can include a display that provides a real-time measurement of the distance from the endoscope 4702 to the surface 4704 or another object in a low light environment. The display can further provide real-time surface area information for the surface 4704 and / or any object, structure, or tool in the low light environment. The accuracy of the measurement can be accurate to within a millimeter.

[0197] The endoscope 4702 can pulse electromagnetic radiation according to a pulsing schedule, such as those illustrated in Figures 5-7E and 15-16, which can further include pulsing the grid array 4706 along with pulsing red, green, and blue light to generate an RGB image, and further generating a grid array 4706 that can be overlaid on the RGB image and / or used to map and track surfaces 4704 and objects in low light environments.

[0198] In one embodiment, the endoscope 4702 includes one or more color blind image sensors. In one embodiment, the endoscope 4702 includes two color blind image sensors for generating a three-dimensional image or map of the low light environment. The image sensor can generate an RGB image of the low light environment according to a pulsing schedule disclosed herein. In addition, the image sensor can determine data for mapping the low light environment and tracking one or more objects in the low light environment based on data determined when the grid array 4706 is pulsed. In addition, the image sensor can determine spectral or hyperspectral data along with the fluorescent imaging data according to a pulsing schedule that can be modified by a user to meet the specific needs of the imaging procedure. In one embodiment, the pulsing schedule includes red, green, and blue pulses along with pulsing of the grid array 4706 and / or pulsing to generate hyperspectral image data and / or fluorescent image data. In various implementations, the pulsing schedule can include any suitable combination of pulses of electromagnetic radiation according to the needs of the user. The repetition frequency of the different wavelengths of electromagnetic radiation can be determined based on, for example, the energy of a particular pulse, the needs of a user, whether particular data (e.g., hyperspectral data and / or fluorescent imaging data) needs to be updated continuously or less frequently, etc.

[0199] The pulsing schedule can be modified in any suitable manner, and a particular pulse of electromagnetic radiation can be repeated at any suitable frequency according to the needs of a user or computer-implemented program for a particular imaging procedure. For example, in an embodiment in which surface tracking data generated based on grid array 4706 is provided to a computer-implemented program for use in, for example, a robotic surgical procedure, grid array 4706 can be pulsed more frequently than if the surface tracking data were provided to a user visualizing a scene during an imaging procedure. In such an embodiment in which the surface tracking data is used in a robotic surgical procedure, the surface tracking data may need to be updated more frequently or may need to be extremely accurate to enable the computer-implemented program to perform the robotic surgical procedure with precision and accuracy.

[0200] In one embodiment, the system 4700 is configured to generate an occupancy grid map comprising an array of cells divided into a grid. The system 4700 is configured to store height values ​​for each of the respective grid cells to determine a surface mapping of a three-dimensional environment in a low-light environment.

[0201] 48 is a schematic flow chart diagram of a method 4800 for hyperspectral imaging in low light environments. The method 4800 may be performed by an imaging system such as the endoscopic imaging system illustrated in FIG.

[0202] The method 4800 includes, at 4802, emitting a plurality of narrowband pulses during a readout period of a monochrome image sensor. In 4802, the pulses can be emitted using a light source including a plurality of emitters that emit electromagnetic energy within a narrow frequency band. For example, the light source can include at least one emitter for a plurality of frequency bands that cover a desired spectrum. In 4804, the monochrome image sensor reads pixel data from the monochrome image sensor and generates a plurality of frames after the readout period. Each frame can include different spectral content. The frames can include a plurality of repeating frames that can be used to generate a digital video stream. Each frame can be based on energy emitted by one or more emitters of the light source. In an embodiment, the frames can be based on a combination of light emitted by the light source to generate a combination of frequencies to match a frequency response of a desired tissue or material. In 4806, the controller, CCU, or other system determines a tissue spectral response for one or more pixels based on the plurality of frames. For example, the frequency response of a particular pixel can be determined based on the value of the pixel in the plurality of frames using the pixel value and knowledge of the frequency of light emitted for each frame. At 4808, the system can generate a combined image based on the multiple frames, the combined image comprising an overlay showing the spectral response of one or more pixels. For example, the combined image can be a grayscale or color image in which pixels corresponding to a particular tissue or classification are shown in bright green.

[0203] 49 is a schematic flow chart diagram of a method 4900 for fluorescence imaging in a low light environment. The method 4900 can be performed by an imaging system such as the endoscopic imaging system illustrated in FIG.

[0204] The method 4900 includes, at 4902, emitting a plurality of narrowband pulses during a readout period of the monochrome image sensor. At 4902, the pulses can be emitted using a light source including a plurality of emitters that emit electromagnetic energy within a narrow frequency band. For example, the light source can include at least one emitter for a plurality of frequency bands that cover a desired spectrum. At 4904, the monochrome image sensor reads pixel data from the monochrome image sensor and generates a plurality of frames after the readout period. Each frame can include a different spectral content. The frames can include a plurality of repeated frames that can be used to generate a digital video stream. Each frame can be based on energy emitted by one or more emitters of the light source. In an embodiment, the frames can be based on a combination of light emitted by the light source to generate a combination of frequencies to match a frequency response of a desired tissue or material. At 4906, the controller, CCU, or other system determines a fluorescent relaxation emission of a reagent for one or more pixels based on the plurality of frames. For example, the frequency response of a particular pixel can be determined using the pixel value and knowledge of the frequency of light emitted for each frame based on the value of the pixel in the plurality of frames. At 4908, the system can generate a combined image based on multiple frames, the combined image comprising an overlay showing the fluorescent relaxation emission of one or more pixels. For example, the combined image can be a grayscale or color image in which pixels corresponding to a particular tissue or classification are shown in bright green. EXAMPLES

[0205] The following examples relate to further embodiments.

[0206] Example 1 is an endoscope system for use in low light environments. The system includes a tube, one or more image sensors, and a lens assembly with at least one optical element corresponding to the one or more image sensors. The system includes a display for a user to visualize the scene. The system includes a light engine including an illumination source generating one or more pulses of electromagnetic radiation. The light engine may further include a lumen for transmitting the one or more pulses of electromagnetic radiation to a distal tip of the endoscope, where at least a portion of the one or more pulses of electromagnetic radiation include electromagnetic radiation between 513 nm and 545 nm, at least a portion of the one or more pulses of electromagnetic radiation include electromagnetic radiation between 565 nm and 585 nm, and at least a portion of the one or more pulses of electromagnetic radiation include electromagnetic radiation between 900 nm and 1000 nm.

[0207] Example 2 is the system of example 1, wherein the duration of the one or more pulses of electromagnetic radiation is variable.

[0208] Example 3 is the system of any of examples 1-2, wherein the pulse duration comprises multiple pulses of electromagnetic radiation having a duration less than the entire pulse duration.

[0209] Example 4 is the system of any of examples 1-3, wherein the multiple pulses of electromagnetic radiation have a duration less than the overall pulse duration has a duration that provides a single frame exposure.

[0210] Example 5 is a system according to any of Examples 1-4, wherein multiple sub-pulses of electromagnetic radiation having sub-durations are emitted during the pulse duration, the sub-durations being shorter than the pulse duration.

[0211] Example 6 is the system of any of examples 1-5, wherein a pulse subduration within the pulse duration varies for each subpulse of electromagnetic radiation.

[0212] Example 7 is the system of any of examples 1-6, wherein the pulse subduration within the pulse duration is the same for each sub-pulse of electromagnetic radiation.

[0213] Example 8 is the system of any of examples 1-7, wherein the pulse subduration within the pulse duration includes two or more subpulses of electromagnetic radiation having equal durations.

[0214] Example 9 is the system of any of Examples 1-8, wherein one or more of the multiple sub-pulses can include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

[0215] Example 10 is a system according to any of Examples 1-9, wherein the subduration of the plurality of subpulses is variable both in the wavelength of the emitted electromagnetic radiation and in the number of emissions occurring per pulse duration.

[0216] Example 11 is the system of any of examples 1-10, wherein a subduration of the multiple sub-pulses is equal to a single pulse duration to provide a single frame exposure.

[0217] Example 12 is a system described in any of Examples 1 to 11, wherein the series of pulses of various wavelengths of electromagnetic radiation is adjustable by a user of the endoscope light engine.

[0218] Example 13 is the system of any of Examples 1 to 12, wherein the system includes a toggle button, and a user changes or adjusts the order of the pulses by manipulating the toggle button.

[0219] Example 14 is a system described in any of Examples 1 to 13, wherein the system includes a control system for appropriately lighting the scene, and the control system automatically adjusts or changes a sequence of pulses of electromagnetic radiation emitted from the light engine based on a threshold value, the threshold value a priori determining appropriate lighting of the scene.

[0220] Example 15 is a system described in any of Examples 1 to 14, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0221] Example 16 is the system of any of Examples 1-15, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0222] Example 17 is a system described in any of Examples 1 to 16, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to a user as an overlay image on a display.

[0223] Example 18 is the system of any of Examples 1 to 17, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0224] Example 19 is a system described in any of Examples 1-18, wherein the illumination source uses one or more laser emitters to generate one or more pulses of electromagnetic radiation.

[0225] Example 20 is a system described in any of Examples 1 to 19, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on the display.

[0226] Example 21 is the system of any of Examples 1-20, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0227] Example 22 is a system described in any of Examples 1 to 21, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as an overlay image on the display.

[0228] Example 23 is the system of any of Examples 1 to 22, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0229] Example 24 is a system described in any of Examples 1 to 23, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 425 nm and 475 nm.

[0230] Example 25 is a system described in any of Examples 1 to 24, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0231] Example 26 is the system of any of Examples 1 to 25, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0232] Example 27 is a system described in any of Examples 1 to 26, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as an overlay image on the display.

[0233] Example 28 is the system of any of Examples 1 to 27, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0234] Example 29 is a system described in any of Examples 1 to 28, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 520 nm and 545 nm.

[0235] Example 30 is a system described in any of Examples 1 to 29, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0236] Example 31 is the system of any of Examples 1-30, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0237] Example 32 is a system described in any of Examples 1 to 31, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to a user as an overlay image on a display.

[0238] Example 33 is the system of any of Examples 1 to 32, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0239] Example 34 is a system described in any of Examples 1 to 33, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 625 nm and 645 nm.

[0240] Example 35 is a system described in any of Examples 1 to 34, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0241] Example 36 is the system of any of Examples 1 to 35, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0242] Example 37 is a system described in any of Examples 1 to 36, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to a user as an overlay image on a display.

[0243] Example 38 is a system described in any of Examples 1 to 37, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0244] Example 39 is a system described in any of Examples 1 to 38, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 760 nm and 795 nm.

[0245] Example 40 is a system described in any of Examples 1 to 39, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0246] Example 41 is the system of any of Examples 1-40, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0247] Example 42 is a system described in any of Examples 1 to 41, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to a user as an overlay image on a display.

[0248] Example 43 is a system according to any of Examples 1 to 42, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0249] Example 44 is a system described in any of Examples 1 to 43, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 795 nm and 815 nm.

[0250] Example 45 is a system described in any of Examples 1 to 44, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0251] Example 46 is a system according to any of Examples 1 to 45, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0252] Example 47 is a system described in any of Examples 1 to 46, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to a user as an overlay image on a display.

[0253] Example 48 is a system described in any of Examples 1 to 47, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0254] Example 49 is a system described in any of Examples 1 to 48, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 370 nm and 420 nm.

[0255] Example 50 is a system described in any of Examples 1 to 49, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0256] Example 51 is the system of any one of Examples 1 to 50, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0257] Example 52 is a system described in any of Examples 1 to 51, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as an overlay image on a display.

[0258] Example 53 is a system described in any of Examples 1 to 52, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0259] Example 54 is a system described in any of Examples 1 to 53, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 600 nm and 670 nm.

[0260] Example 55 is a system described in any of Examples 1 to 54, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0261] Example 56 is a system described in any of Examples 1 to 55, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0262] Example 57 is a system described in any of Examples 1 to 56, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as an overlay image on a display.

[0263] Example 58 is a system described in any of Examples 1 to 57, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits.

[0264] Example 59 is a system described in any of Examples 1 to 58, wherein the light engine includes a polarizing filter.

[0265] Example 60 is a system described in any of Examples 1-59, wherein a polarizing filter is placed in the path of the electromagnetic radiation.

[0266] Example 61 is a system according to any of Examples 1 to 60, wherein the polarizing filter is disposed at a proximal end of the lumen.

[0267] Example 62 is a system according to any of Examples 1 to 61, wherein the polarizing filter is disposed at a distal end of the lumen.

[0268] Example 63 is the system of any of Examples 1-62, wherein the lens assembly includes an electromagnetic radiation filter.

[0269] Example 64 is a system described in any of Examples 1 to 63, wherein the lens assembly includes a polarizing filter.

[0270] Example 65 is a system described in any of Examples 1 to 64, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are supplied to a corresponding system that provides the location of a tissue structure of interest.

[0271] Example 66 is a system described in any of Examples 1 to 65, wherein the locations of important structures are received by the endoscopic system and overlaid on the display, and the important structures are coded in any color selected either by an algorithm or by a user.

[0272] It will be appreciated that the various features described herein represent important advantages and advances in the art, some of which are set forth in the claims that follow.

[0273] In the foregoing Detailed Description, various features of the disclosure are grouped together in a single embodiment for purposes of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. Rather, inventive aspects lie in less than all features of a single embodiment of the foregoing disclosure.

[0274] It is to be understood that the above-described arrangements are merely illustrative of the application of the principles of the present disclosure. Numerous modifications and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present disclosure, and the present disclosure is intended to cover all such modifications and arrangements.

[0275] Thus, while the present disclosure has been illustrated and described above with specificity and detail, it will be apparent to those skilled in the art that numerous modifications, including but not limited to modifications in size, material, shape, form, function and mode of operation, assembly and use, can be made without departing from the principles and concepts described herein.

[0276] Further, where appropriate, the functions described herein may be implemented by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) may be programmed to execute one or more of the systems and procedures described herein. Certain terms are used in the following description and throughout this disclosure to refer to particular system components. As will be appreciated by those skilled in the art, components may be referred to by different names. This specification does not intend to distinguish between components that differ in name but not function.

[0277] The foregoing description has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. Furthermore, it should be noted that any or all of the above-described alternative embodiments may be used in any combination desired to form further hybrid embodiments of the present disclosure.

[0278] Moreover, although particular embodiments of the present disclosure have been described and illustrated, the disclosure should not be limited to the specific forms and arrangements of parts so described and illustrated. The scope of the present disclosure is to be defined by the claims appended hereto, and any future claims filed in this and any different applications, and their equivalents.

[0279] [Embodiment] (1) An endoscope system for use in a low-light environment, comprising: 1. An imaging device, comprising: A tube and one or more image sensors; a lens assembly including at least one optical element corresponding to the one or more image sensors; and a display for a user to visualize the scene; an image signal processing controller; A light engine, an illumination source generating one or more pulses of electromagnetic radiation; 1. An endoscopic system comprising: a light engine comprising a lumen that delivers one or more pulses of electromagnetic radiation to a distal tip of an endoscope, wherein at least a portion of the one or more pulses of electromagnetic radiation comprise electromagnetic radiation between 513 nm and 545 nm, at least a portion of the one or more pulses of electromagnetic radiation comprise electromagnetic radiation between 565 nm and 585 nm, and at least a portion of the one or more pulses of electromagnetic radiation comprise electromagnetic radiation between 900 nm and 1000 nm. (2) An endoscopic system as described in embodiment 1, wherein the duration of one or more pulses of electromagnetic radiation is variable. (3) An endoscopic system as described in embodiment 2, wherein the pulse duration includes multiple pulses of electromagnetic radiation having a duration less than the entire pulse duration. (4) An endoscopic system as described in embodiment 3, wherein the multiple pulses of electromagnetic radiation having a duration less than the entire pulse duration have a duration that provides a single frame exposure. (5) An endoscopic system as described in embodiment 1, wherein during a pulse duration, multiple sub-pulses of electromagnetic radiation having sub-durations are emitted, the sub-durations being shorter than the pulse duration.

[0280] (6) An endoscopic system as described in embodiment 5, wherein the pulse sub-duration within the pulse duration varies for each sub-pulse of electromagnetic radiation. (7) An endoscopic system as described in embodiment 5, wherein the pulse sub-duration within the pulse duration is the same for each sub-pulse of electromagnetic radiation. (8) An endoscopic system as described in embodiment 5, wherein the pulse sub-duration within the pulse duration includes two or more sub-pulses of electromagnetic radiation having equal durations. (9) An endoscopic system as described in embodiment 5, wherein one or more of the multiple sub-pulses can include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse. (10) An endoscopic system as described in embodiment 5, wherein the subdurations of the multiple subpulses are variable in both the wavelength of the emitted electromagnetic radiation and the number of emissions occurring per pulse duration.

[0281] (11) An endoscopic system as described in embodiment 10, wherein the subdurations of the multiple sub-pulses are equal to a single pulse duration providing a single frame exposure. (12) An endoscopic system as described in embodiment 1, wherein the sequence of pulses of various wavelengths of electromagnetic radiation is adjustable by a user of the endoscope light engine. (13) An endoscopic system as described in embodiment 1, wherein the system includes a toggle button, and a user changes or adjusts the sequence of pulses by operating the toggle button. (14) An endoscopic system as described in embodiment 1, wherein the system includes a control system for appropriately illuminating the scene, the control system automatically adjusting or modifying the sequence of pulses of electromagnetic radiation emitted from the light engine based on a threshold, the threshold a priori determining appropriate illumination of the scene. (15) An endoscopic system as described in embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0282] (16) An endoscopic system as described in embodiment 15, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (17) An endoscopic system as described in embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (18) An endoscopic system as described in embodiment 17, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (19) An endoscopic system as described in embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation using one or more laser emitters. (20) An endoscopic system as described in embodiment 19, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0283] (21) An endoscopic system as described in embodiment 20, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (22) An endoscopic system as described in embodiment 19, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (23) An endoscopic system as described in embodiment 22, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (24) The endoscopic system of embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at a wavelength between 425 nm and 475 nm. (25) An endoscopic system as described in embodiment 24, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0284] (26) An endoscopic system as described in embodiment 25, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (27) An endoscopic system as described in embodiment 24, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (28) An endoscopic system as described in embodiment 27, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (29) The endoscopic system of embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at a wavelength between 520 nm and 545 nm. (30) An endoscopic system as described in embodiment 29, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0285] (31) An endoscopic system as described in embodiment 30, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (32) An endoscopic system as described in embodiment 29, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (33) An endoscopic system as described in embodiment 32, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (34) The endoscopic system of embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at a wavelength between 625 nm and 645 nm. (35) An endoscopic system as described in embodiment 34, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0286] (36) An endoscopic system as described in embodiment 35, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (37) An endoscopic system as described in embodiment 34, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (38) An endoscopic system as described in embodiment 37, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (39) The endoscopic system of embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at a wavelength between 760 nm and 795 nm. (40) An endoscopic system as described in embodiment 39, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0287] (41) An endoscopic system as described in embodiment 40, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (42) An endoscopic system as described in embodiment 39, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (43) An endoscopic system as described in embodiment 42, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (44) The endoscopic system of embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 795 nm and 815 nm. (45) An endoscopic system as described in embodiment 44, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0288] (46) An endoscopic system as described in embodiment 45, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (47) An endoscopic system as described in embodiment 44, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (48) An endoscopic system as described in embodiment 47, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (49) The endoscopic system of embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths between 370 nm and 420 nm. (50) An endoscopic system as described in embodiment 49, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0289] (51) An endoscopic system as described in embodiment 50, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (52) An endoscopic system as described in embodiment 49, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (53) An endoscopic system as described in embodiment 52, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (54) The endoscopic system of embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at a wavelength between 600 nm and 670 nm. (55) An endoscopic system as described in embodiment 54, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as a single image on the display.

[0290] (56) An endoscopic system as described in embodiment 55, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (57) An endoscopic system as described in embodiment 54, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to a user as an overlay image on the display. (58) An endoscopic system as described in embodiment 57, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits or 16 bits or n bits. (59) An endoscopic system as described in embodiment 1, wherein the light engine is equipped with a polarizing filter. (60) An endoscopic system as described in embodiment 59, wherein the polarizing filter is positioned in the path of the electromagnetic radiation.

[0291] (61) An endoscopic system as described in embodiment 60, wherein the polarizing filter is disposed at a proximal end of the lumen. (62) An endoscopic system as described in embodiment 60, wherein the polarizing filter is disposed at a distal end of the lumen. (63) An endoscopic system as described in embodiment 1, wherein the lens assembly includes an electromagnetic radiation filter. (64) An endoscopic system as described in embodiment 1, wherein the lens assembly includes a polarizing filter. (65) An endoscopic system as described in embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame produced by the one or more image sensors, and the one or more exposure frames are supplied to a corresponding system that provides the location of a tissue structure of interest.

[0292] (66) An endoscopic system as described in embodiment 65, wherein the positions of the important structures are received by the endoscopic system and overlaid on a display, and the important structures are coded in any color selected either by an algorithm or a user.

Claims

1. 1. An endoscope system for use in low light environments, comprising:

1. An imaging device, comprising: an endoscope tube having a proximal end and a distal end; one or more image sensors disposed at the distal end; a lens assembly including at least one optical element corresponding to the one or more image sensors; and A display for the user to visualize the scene and an image signal processing controller; A light engine, an illumination source including an emitter that generates a plurality of pulses of electromagnetic radiation; a collection area configured to provide a plurality of electromagnetic radiation pulses generated by the emitter to a waveguide; the waveguide is configured to transmit the plurality of electromagnetic radiation pulses to the distal end; The emitter comprises a visible light emitting element and a plurality of hyperspectral emitters; The plurality of hyperspectral emitters include: a first hyperspectral emitter configured to pulse electromagnetic radiation covering a wavelength band from 500 nm to 540 nm; a second hyperspectral emitter configured to pulse electromagnetic radiation covering a wavelength band from 540 nm to 640 nm; and a third hyperspectral emitter configured to pulse electromagnetic radiation covering a wavelength band from 900 nm to 1000 nm; the emitter is configured to generate the plurality of pulses of electromagnetic radiation during a blanking period of an image sensor; the one or more image sensors capture reflections of electromagnetic radiation from the emitters to generate hyperspectral image data; The hyperspectral image data is read out to generate one or more hyperspectral exposure frames.

2. 2. The endoscopic system of claim 1, wherein the image signal processing controller is configured to instruct one or more of the emitters to modulate the intensity and / or duration of one or more of the plurality of electromagnetic radiation pulses.

3. 2. The endoscopic system of claim 1, wherein the plurality of electromagnetic radiation pulses includes sub-pulses having sub-durations shorter than a duration of the plurality of electromagnetic radiation pulses, the sub-pulses having a duration that provides a single frame exposure.

4. The endoscopic system of claim 3 , wherein the sub-duration is variable for each sub-pulse of electromagnetic radiation.

5. The endoscopic system of claim 3 , wherein the sub-duration comprises two or more sub-pulses of electromagnetic radiation of equal duration.

6. 4. The endoscopic system of claim 3, wherein the sub-pulse is a plurality of sub-pulses, and one or more of the plurality of sub-pulses comprises electromagnetic radiation emitted at two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

7. 6. The endoscopic system of claim 5, wherein the sub-durations of the plurality of sub-pulses are variable in both the emitted electromagnetic radiation wavelength and the number of emissions occurring per pulse duration.

8. 10. The endoscopic system of claim 1, wherein the sequence of pulses of different wavelengths of electromagnetic radiation is adjustable by a user of the light engine.

9. 2. The endoscopic system of claim 1, further comprising a control system for appropriately illuminating the scene, the control system automatically adjusting or modifying a sequence of multiple electromagnetic radiation pulses emitted from the light engine based on a threshold, the threshold a priori determining appropriate illumination of the scene.

10. 2. The endoscopic system of claim 1, wherein each pulse of the plurality of electromagnetic radiation pulses results in one or more exposure frames produced by one or more image sensors, the one or more exposure frames consisting of one or more hyperspectral exposure frames, and the one or more exposure frames are displayed to a user as a single image on a display or as overlay images on a display.

11. The endoscope system of claim 10, wherein the single image or the overlay image is assigned a visible color for use on a display, the visible color being 8 bits, 16 bits or n bits.

12. The endoscopic system of claim 1 , wherein the illumination source uses one or more laser emitters to generate the plurality of pulses of electromagnetic radiation.

13. The plurality of hyperspectral emitters of the illumination source further include: a fourth hyperspectral emitter generating one or more pulses of electromagnetic radiation covering a wavelength band between 425 nm and 475 nm; a fifth hyperspectral emitter generating one or more pulses of electromagnetic radiation covering a wavelength band between 520 nm and 545 nm; a sixth hyperspectral emitter generating one or more pulses of electromagnetic radiation covering a wavelength band between 625 nm and 645 nm; a seventh hyperspectral emitter generating one or more pulses of electromagnetic radiation covering a wavelength band between 760 nm and 795 nm; an eighth hyperspectral emitter generating one or more pulses of electromagnetic radiation covering a wavelength band from 795 nm to 815 nm; a ninth hyperspectral emitter generating one or more pulses of electromagnetic radiation covering a wavelength band between 370 nm and 420 nm; or a tenth hyperspectral emitter generating one or more pulses of electromagnetic radiation covering a wavelength band between 600 nm and 670 nm; The endoscopic system of claim 1 , further comprising one or more of the following:

14. 2. The endoscopic system of claim 1, wherein the light engine comprises a polarizing filter, the polarizing filter being disposed in a path of the plurality of electromagnetic radiation pulses, at a proximal end of the waveguide, or at a distal end of the waveguide.

15. The endoscopic system of claim 1 , wherein the lens assembly comprises an electromagnetic radiation filter and / or a polarizing filter.

16. 2. The endoscope system according to claim 1, wherein the blanking period comprises one or more different blanking periods, and the one or more different blanking periods are variable.

17. 2. The endoscopic system of claim 1, wherein the plurality of hyperspectral emitters are configured to elicit one or more spectral responses from tissue, and the hyperspectral image data includes the one or more spectral responses.

18. The endoscopic system of claim 1 , wherein the one or more hyperspectral exposure frames are provided to a corresponding system that provides the location of one or more tissue structures.

19. The endoscopic system of claim 18, wherein the locations of the one or more tissue structures are received by the endoscopic system and overlaid on a display, and the one or more tissue structures are coded in any color selected either by an algorithm or by a user.

20. The endoscopic system of claim 1 , wherein the plurality of hyperspectral emitters emit electromagnetic radiation that penetrates 5 nm, 8 nm, or 10 nm into tissue.

Images