Hyperspectral imaging in light-deficient environments

Abstract

To provide a system for generating an image in a light deficient environment, a related structure and a method.SOLUTION: An endoscopic imaging system in a light deficient environment includes an imaging device having a tube, one or more image sensors, and a lens assembly including at least one optical element that corresponds to the one or more image sensors. The endoscopic system includes a display for a user to visualize a scene and an image signal processing controller. The endoscopic system includes a light engine having an illumination source generating one or more pulses of electromagnetic radiation and a lumen transmitting one or more pulses of electromagnetic radiation to a distal tip of an endoscope.SELECTED DRAWING: Figure 1

Description

Background Art

[0001] With the progress of technology, the progress of medical imaging capabilities has been brought about. Using an endoscope, it is possible to view the inside of the body and examine the inside of body organs or body cavities. The endoscope can be used to investigate a patient's symptoms, confirm a diagnosis, or provide medical treatment. A medical endoscope can be used, for example, through a small incision to visualize various body tissues and body parts such as the digestive tract, airway, urinary tract, abdominal cavity, etc. The endoscope can also be used for surgical procedures such as plastic surgery techniques, procedures performed on joints or bones, procedures performed on the nervous system, procedures performed inside the abdominal cavity, etc.

[0002] Endoscopes are also used in non-medical fields for visualizing and inspecting spaces that may be inaccessible or difficult to view. For example, an endoscope can be used by a planner or architect to visualize a proposed building or a scale model of a city. An endoscope can be used to visualize the internal space of a complex system such as a computer. An endoscope can even be used by law enforcement agencies or military personnel to monitor a narrow space or to inspect a blasting device.

[0003] Among the various applications of endoscopes, visualizing space in color can be beneficial. A digital color image can contain at least three layers or "color channels" for each pixel in the image. Each color channel measures the intensity and chrominance of light in a spectral band. Generally, a digital color image contains color channels for the red, green, and blue spectral bands of light (this is sometimes referred to as an RGB image). Each of the red, green, and blue color channels contains luminance information for the red, green, or blue spectral band of light. By combining the luminance information from separate red, green, and blue layers, a digital color image can be created. Because a color image consists of separate layers, a digital camera image sensor generally includes a color filter array that allows red, green, and blue visible light wavelengths to collide with selected pixel sensors. Each individual pixel sensor element is sensitive to red, green, or blue wavelengths and returns only image data for that wavelength. By combining the image data from the entire array of pixel sensors, an RGB image is generated.

[0004] In endoscopic imaging for medical diagnosis or treatment, visualizing body cavities with color images can be beneficial, and may even be necessary. For example, when visualizing the abdominal cavity of the body using an endoscope, color images can provide useful information to aid in the identification of different organs or tissues within the abdomen, or in the identification of specific conditions or diseases within the space. As discussed above, a digital camera capable of capturing color images may have at least three different types of pixel sensors to individually capture the red, green, and blue layers of the color image. At least three different types of pixel sensors may consume a relatively large physical space (compared to a pixel array that cannot perceive color), and therefore the entire pixel array cannot be fitted to the small distal end of an endoscope inserted into the body. Since a color digital camera may include at least three different types of pixel sensors, the entire pixel array (i.e., the image sensor) is generally located within the handpiece unit of the endoscope, which is held by the endoscope operator and not placed inside the body cavity. In such an endoscope, light is transmitted along the length of the endoscope, from the handpiece unit to the distal end of the endoscope placed inside the body cavity. This endoscopic configuration has significant limitations. Endoscopes with this configuration are delicate and prone to displacement or damage when bumped or impacted during normal use. This can significantly degrade the quality of images produced by the endoscope and may necessitate frequent repair or replacement of the endoscope.

[0005] In some cases, particularly in medical imaging or medical procedures, visibility may be more beneficial than color imaging. A color image reflects what the human eye detects when viewed in a given environment. However, the human eye is limited to seeing only visible light and cannot detect other wavelengths of the electromagnetic spectrum. Electromagnetic spectrum wavelengths beyond "visible light" can provide additional information about the environment. One way to detect this additional environmental information beyond what the human eye can detect is hyperspectral imaging. For medical imaging purposes, hyperspectral imaging can provide a unique view of body cavities that offers additional information beyond what the human eye or a digital color camera can detect.

[0006] Hyperspectral imaging can be used to identify different materials or objects, and different processes. The human eye sees visible light colors in three main wavelength bands, perceived as a combination of red, green, and blue. A typical human eye responds to wavelengths of electromagnetic radiation from approximately 350 nm to 750 nm, a band sometimes referred to as the "visible spectrum." Hyperspectral imaging differs in that it divides the spectrum of light wavelengths into many more bands based on the reflection of tissue, object, or material. This allows hyperspectral imaging to provide information beyond what the human eye can see. Hyperspectral imaging can provide useful information in a variety of applications where it is difficult or impossible to identify specific materials, objects, or processes with the human eye or a computer. Hyperspectral imaging can be particularly effective in certain cases because some objects leave a unique "fingerprint" in the electromagnetic spectrum, sometimes referred to as a spectral signature. These spectral signatures enable the identification of specific materials, objects, or processes that cannot be seen by the human eye or a computer.

[0007] Hyperspectral imaging was originally developed for mining and geological applications. Unlike conventional camera images, which provide limited information to the human eye, hyperspectral imaging can identify specific minerals based on their spectral signatures. Hyperspectral imaging can even be useful when taking aerial images and can provide information about oil or gas leaks from pipelines or natural wells, and their impact on nearby vegetation. This information is collected based on the spectral signatures of specific materials, objects, or processes that can be identified by hyperspectral imaging.

[0008] However, hyperspectral imaging is extremely complex and may require high-speed computer processing power, sensitive detectors, and large data storage capacity. In addition, when hyperspectral images are superimposed on a black and white or color image to provide the situation to the practitioner, the camera (or multiple cameras) capable of generating the superimposed image may have many different types of pixel sensors sensitive to specific electromagnetic radiation ranges. This may include three separate types of pixel sensors for generating RGB color images in a conventional manner, along with additional pixel sensors for generating hyperspectral image data at different wavelengths of the electromagnetic spectrum. This may consume a relatively large physical space and require a large pixel array to ensure sufficient image resolution. In the case of endoscopic imaging, the physical space required and the pixel array required would be too large to place multiple wavelength-sensitive pixel sensors at the distal end of the endoscope in the body cavity, so one or more cameras may be placed within the endoscopic hand unit or robotic unit. This results in the same disadvantages as described above and can make the endoscope very delicate, so that image quality deteriorates significantly if it is bumped or impacted during use. [Overview of the project] [Means for solving the problem]

[0009] This disclosure relates, in general, to electromagnetic sensing and sensors that can be applied to endoscopic imaging. This disclosure also relates to low-energy electromagnetic input conditions and low-energy electromagnetic throughput conditions. More specifically, this disclosure relates to systems and related structures, methods and features for generating images in light-deficient environments, though not necessarily as a whole, the methods may include controlling a light source over time, intensity, or both; pulsing a component-controlled light source during the blanking period of the image sensor; maximizing the blanking period to enable optimal light; and maintaining color balance.

[0010] The features and advantages of this disclosure will be described below and will be evident to some extent from this description, and / or acquired through the practice of this disclosure without appropriate experimentation. The features and advantages of this disclosure can be realized and obtained by the apparatus and combinations specifically indicated in the appended claims. [Brief explanation of the drawing]

[0011] Non-restrictive and non-exclusive implementations of this disclosure are described with reference to the following figures, and similar reference numbers refer to similar parts throughout the various figures unless otherwise specified. The merits of this disclosure will be better understood with reference to the following description and accompanying drawings. [Figure 1] This is a schematic diagram of an embodiment of a system consisting of a pair of sensors and electromagnetic emitters in operation, used for generating images in a light-deficient environment, according to one embodiment. [Figure 2] This is a schematic diagram of complementary system hardware. [Figure 2A] This is a diagram illustrating the operation cycle of a sensor used to construct a single image frame according to an embodiment of the present disclosure. [Figure 2B] This is a diagram illustrating the operation cycle of a sensor used to construct a single image frame according to an embodiment of the present disclosure. [Figure 2C]This is a diagram illustrating the operation cycle of a sensor used to construct a single image frame according to an embodiment of the present disclosure. [Figure 2D] This is a diagram illustrating the operation cycle of a sensor used to construct a single image frame according to an embodiment of the present disclosure. [Figure 3] This is a graphical representation of the operation of one embodiment of an electromagnetic emitter, according to one embodiment. [Figure 4] This is a graphical representation of varying the duration and amplitude of emitted electromagnetic pulses to provide exposure control according to one embodiment. [Figure 5] Figures 2A to 4 show a graphical representation of an embodiment of the present disclosure, combining the operating cycles of the sensor, electromagnetic emitter, and emitted electromagnetic pulse, illustrating an imaging system in operation according to one embodiment. [Figure 6] This is a schematic diagram of two different processes over a period of time t(0) to t(1) for recording video frames of whole-spectrum light and divided-spectrum light according to one embodiment. [Figure 7A] This is a schematic diagram of a process over a time interval for recording video frames of both whole-spectrum and segmented-spectrum light in accordance with the principles and teachings of this disclosure. [Figure 7B] This is a schematic diagram of a process over a time interval for recording video frames of both whole-spectrum and segmented-spectrum light in accordance with the principles and teachings of this disclosure. [Figure 7C] This is a schematic diagram of a process over a time interval for recording video frames of both whole-spectrum and segmented-spectrum light in accordance with the principles and teachings of this disclosure. [Figure 7D] This is a schematic diagram of a process over a time interval for recording video frames of both whole-spectrum and segmented-spectrum light in accordance with the principles and teachings of this disclosure. [Figure 7E] This is a schematic diagram of a process over a time interval for recording video frames of both whole-spectrum and segmented-spectrum light in accordance with the principles and teachings of this disclosure. [Figure 8] A diagram illustrating the adjustment of both an electromagnetic emitter and a sensor, and such adjustment can be performed simultaneously in some embodiments in accordance with the principles and teachings of the present disclosure. [Figure 9] A diagram illustrating the adjustment of both an electromagnetic emitter and a sensor, and such adjustment can be performed simultaneously in some embodiments in accordance with the principles and teachings of the present disclosure. [Figure 10] A diagram illustrating the adjustment of both an electromagnetic emitter and a sensor, and such adjustment can be performed simultaneously in some embodiments in accordance with the principles and teachings of the present disclosure. [Figure 11] A diagram illustrating the adjustment of both an electromagnetic emitter and a sensor, and such adjustment can be performed simultaneously in some embodiments in accordance with the principles and teachings of the present disclosure. [Figure 12] A diagram illustrating the adjustment of both an electromagnetic emitter and a sensor, and such adjustment can be performed simultaneously in some embodiments in accordance with the principles and teachings of the present disclosure. [Figure 13] A schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 14] A schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. <00,00085> [Figure 15] A schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 16] A schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 17] A schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 18] A schematic diagram of a sensor correction method and hardware for use with a split optical system according to an embodiment of the present disclosure. [Figure 19]Schematic diagrams of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 20] Schematic diagrams of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 21] Schematic diagrams of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 22] Schematic diagrams of a method and hardware for increasing the dynamic range in a closed environment or a limited optical environment, according to an embodiment of the present disclosure. [Figure 23] Schematic diagrams of a method and hardware for increasing the dynamic range in a closed environment or a limited optical environment, according to an embodiment of the present disclosure. [Figure 24] A diagram illustrating the effect on the signal-to-noise ratio of color correction of a typical Bayer-based sensor compared to one without color correction. [Figure 25] A diagram illustrating the chromaticity of three monochromatic lasers compared to the sRGB color gamut. [Figure 26] Schematic diagrams of a method and hardware for increasing the dynamic range in a closed environment or a limited optical environment, according to an embodiment of the present disclosure. [Figure 27A] Schematic diagrams of a method and hardware for increasing the dynamic range in a closed environment or a limited optical environment, according to an embodiment of the present disclosure. [Figure 27B] Schematic diagrams of a method and hardware for increasing the dynamic range in a closed environment or a limited optical environment, according to an embodiment of the present disclosure. [Figure 28A] A diagram illustrating the use of white light emission pulsed and / or synchronized by a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 28B] A diagram illustrating the use of white light emission pulsed and / or synchronized by a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 28C]This figure illustrates the use of white light emission, which is pulsed and / or synchronized by a corresponding color sensor, according to embodiments of the present disclosure. [Figure 29A] This figure illustrates an implementation configuration having multiple pixel arrays for generating a three-dimensional image according to an embodiment of the present disclosure. [Figure 29B] This figure illustrates an implementation configuration having multiple pixel arrays for generating a three-dimensional image according to an embodiment of the present disclosure. [Figure 30A] These are perspective and side views of an imaging sensor mounting configuration constructed on multiple substrates, where multiple pixel rows forming a pixel array are located on a first substrate, and multiple circuit rows are located on a second substrate, showing the electrical connections and communication between one pixel row and its associated or corresponding circuit row. [Figure 30B] These are perspective and side views of an imaging sensor mounting configuration constructed on multiple substrates, where multiple pixel rows forming a pixel array are located on a first substrate, and multiple circuit rows are located on a second substrate, showing the electrical connections and communication between one pixel row and its associated or corresponding circuit row. [Figure 31A] These are perspective and side views, respectively, of an implementation configuration of an imaging sensor having multiple pixel arrays for generating a three-dimensional image, where the multiple pixel arrays and image sensor are constructed on multiple substrates. [Figure 31B] These are perspective and side views, respectively, of an implementation configuration of an imaging sensor having multiple pixel arrays for generating a three-dimensional image, where the multiple pixel arrays and image sensor are constructed on multiple substrates. [Figure 32] This figure illustrates embodiments of emitters with various mechanical filter and shutter configurations according to embodiments of the present disclosure. [Figure 33] This figure illustrates embodiments of emitters with various mechanical filter and shutter configurations according to embodiments of the present disclosure. [Figure 34] This figure illustrates embodiments of emitters with various mechanical filter and shutter configurations according to embodiments of the present disclosure. [Figure 35]This figure illustrates embodiments of emitters with various mechanical filter and shutter configurations according to embodiments of the present disclosure. [Figure 36] This figure illustrates embodiments of emitters with various mechanical filter and shutter configurations according to embodiments of the present disclosure. [Figure 37] This is a schematic diagram illustrating a system for providing lighting in light-deficient environments, according to one embodiment. [Figure 38] This is a schematic block diagram illustrating a light source having multiple emitters according to one embodiment. [Figure 39] This is a schematic block diagram illustrating a light source having multiple emitters according to another embodiment. [Figure 40] This is a schematic block diagram illustrating a light source having multiple emitters according to yet another embodiment. [Figure 41] This is a schematic diagram illustrating a single optical fiber that outputs via a diffuser at the output for illuminating a scene, according to one embodiment. [Figure 42] This is a block diagram illustrating the generation of a filtered image using a filter, according to one embodiment. [Figure 43] This figure illustrates a portion of an electromagnetic spectrum divided into multiple different subspectrals, which can be emitted by the emitter of a light source according to one embodiment. [Figure 44] This is a schematic diagram illustrating the emission and readout timing diagram for generating a multispectral or hyperspectral image according to one embodiment. [Figure 45] This is a block diagram illustrating the generation of a filtered image using a filter, according to one embodiment. [Figure 46] This is a block diagram illustrating the generation of an image filtered using multiple filters according to one embodiment. [Figure 47] This is a schematic diagram illustrating a grid array for tracking objects and / or surfaces according to one embodiment. [Figure 48]This is a schematic flowchart illustrating an example of an emission and readout method for generating a multispectral or hyperspectral image according to one embodiment. [Figure 49] This is a schematic flowchart illustrating an emission and readout method for generating a fluorescence image according to one embodiment. [Modes for carrying out the invention]

[0012] This disclosure extends primarily to methods, systems, and computer-based products for digital imaging that may be suitable for medical applications such as medical endoscopic imaging. Such methods, systems, and computer-based products disclosed herein may provide imaging or diagnostic capabilities for use in medical robotics applications, such as the use of robotics to perform imaging procedures, surgical techniques, and the like. The following description of this disclosure refers to accompanying drawings that form part of this specification and illustrate specific embodiments capable of practicing embodiments of the present invention. It will be understood that other implementations may be available and that structural modifications may be made without departing from the scope of this disclosure.

[0013] Endoscopes have a wide variety of applications and can provide considerable benefits in the medical field. Endoscopic examinations are used in the medical field to view the inside of the body and, in some cases, can provide imaging that cannot be seen by other means or that would require invasive surgical procedures. Endoscopes can be used for medical diagnosis, investigation, or research, and can even be used to perform medical procedures in a minimally invasive manner. Medical endoscopes can provide considerable benefits to patients and healthcare professionals by eliminating the need for painful and invasive corrections or experimental surgeries.

[0014] As disclosed herein, an endoscopic system for use in light-deficient environments such as body cavities may include an imaging device and a light engine. The light engine may include an illumination source for generating pulses of electromagnetic radiation, and may further include a lumen for transmitting pulses of electromagnetic radiation at the distal tip of the endoscope. The lumen can transmit pulses of electromagnetic radiation at specific wavelengths or wavelength bands of the electromagnetic spectrum. The lumen can transmit such pulses in a time-coordinated sequence, and imaging data can be captured by a sensor during each pulse. Using imaging data associated with pulses of different wavelengths, red-green-blue (RGB) images and / or multispectral or hyperspectral images can be generated. In one embodiment, the multispectral or hyperspectral image can be superimposed on a black-and-white or RGB image.

[0015] As disclosed herein, systems, methods, and apparatus for endoscopic imaging systems can provide specialized image data for light-deficient environments. Using this specialized image data, multispectral or hyperspectral images can be generated, and / or specific materials, tissues, components, or processes can be identified within a light-deficient environment. In certain embodiments, the multispectral or hyperspectral images can be provided to a practitioner or a computer-implemented program to enable the identification of specific structures or tissues within the body. Such multispectral or hyperspectral image data can be overlaid on black and white or RGB images to provide additional information and context.

[0016] Furthermore, such endoscopic imaging systems, methods, and apparatus can be used in conjunction with specific reagents or dyes. In medical imaging implementations, specific reagents or dyes can be administered to a patient, and such reagents or dyes can fluoresce or react to electromagnetic radiation of specific wavelengths. The endoscopic imaging systems disclosed herein can transmit electromagnetic radiation of a specified wavelength to cause the reagents or dyes to fluoresce. The fluorescence of the reagents or dyes can be captured by an image sensor to generate an image, which can assist in the identification of tissues or structures and / or in diagnosis or investigation. In one implementation, the patient can be administered multiple reagents or dyes, each configured to fluoresce at different wavelengths and / or to provide indications for different structures, tissues, chemical reactions, biological processes, etc. In such implementations, the endoscopic systems disclosed herein can emit each of the applicable wavelengths to cause the applicable reagents or dyes to fluoresce. This eliminates the need for performing individual imaging procedures for each of the multiple reagents or dyes.

[0017] Medical endoscopes can provide a continuous digital image stream of the internal space of the body into which the distal end of the endoscope is inserted. In various implementations, it may be beneficial, or even necessary, for the digital image stream to provide full-color imaging so that healthcare professionals can better distinguish between tissues and structures within the body. In further implementations, it may be beneficial to provide hyperspectral imaging data to distinguish structures, tissues, processes, and conditions with increased precision. In addition, hyperspectral imaging can enable healthcare professionals or computer programs to receive information about human body conditions that are not visible to the human eye or cannot be identified in RGB color images.

[0018] This specification discloses systems, methods, and apparatus for generating color image data and / or hyperspectral image data by an endoscope. The system of this disclosure includes an imaging device having a tube, one or more image sensors, and a lens assembly. The lens assembly may include at least one optical element corresponding to at least one of the one or more image sensors. The system further includes a display for visualizing a scene and an image signal processing controller. The system may further include a light engine. The light engine includes an illumination source configured to generate one or more pulses of electromagnetic radiation, and a lumen for transmitting one or more pulses of electromagnetic radiation to the distal tip of an endoscope. In one embodiment, at least a portion of one or more pulses of electromagnetic radiation includes electromagnetic radiation in the range of 513 nm to 545 nm. In one embodiment, at least a portion of one or more pulses of electromagnetic radiation includes electromagnetic radiation in the range of 565 nm to 585 nm. In one embodiment, at least a portion of one or more pulses of electromagnetic radiation includes electromagnetic radiation in the 900 nm to 1000 nm range. In the implementation of this disclosure, pulsation by the light engine can have a very short duration and an on / off time shorter than 1 microsecond (μs).

[0019] In embodiments of this disclosure, the endoscope system illuminates a light source for spectral imaging or hyperspectral imaging, pulsing electromagnetic radiation. Spectral imaging uses multiple bands across the electromagnetic spectrum. This differs from conventional cameras, which only capture light across three wavelengths based on the visible spectrum, including red, green, and blue wavelengths for generating RGB images, which are discernible to the human eye. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, the visible spectrum, the ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can superimpose an image generated based on the invisible band (e.g., infrared) on top of an image based on the visible band (e.g., a standard RGB image), providing additional information that is easily discernible by humans or computer algorithms.

[0020] Hyperspectral imaging is a subcategory of spectral imaging. Hyperspectral imaging includes spectroscopy and digital photography. In one embodiment of hyperspectral imaging, a complete spectrum or some spectral information is collected at every pixel in the image plane. A hyperspectral camera can capture any number of suitable wavelength bands per pixel, which can be interpreted as a complete spectrum, using specialized hardware. The goals of hyperspectral imaging can vary for different applications. In one application, the goal of hyperspectral imaging is to obtain the entire electromagnetic range of each pixel in the image scene. This can make it possible to find certain objects that might otherwise be indistinguishable under the visible light wavelength band. This can make it possible to accurately identify certain materials or tissues when they might be indistinguishable under the visible light wavelength band. Furthermore, this can make it possible to detect certain processes by capturing an image across the electromagnetic spectrum of all wavelengths.

[0021] Hyperspectral imaging can offer specific advantages over conventional imaging in medical applications. The information obtained through hyperspectral imaging can enable healthcare professionals and / or computer-implemented programs to accurately identify specific tissues or conditions that may be impossible or inaccurate when using conventional imaging methods such as RGB imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, enabling healthcare professionals to, for example, visualize tissues located behind specific tissues or fluids, identify atypical cancer cells in contrast to typical healthy cells, identify specific tissues or conditions, and identify important structures. Hyperspectral imaging can provide specialized diagnostic information regarding tissue physiology, morphology, and composition that cannot be generated by conventional imaging methods.

[0022] Endoscopic hyperspectral imaging can offer advantages over conventional imaging in various applications and implementations of this disclosure. In medical implementations, endoscopic hyperspectral imaging enables practitioners or computer-implemented programs to identify, for example, nerve tissue, muscle tissue, various blood vessels, blood flow direction, etc. Hyperspectral imaging can enable precise differentiation of atypical cancerous tissue from typical healthy tissue, thus enabling practitioners or computer-implemented programs to identify the boundaries of cancerous tumors during operation or investigational imaging. In addition, hyperspectral imaging in light-deficient environments disclosed herein can, in combination with the use of reagents or dyes, enable further differentiation of specific tissues or substances. In one such embodiment, a reagent or dye can fluoresce in a specific wavelength band within the electromagnetic spectrum, thus providing specific information for the purpose of that reagent or dye. The systems, methods, and apparatus disclosed herein can enable the pulsement of any number of wavelength bands so that one or more reagents or dyes can fluoresce at different times. In certain embodiments, this can enable the identification or investigation of several medical conditions during a single imaging procedure.

[0023] Medical endoscopes can pulse electromagnetic radiation in wavelength bands outside the visible light spectrum to enable the generation of hyperspectral images. Endoscopic hyperspectral imaging is a non-contact and non-invasive method for medical imaging that does not require patients to undergo harmful radiation exposure, which is common in other imaging methods.

[0024] Conventional endoscopes used in robotic endoscopic procedures such as arthroscopy and laparoscopy are designed so that the image sensor is located within a handpiece unit that is typically held by the endoscope operator and not inserted into the cavity. In such a configuration, the endoscope unit transmits incident light along the length of the endoscope tube to the sensor via a complex set of precisely coupled optical components, minimizing loss and distortion. Optical components are expensive, and their manufacturing process is labor-intensive, so the optical components account for a large portion of the cost of the endoscope unit. Furthermore, this type of endoscope is mechanically delicate, and even relatively light impacts can easily damage the components or misalign them. Even slight misalignment of endoscope components (such as precisely coupled optical components) can result in a significant decrease in image quality or render the endoscope unusable. If the components are misaligned, the amount of incident light traveling along the length of the endoscope decreases, resulting in little to no light at the distal end of the endoscope, rendering the endoscope unusable. Conventional endoscopes require such precise and complex optical components, and because these components can easily become misaligned, such endoscopes require frequent and expensive repair cycles to maintain image quality.

[0025] One solution to this challenge is to place the image sensor at the distal end within the endoscope itself. Such a solution eliminates the need for complex and precise acquisition of linked optical components that can easily become misaligned and / or damaged. This solution potentially approaches the optical simplicity, robustness, and cost-effectiveness universally achieved, for example, in the cameras of mobile phones. However, it should be understood that many of the advantages offered by endoscopes arise from the compactness of the distal end of the endoscope. If the distal end of the endoscope were enlarged to accommodate multiple different wavelength-sensitive pixel sensors conventionally used for color imaging or hyperspectral imaging, the pixel array may become too large, potentially causing the endoscope to no longer fit in confined spaces or to be disruptive or invasive when used in medical implementations. Because the distal end of the endoscope must remain very small, it is difficult to place one or more image sensors at the distal end. An acceptable solution to this method presents a set of engineering challenges of its own, which are by no means trivial, but among them is the fact that the sensors for color imaging and / or hyperspectral imaging must be contained within a very limited area. This is particularly difficult when the pixel array of a conventional camera includes separate pixel sensors for each of the red, green, and blue visible light bands, along with additional pixel sensors for other wavelength bands used for hyperspectral imaging. The area at the distal tip of the endoscope may be limited to left and right in the X and Y dimensions, but there is more space along the length of the endoscope tube in the Z dimension.

[0026] Many of the advantages of endoscopes arise from the miniaturization of the distal end; therefore, when the image sensor is located at the distal end, invasive constraints must be imposed on the image sensor area. These invasive constraints imposed on the sensor area inevitably result in fewer and / or smaller pixels in the pixel array. Reducing the number of pixels can directly affect spatial resolution, while reducing the pixel area can reduce the available signal capacity and the resulting sensitivity of the pixels, as well as the optimization of the number of pixels to maximize image quality, the minimum pixel resolution using maximum pixel quality and pitch, thus making resolution insignificant and reducing the signal-to-noise ratio (SNR) of each pixel. Reducing signal capacity reduces the dynamic range, i.e., the ability of the imaging device or camera to simultaneously capture all useful information from a scene with a wide range of light intensities. Various methods exist to extend the dynamic range of an imaging system beyond the dynamic range of its own pixels. However, all of these have some kind of loss (e.g., in resolution or frame rate) and can potentially introduce undesirable artifacts, which can be problematic in extreme cases. Reducing sensitivity results in requiring more light power to bring darker areas of the scene to an acceptable signal level. Lowering the f-number (increasing the aperture) can compensate for the loss of sensitivity, but at the expense of spatial distortion and a shallower depth of field.

[0027] In the sensor industry, complementary metal-oxide-semiconductor ("CMOS") image sensors have largely replaced conventional charge-coupled device ("CCD") image sensors in modern camera applications. Compared to CCD image sensors, CMOS image sensors are easier to integrate and operate, offer better or equivalent image quality, greater versatility, and lower cost. Typically, a CMOS image sensor can include the circuitry necessary to convert image information into digital data and subsequently incorporate various levels of digital processing. This can range from fundamental algorithms to compensate for non-idealities, such as those arising from variations in amplifier behavior, to a full image signal processing (ISP) chain providing image data in a standard red-green-blue (RGB) color space, for example, on-chip.

[0028] The control unit for the endoscope or image sensor can be located remotely from the image sensor, and may even be at a considerable physical distance from the image sensor. When the control unit is remote from the sensor, it may be preferable to transmit data within the digital domain, as this is less affected by interference noise and signal degradation compared to transmitting an analog data stream. It will be understood that various electro-digital signaling standards (e.g., LVDS (low voltage differential signaling), sub-LVDS, SLVS (scalable low voltage signaling), or other electro-digital signaling standards) can be used.

[0029] There may be a strong desire to minimize the number of electrical conductors, reduce the number of pads consuming space on the sensor, and lower the complexity and cost of sensor manufacturing. Adding analog-to-digital conversion to the sensor may be advantageous, but the additional area occupied by the conversion circuit is offset by the considerable reduction in analog buffering power required by the initial conversion to digital signals.

[0030] Regarding area consumption, considering the typical feature sizes available in CMOS image sensor (CIS) technology, in some implementations, it may be preferable to generate all internal logic signals on the same chip as the pixel array via a set of control registers and a simple command interface.

[0031] Some implementations of this disclosure may include combined sensor and system designs that enable high-resolution imaging with reduced pixel count in highly controlled lighting environments. This can be achieved by pulsing a single color wavelength on a frame-by-frame basis, and by switching or alternating each frame between a single different color wavelength and a specially designed corresponding monochrome sensor using a controlled light source in conjunction with a high frame acquisition speed. In addition, electromagnetic radiation outside the visible light spectrum can be pulsed to enable the generation of hyperspectral images. Pixels may be color-indistinguishable, and thus each pixel can generate data for each pulse of electromagnetic radiation, including pulses of red, green, and blue visible light wavelengths, along with other wavelengths that can be used for hyperspectral imaging.

[0032] As used herein, a monochrome sensor refers to an unfiltered imaging sensor. Since pixels are color-agnostic, their effective spatial resolution is considerably higher than that of their color relative to conventional single-sensor cameras (which are typically filtered by a Bayer pattern). Pixels can also have higher quantum efficiency because far less incident light is wasted between individual pixels. Furthermore, Bayer-based spatial color modulation requires a reduction in the modulation transfer function (MTF) of the associated optical components compared to the monochromatic case in order to blur color artifacts associated with the Bayer pattern. This has a negative impact on the actual spatial resolution that can be achieved using color sensors.

[0033] This disclosure also relates to a system solution for endoscopic applications where the image sensor is located at the distal end of the endoscope. In pursuing the smallest area sensor-based system, there are other design modes that can be developed besides reducing the number of pixels. The area of ​​the digital portion of the chip can be minimized. In addition, the number of connections to the chip (pad) can also be minimized. This disclosure describes a novel method for achieving these goals to realize such a system. This involves the design of a fully custom CMOS image sensor with several novel features.

[0034] For the purpose of facilitating the understanding of the principles presented herein, embodiments shown in the drawings are referenced herein and described using a specific language. Nevertheless, it will be understood that this is not intended to limit the scope of this disclosure. Any modifications and further alterations of the features of the invention illustrated herein, and any additional applications of the principles of this disclosure illustrated herein, would normally occur to those skilled in the art, and the ownership of this disclosure would be deemed to be within the scope of the claimed disclosure.

[0035] Before disclosing and describing structures, systems, and methods for generating images in low-light environments, it should be understood that this disclosure is not limited to such structures, structures, processes, and materials, and that the specific structures, configurations, processes, and materials disclosed herein may differ to some extent. Furthermore, it should be understood that the scope of this disclosure is limited only by the attached "Claims" and their equivalents, and that the terminology used herein is intended to describe only specific embodiments and is not intended to be limiting.

[0036] In the subject matter description and claims of this disclosure, the following technical terms will be used in accordance with the definitions set forth below.

[0037] When used in this specification and the appended claims, it should be noted that the singular forms "a," "an," and "the" include multiple references unless otherwise clearly indicated by the surrounding context.

[0038] As used herein, the terms “comprising,” “including,” “containing,” “characterized by,” and their grammatical equivalents are comprehensive, or broadly interpretable, terms that do not exclude further, undescribed elements or steps of method.

[0039] As used herein, the phrase "consisting of" and its grammatical equivalents exclude any element or process not specified in the claims.

[0040] As used herein, the phrase "essentially derived from" and its grammatical equivalents limit the scope of the claims to the specified materials or processes and to those that do not materially affect the fundamental and new features or features of the claimed disclosure.

[0041] As used herein, the term “proximal” broadly refers to the concept of the part closest to the origin.

[0042] As used herein, the term “distal” is the opposite of “proximal,” and therefore, depending on the context, generally refers to the part that is farther from the origin or the furthest part.

[0043] As used herein, a color sensor or multispectral sensor is a sensor known to have a color filter array (CFA) on it to filter incident electromagnetic radiation into its separate components. In the field of the electromagnetic spectrum, such a CFA may be constructed on or a modification thereof of a Bayer pattern to separate the green, red, and blue spectral components of light.

[0044] The following describes a system and method for generating images in low-light environments, with reference to Figures 1 to 5. Figure 1 illustrates a schematic diagram of a pair of sensors and electromagnetic emitters in operation for use in generating images in low-light environments. Such a configuration enables enhanced functionality in light-controlled or ambient-light-deficient environments.

[0045] As used herein, the term “light” means both particles and wavelengths, and refers to electromagnetic radiation detectable by a pixel array, and should be noted that this may include wavelengths in the visible and invisible spectra of electromagnetic radiation. The term “barrier” is used herein to mean a given range of wavelengths in the electromagnetic spectrum that is less than the whole spectrum, in other words, wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that may be controllable in terms of the physical properties of its components, the intensity of emission, or the duration of emission, or in terms of any portion of the electromagnetic spectrum in which it is emitted or can operate. An emitter may emit light in any dithered, diffused, or collimated emission type and may be controlled digitally or via analog methods or analog systems. As used herein, an electromagnetic emitter is a source of electromagnetic energy and includes light sources such as lasers, LEDs, incandescent light, or any light source that can be digitally controlled.

[0046] The pixel array of the image sensor may be electrically paired with an emitter so that they receive emission from each other during operation and are synchronized with adjustments made within the system. As can be seen in Figure 1, the emitter 100 can be tuned to emit electromagnetic radiation in the form of a laser, which can be pulsed to illuminate an object 110. The emitter 100 can be pulsed at intervals corresponding to the operation and function of the pixel array 122. The emitter 100 can pulse light at multiple electromagnetic partitions 105 so that the pixel array receives electromagnetic energy and generates a dataset corresponding (in time) to each particular electromagnetic partition 105. For example, Figure 1 shows a system having a monochrome sensor 120 with a pixel array (monochrome) 122 and a support circuit, where the pixel array 122 is sensitive to electromagnetic radiation of any wavelength. The optical emitter 100 illustrated in the figure may be a laser emitter capable of emitting a red electromagnetic barrier 105a, a blue electromagnetic barrier 105b, and a green electrical barrier 105c in any desired order. In one embodiment capable of generating a hyperspectral image, the optical emitter 100 can pulse electromagnetic radiation at any wavelength in the electromagnetic spectrum so as to generate a hyperspectral image. It will be understood that other optical emitters 100, such as digital or analog-based emitters, may be used in Figure 1 without departing from the scope of this disclosure.

[0047] During operation, a specific color or wavelength barrier can be assigned to the data generated by the monochrome sensor 120 for any individual pulse, and this assignment is based on the timing of the pulsed color or wavelength barrier from the emitter 100. Even if a pixel 122 is not dedicated to color, such a pixel can be assigned the color of any given dataset based on prior information about the emitter.

[0048] In an exemplary embodiment of this disclosure, emitter 100 pulses electromagnetic radiation at specialized wavelengths. Such pulses can enable the generation of specialized hyperspectral images particularly suitable for specific medical or diagnostic applications. In an exemplary embodiment, emitter 100 pulses electromagnetic radiation in the wavelengths of 513 nm to 545 nm, 565 nm to 585 nm, and 900 nm to 1000 nm.

[0049] In one embodiment, three datasets representing red, green, and blue electromagnetic pulses may be combined to form a single image frame. One or more additional datasets representing other wavelength barriers may be superimposed on the single image frame based on the red, green, and blue pulses. One or more additional datasets may represent pulses in the ranges of 513nm to 545nm, 565nm to 585nm, and 900nm to 1000nm, for example. One or more additional datasets may represent fluorescence imaging and / or hyperspectral imaging that can be superimposed on the single image frame based on the red, green, and blue pulses.

[0050] This disclosure is not limited to any particular combination of colors or any particular electromagnetic barrier, and any combination of colors or any electromagnetic barrier may use cyan, magenta, and yellow, ultraviolet, infrared, any combination of the aforementioned, or any other combination of colors, including all visible and invisible wavelengths, instead of red, green, and blue. In the figure, the imaged object 110 includes a red portion 110a, a green portion 110b, and a blue portion 110c. As shown in the figure, the reflected light from the electromagnetic pulse contains data only for the portion of the object having a specific color corresponding to the pulsed color barrier. Then, in 130, the image can be reconstructed by combining the datasets using these separate color (or color interval) datasets.

[0051] In one embodiment, multiple datasets representing red, green, and blue electromagnetic pulses, along with additional wavelength barriers along the electromagnetic spectrum, can be combined to form a single image frame having an RGB image with hyperspectral image data superimposed on the RGB image. Depending on the application or circumstances, different combinations of wavelength datasets may be desirable. For example, in some implementations, datasets representing specific wavelength barriers can be used to generate specialized hyperspectral images for diagnosing specific medical conditions, examining specific body tissues, and so on.

[0052] As illustrated in Figure 2, embodiments of the present disclosure may include or utilize a dedicated or general-purpose computer, including, for example, one or more processors and system memory, as will be discussed in detail below. Embodiments within the scope of the present disclosure may also include physical media and other computer-readable media that support, i.e., store, computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or dedicated computer system. A computer-readable medium that stores computer-executable instructions is a computer storage medium (device). A computer-readable medium that supports computer-executable instructions is a transmission medium. Thus, in non-limiting examples, embodiments of the present disclosure may include at least two distinctly different types of computer-readable media, i.e., computer storage media (devices) and transmission media.

[0053] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("solid state drives, SSDs") (e.g., RAM-enabled), flash memory, phase-change memory ("phase-change memory, PCM"), other forms of memory, other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired program code means in the form of computer-executable instructions or data structures, and that are accessible by a general-purpose or dedicated computer.

[0054] A “network” is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules and / or other electronic devices. In one implementation, sensors and camera control units may be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transferred to or provided to a computer over a network or by another communication connection (either wired, wireless, or a combination of wired and wireless), the computer appropriately perceives the connection as a transmission medium. The transmission medium may include networks and / or data links that can be used to convey desired program code means in the form of computer-executable instructions or data structures and are accessible by a general-purpose or dedicated computer. The above combination should also fall within the scope of computer-readable media.

[0055] Furthermore, program code means in the form of computer-executable instructions or data structures that, upon reaching various components of a computer system, can be automatically transferred from the transmission medium to the computer storage medium (device) (and vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., "NIC") and then ultimately transferred to the computer system RAM and / or a computer storage medium (device) with low volatility in the computer system. Examples of RAM include solid-state drives (SSDs or PCIx-enabled real-time memory tiered storage, such as FusionIO). Therefore, it should be understood that computer storage media (devices) can be included in the components of a computer system that also utilize (or primarily utilize) the transmission medium.

[0056] Computer-executable instructions include, for example, instructions and data that, when executed on a processor, cause a general-purpose computer, a dedicated computer, or a dedicated processing unit to perform a specific function or group of functions. Computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, or source code. While the subject matter has been described in language specific to structural features and / or actions of the method, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or actions. Rather, the described features and actions are disclosed as representative forms of performing this disclosure.

[0057] Those skilled in the art will understand that this disclosure can be put into practice in network computing environments having many forms of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, control units, camera control units, portable devices, handpieces, multiprocessor systems, microprocessor-enabled or programmable consumer electronic products, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, and various storage devices. It should be noted that any of the computing devices described above may be provided by or located within a brick-and-mortar location. This disclosure can also be put into practice in distributed system environments where local and remote computer systems, connected via a network (either by wired data links, wireless data links, or a combination of wired data links and wireless data links), both perform tasks. In a distributed system environment, program modules may reside in both local and remote memory storage devices.

[0058] Furthermore, where appropriate, the functions described herein can be performed by one or more hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) can be programmed to perform one or more of the systems and procedures described herein. Certain terms are used throughout the following description and claims to refer to specific system components. Components may be referred to by different names as will be understood by those skilled in the art. This specification does not intend to distinguish between components that have different names but the same function.

[0059] Figure 2 is a block diagram illustrating an exemplary computing device 150. The computing device 150 can be used to perform various procedures, such as those considered herein. The computing device 150 can function as a server, a client, or any other computing entity. The computing device 150 can perform various monitoring functions, such as those considered herein, and can run one or more application programs, such as the application programs described herein. The computing device 150 can be any of the many different types of computing devices, such as a desktop computer, a notebook computer, a server computer, a portable computer, a camera control unit, or a tablet computer.

[0060] The computing device 150 includes one or more processors 152, one or more storage devices 154, one or more interfaces 156, one or more mass storage devices 158, one or more input / output (I / O) devices 160, and a display device 180, all of which are connected to a bus 162. The processor(s) 152 include one or more processors or controllers that execute instructions stored in the storage devices 154 and / or mass storage devices 158. The processor(s) 152 may also include various types of computer-readable media, such as cache memory.

[0061] The storage device(s) 154 can be any of the various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 164) and / or non-volatile memory (e.g., read-only memory (ROM) 166). The storage device(s) 154 can also be any of the rewritable ROM, such as flash memory.

[0062] Examples of mass storage devices (multiple) 158 include various computer-readable media such as magnetic tape, magnetic disks, optical disks, and solid-state memory (e.g., flash memory). As shown in Figure 2, a specific mass storage device is a hard disk drive 174. Various drives may also be included in the mass storage device (multiple) 158 to enable reading from and / or writing to various computer-readable media. The mass storage device (multiple) 158 includes removable media 176 and / or non-removable media.

[0063] I / O devices 160 may include a variety of devices that enable the input or retrieval of data and / or other information to the computing device 150. Examples of I / O devices 160 include digital imaging devices, electromagnetic sensors and emitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs, or other image capture devices.

[0064] The display device 180 can be any type of device capable of displaying information to one or more users of the computing device 150. Examples of the display device 180 include monitors, display terminals, video projection devices, and the like.

[0065] Interfaces 106 may include a variety of interfaces that enable the computing device 150 to interact with other systems, devices, or computing environments. Exemplary interfaces 156 may include any number of different network interfaces 170, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet. Other interfaces may include user interfaces 168 and peripheral device interfaces 172. Interfaces 156 may also include one or more user interface elements 168. Interfaces 156 may also include one or more peripheral device interfaces, such as a printer, a pointing device (mouse, trackpad, etc.), and a keyboard.

[0066] Bus 162 enables the processor(s) 152, memory(s) 154, interface(s) 156, mass storage(s) 158, and I / O(s) 160 to communicate with each other and with other devices or components connected to bus 162. Bus 162 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, and USB bus.

[0067] For illustrative purposes, programs and other executable program components are shown herein as individual blocks, but it is understood that such programs and components may reside at different times within different storage components of the computing device 150 and may be executed by processor(s) 152. Alternatively, the systems and procedures described herein may be implemented in hardware, or in combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to perform one or more of the systems and procedures described herein.

[0068] Figure 2A illustrates the operating cycle of the sensor used in rolling readout mode or during sensor readout 200. Frame readout may be represented by a vertical line 210, where it may begin. Readout periods are represented by diagonal or slanted lines 202. The sensor can be read out row by row, with the top of the downward slanted edge being the top row 212 of the sensor and the bottom of the downward slanted edge being the bottom row 214 of the sensor. The time between the readout of the last row and the next readout cycle may be referred to as the blanking time 216. Note that some of the sensor pixel rows may be covered with a blackout (e.g., a metallic coating or a substantially black layer of any other kind of material). These covered pixel rows may be referred to as optical black rows 218 and 220. Optical black rows 218 and 220 can be used as input for a correction algorithm. As shown in Figure 2A, these optically black rows 218 and 220 can be located at the top of the pixel array, at the bottom of the pixel array, or at both the top and bottom of the pixel array. Figure 2B illustrates a process for controlling the amount of electromagnetic radiation, such as light, that is exposed to the pixels and thereby integrated or accumulated by the pixels. It will be understood that photons are the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into electric charge or electric current. An electron shutter or rolling shutter (shown by the dashed line 222) can be used to initiate integration time by resetting the pixels. The light can then be integrated until the next readout stage. The position of the electron shutter 222 can be moved between two readout cycles 202 to control pixel saturation for a given amount of light. Note that this technique allows for a constant integration time between two different lines, but introduces a delay when moving from the top row to the bottom row. Figure 2C illustrates the case where the electron shutter 222 is removed. In this configuration, the integration of incident light can be started during readout 202 and can be ended in the next readout cycle 202, which also defines the start of the next integration. Figure 2D shows a configuration without an electron shutter 222 but with controlled and pulsed light 230 during blanking time 216.This ensures that all rows see the same light emitted from the same light pulse 230. In other words, each row can begin its integration in a dark environment, which may be in the row 220 following the optical black of the readout frame (m) relative to the maximum light pulse width, then receive the light strobe, and then end its integration in a dark environment, which may be in the row 218 following the optical black of the next readout frame (m+1) relative to the maximum light pulse width. In the embodiment of Figure 2D, the image generated from the light pulse is available only during the readout of frame (m+1) without any interference with frames (m) and (m+2). Note that the condition for having a light pulse that is read out in only one frame and does not interfere with adjacent frames is that a given light pulse is emitted during the blanking time 216. Since the optical black rows 218, 220 are insensitive to light, the trailing optical black row 220 of frame (m) and the trailing optical black row 218 of frame (m+1) can be added to the blanking time 216 to determine the maximum range of the emission time of the light pulse 230. As illustrated in Figure 2A, the sensor can be cycled many times to receive data for each pulsed color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum). Each cycle can be time-tuned. In one embodiment, the cycles can be time-tuned to operate within 16.67 ms intervals. In another embodiment, the cycles can be time-tuned to operate within 8.3 ms intervals. It will be understood that other time intervals are contemplated and intended to be within the scope of the disclosure.

[0069] Figure 3 graphically illustrates the operation of an embodiment of an electromagnetic emitter. Since the emitter can time itself to correspond to the sensor cycle, electromagnetic radiation is emitted within and / or during a portion of the sensor operating cycle. Figure 3 illustrates pulse 1 at 302, pulse 2 at 304, and pulse 3 at 306. In one embodiment, the emitter can pulse during the read portion 202 of the sensor operating cycle. In one embodiment, the emitter can pulse during the blanking portion 216 of the sensor operating cycle. In one embodiment, the emitter can generate pulses for durations in two or more portions of the sensor operating cycle. In one embodiment, the emitter can start a pulse during the blanking portion 216 or during the optically black portion 220 of the read portion 202, and can end the pulse during the read portion 202 or during the optically black portion 218 of the read portion 202 of the subsequent cycle. Any of the above combinations are intended to fall within the scope of this disclosure, insofar as the emitter pulse and the sensor cycle coincide.

[0070] Figure 4 graphically represents the variation in duration and amplitude of emitted electromagnetic pulses (e.g., pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406) for controlling exposure. An emitter with a constant output amplitude can be pulsed at intervals to provide the required electromagnetic energy to the pixel array, during any of the cycles described above with reference to Figures 2D and 3. By pulsing with longer intervals, the emitter with a constant output amplitude can impart greater electromagnetic energy to the pixels, or by pulsing with shorter intervals, it can impart less electromagnetic energy. Whether longer or shorter intervals are required depends on the operating conditions.

[0071] In contrast to the emitter adjusting the time interval that pulses a constant output amplitude, the amplitude of the emission itself can be increased to provide more electromagnetic energy to the pixel. Similarly, reducing the pulse amplitude provides less electromagnetic energy to the pixel. It should be noted that one embodiment of the system may have the ability to adjust both amplitude and duration simultaneously, if desired. The sensor may also be adjusted to increase the desired sensitivity and duration for optimal image quality. Figure 4 illustrates the variation of pulse amplitude and duration. In the figure, pulse 1 at 402 has a larger amplitude or intensity than either pulse 2 at 404 or pulse 3 at 406. In addition, pulse 1 at 402 has a shorter duration than pulse 2 at 404 or pulse 3 at 406, and the electromagnetic energy provided by the pulse is illustrated by the area below the pulse shown in the figure. In the figure, pulse 2 at 404 has a relatively smaller amplitude or intensity and a longer duration compared to either pulse 1 at 402 or pulse 3 at 406. Finally, in the figure, pulse 3 at 406 has an intermediate amplitude or intensity and duration compared to pulse 1 at 402 and pulse 2 at 404.

[0072] Figure 5 shows a graphical representation of an embodiment of the present disclosure demonstrating an imaging system in operation by combining the operating cycles, electromagnetic emitter, and emitted electromagnetic pulses shown in Figures 2–4, in accordance with the principles and teachings of the present disclosure. As seen in the figure, the electromagnetic emitter pulses emission mainly during the sensor blanking period 216, thereby charging the pixels during the readout portion 202 of the sensor cycle and preparing them for readout. The dashed portions in the pulses (from Figure 3) illustrate the potential or ability to emit electromagnetic energy during the optically black portions 220 and 218 of the readout cycle (sensor cycle) 200 when additional time is required or desired to pulse the electromagnetic energy.

[0073] Referring below to Figures 6 to 9A, Figure 6 shows a schematic diagram of two different processes over the period t(0) to t(1) for recording video frames of full-spectrum light and segmented-spectrum light. Note that a color sensor has a color filter array (CFA) for filtering light of a specific wavelength per pixel, which is commonly used for full-spectrum light reception. An example of a CFA is a Bayer pattern. Because a color sensor may contain pixels in an array made sensitive to a single color from the entire spectrum, the pixel array has a pixel space dedicated to only a single light from the entire spectrum, resulting in low-resolution image results. Typically, such an arrangement is formed in a checkerboard pattern across the entire array.

[0074] In contrast, when a segmented spectrum of light is used, the pixel array is instructed to sense electromagnetic energy from predetermined partitions of the entire spectrum of electromagnetic energy in each cycle, thus allowing for the creation of a sensor that can be sensitive to or respond to the amplitude of all light energy. Therefore, to form an image, the sensor is circulated only through multiple different partitions within the entire spectrum of light, and then the image is reassembled to display a predetermined mixture of color values ​​for all pixels in the array. Thus, for each color pulse, a higher resolution image is also provided because the distance is reduced compared to a Bayer sensor between pixel centers of the same color sensitivity. As a result, the resulting color image has a higher modulation transfer function (MTF). Because the image from each color segmented frame cycle has a higher resolution, the resulting image produced when the segmented light frames are integrated into a full-color frame also has a higher resolution. In other words, since every pixel in the array (at most, every second pixel in the sensor with a color filter) senses the energy amplitude for a given pulse and a given scene, a high-resolution image can be created for each scene using less obtained (less accurate) data that needs to be introduced for only a very short individual time.

[0075] For example, white light or full-spectrum visible light is a combination of red, green, and blue light. In the embodiment shown in Figure 6, it can be seen that in both the segmented-spectrum process 620 and the full-spectrum process 610, the time for capturing the image is t(0) to t(1). In the full-spectrum process 610, at 612, white light or full-spectrum electromagnetic energy is emitted. At 614, white or full-spectrum electromagnetic energy is sensed. At 616, the image is processed and displayed. Thus, between times t(0) and t(1), the image is processed and displayed. Conversely, in the segmented-spectrum process 620, at 622, the first barrier is emitted and sensed at 624. At 626, the second barrier is emitted and then sensed at 628. At 630, the third barrier is emitted and then sensed at 632. At 634, the image is processed and displayed. Any system that uses an image sensor cycle at least twice as fast as the white light cycle is intended to be included within the scope of this disclosure.

[0076] In the embodiment shown in Figure 6, the sensors of the segmented spectral system 620 cycle three times for each of the entire spectral systems, as can be seen graphically between times t(0) and t(1). In the segmented spectral system 620, the first of the three sensor cycles relates to the green spectra 622 and 624, the second relates to the red spectra 626 and 628, and the third relates to the blue spectra 630 and 632. Therefore, in one embodiment where the display device (LCD panel) operates at 50 to 60 frames per second, the segmented optical system should operate at 150 to 180 frames per second to maintain the continuity and smoothness of the displayed image.

[0077] In other embodiments, different capture and display frame rates may exist. Furthermore, the average capture rate may be any multiple of the display rate.

[0078] In one embodiment, it may be desirable that not all barriers be represented equally within the system frame rate. In other words, all light sources must be pulsed with the same regularity in order to emphasize or de-emphasize aspects of the recorded scene as desired by the user. It should also be understood that invisible and visible barriers of the electromagnetic spectrum may be pulsed together within the system, and their respective data values ​​are stitched into the video output when desired for display to the user.

[0079] One embodiment may include the following pulse cycle pattern. i. Green pulse, ii. Red pulse, iii. Blue pulse, iv. Green pulse, v. Red pulse, vi. Blue pulse, vii. Infrared (IR) pulse, viii. (Repeat)

[0080] As can be seen in the embodiments, infrared barriers or special wavelength barriers (e.g., 513-545 nm, 565-585 nm, and / or 900-100 nm) can be pulsed at a different speed than the other barrier pulses. This may be done to highlight specific aspects of a scene, and the IR data can simply be superimposed with other data in the video output to achieve the desired highlighting. It should be noted that the addition of electromagnetic barriers to the tops of the red, green, and blue barriers does not necessarily require the serialized system to operate at four times the speed of the non-serial system for the entire spectrum, as it does not require all barriers to be represented equally within the pulse pattern. As seen in this embodiment, the addition of barrier pulses that represent fewer pulse patterns (infrared in the above embodiments) can result in an increase of less than 20% of the sensor's cycle speed to accommodate irregular barrier sampling.

[0081] In various embodiments, the pulse cycle pattern may include any of the following wavelengths in any preferred order. Such wavelengths may be particularly suitable for determining multispectral or hyperspectral image data, or for determining image data based on the relaxation emission of a fluorescent reagent. i.465±5nm, ii. 533±4nm, iii. 638±5nm, iv.780±5nm, v.805±5nm, vi.975±5nm, vii. 577±2nm, or viii. 523±4nm.

[0082] In one embodiment, a dye-sensitive electromagnetic barrier may emit a light used to highlight aspects of a scene. In this embodiment, it may be sufficient to highlight the location of the dye or material without requiring high resolution. In such an embodiment, the dye-sensitive electromagnetic barrier can be cycled much less frequently than other barriers in the system to contain the highlighted data.

[0083] The partition cycle may be divided to correspond to or approximate various imaging and video standards. In one embodiment, as best illustrated in Figures 7A to 7D below, the partition cycle may include pulses of electromagnetic energy in the red, green, and blue spectra. In Figure 7A, different light intensities are achieved by adjusting the width or duration of the light pulse within the operating range indicated by the gray vertical dashed line. In Figure 7B, different light intensities are achieved by adjusting the optical power or the power of an electromagnetic emitter, which may be a laser emitter or an LED emitter, while keeping the pulse width or duration constant. Figure 7C shows a case where both the optical power and the optical pulse width are adjusted, which leads to greater flexibility. The partition cycle can use CMY, IR, and ultraviolet, using a visible pulse source and an invisible pulse source mixed with any other color space, as required to generate an image or to approach a desired video standard that is currently known or may be further developed. It should also be understood that the system may be able to switch between color spaces on the fly to provide the desired image output quality.

[0084] In embodiments using the green-blue-red color space (as seen in Figure 7D), users may want to pulse the luminance component more frequently than the chrominance component, since they are generally more sensitive to differences in light amplitude than to differences in light color. This principle can be utilized using a monochrome sensor, as illustrated in Figure 7D. In Figure 7D, green, which contains the most luminance information, may be pulsed more frequently or with greater intensity in the (GBGRGBGR...) scheme to obtain luminance data. Such a configuration creates a video stream with more perceptible detail without creating and transmitting unperceptible data.

[0085] In one embodiment, duplicating weaker partition pulses may be used to generate an output tuned for weaker pulses. For example, blue laser light is considered weaker compared to the sensitivity of silicon-based pixels and is more difficult to generate compared to red or green light, and therefore may be pulsed more frequently during the frame cycle to compensate for the weakness of the light. These additional pulses can be achieved by using multiple lasers pulsed sequentially over time or simultaneously to produce the desired compensation effect. It should be noted that by pulsing during the blanking period (the time when the sensor is not reading the pixel array), the sensor is insensitive to differences / mismatches of the same type of laser and simply accumulates light for the desired output. In another embodiment, the maximum optical pulse range may differ between frames. This is shown in Figure 7E, where the optical pulses differ between frames. The sensor can be constructed so that different blanking times can be programmed by repeating patterns of two, three, four, or n frames. In Figure 7E, four different optical pulses are illustrated, where pulse 1 can be repeated, for example, after pulse 4, and can have a pattern of four frames with different blanking times. Using this technique, the strongest barrier can be placed for the shortest blanking time, and therefore the weakest barrier can have a wider pulse in one of the next frames without the need to increase the readout speed. The reconstructed frame can still have a regular pattern between frames when composed of many pulsed frames.

[0086] As can be seen in Figure 8, since the light in each divided range can have different energy values, the sensor and / or optical emitter can be adjusted to compensate for the differences in energy values. In 810, data obtained from the histogram from the previous frame can be analyzed. In 820, the sensor can be adjusted as described below. In addition, in 830, the emitter can be adjusted. In 840, the image can be obtained from the sensor at the adjusted sampling time, or the image can be obtained by the adjusted (either increased or decreased) emitted light, or a combination of the above. For example, since the red light spectrum is more easily detected by the sensor in the system than the blue light spectrum, the sensor can be adjusted to be less sensitive during the red partition cycle and more sensitive during the blue partition cycle, because the quantum efficiency of the blue partition is lower with respect to silicon (best illustrated in Figure 9). Similarly, the emitter can be adjusted to provide an adjusted partition (e.g., higher or lower intensity and duration). Furthermore, adjustments can be made at both the sensor level and the emitter level. The emitter can also be designed to emit at a single specific frequency, or, if desired for a particular application, it can be modified to emit multiple frequencies at specific diaphragms to broaden the spectrum of the emitted light.

[0087] Figure 10 shows a schematic diagram of a non-shared 4T pixel. The TX signal is used to transfer the stored charge from the photodiode (PPD) to the floating diffusion (FD). The reset signal is used to reset the FD to the reset bus. If the reset signal and the TX signal are "on" at the same time, the PPD is always reset (each photocharge generated in the PPD is collected directly in the reset bus), and the PPD is always empty. A typical implementation of a pixel array includes a horizontal reset line that attaches the reset signals for all pixels in a row, and a horizontal TX line that attaches the TX signals for all pixels in a row.

[0088] In one embodiment, the timing of sensor sensitivity adjustment is illustrated, and the adjustment of sensor sensitivity can be achieved using a global reset mechanism (i.e., means for simultaneously emitting a reset signal to all pixel arrays) and a global TX mechanism (i.e., means for simultaneously emitting a TX signal to all pixel arrays). This is shown in Figure 11. In this case, the optical pulse has a constant duration and amplitude, but the light integrated within all pixels starts from the "on" state of the global TX, transitions to the "off" state, and is then terminated by the optical pulse. Thus, modulation is achieved by shifting the falling edge of the global TX pulse.

[0089] Conversely, the emitter can emit red light at a lower intensity than blue light to produce a properly exposed image (best illustrated in Figure 12). In step 1210, data obtained from the histogram from the previous frame can be analyzed. In step 1220, the emitter can be adjusted. In step 1230, an image can be obtained from the adjusted synchrotron radiation. In addition, in one embodiment, the emitter and sensor can be adjusted simultaneously.

[0090] In some embodiments, reconstructing the segmented spectral frames into a full spectral frame for a later output may be as simple as blending the sensed values ​​for each pixel in the array. In addition, the blending and mixing of values ​​can be a simple average, or adjusted to a predetermined lookup table (LUT) of values ​​for the desired output. In one embodiment of a system using segmented optical spectra, the sensed values ​​can be pre-processed or remotely refined remotely from the sensor by an image or secondary processor and immediately before output to a display.

[0091] Figure 13 illustrates a basic embodiment of the monochrome ISP in the 1300 and how the ISP chain can be assembled for the purpose of generating an sRGB image sequence from raw sensor data obtained in a GRGB light pulsed scheme.

[0092] The first stage involves making corrections (see 1302, 1304, and 1306 in Figure 13) to compensate for any non-idealities in the sensor technology that are best suited to function in the raw data domain (see Figure 21).

[0093] In the next stage, each final frame derives data from three raw frames, and then buffers two frames (see 1308 and 1310 in Figure 13). At 1314, frame reconstruction begins with sampled data from the current frame and the two buffered frames (1308 and / or 1310). The reconstruction process yields the full color frame in a linear RGB color space.

[0094] In this embodiment, the white balance coefficient at 1318 and the color correction matrix at 1320 are applied at 1318 and 1320, respectively, before conversion to YCbCr space at 1322 for subsequent edge enhancement at 1324. Where applicable, after edge enhancement at 1324, the image is inversely converted to linear RGB at 1326 for scaling at 1328.

[0095] Finally, at 1330, the gamma transfer function is applied, and at 1332, the data is translated into the sRGB domain.

[0096] Figure 14 shows an embodiment of one design of color fusion hardware. In 1402, the color fusion hardware takes an RGBGRGBGRGBG video data stream and in 1405 converts it into a parallel RGB video data stream. The input bit width may be, for example, 12 bits per color. The output width in this embodiment is 36 bits per pixel. Other embodiments may have different initial bit widths and three times the number of output widths. In 1402, a memory writer block takes an RGBG video stream as its input and in 1404 writes each frame to its correct frame memory buffer (the memory writer triggers the same pulse generator 1410 that operates the laser light source). As illustrated in 1404, the write to memory follows a pattern of red, green 1, blue, green 2, and then red again. In 1406, the memory reader reads three frames simultaneously to construct RGB pixels. Each pixel is three times the bit width of the individual color components. At 1410, the reader also triggers the laser pulse generator. The reader waits until the red, green 1, and blue frames are written, then proceeds to read them, while the writer continues writing green 2 and then starts red again. Once red is complete, the reader begins reading from blue, green 2, and red. This pattern continues indefinitely.

[0097] Referring below to Figures 15 and 16, in one embodiment, the reconstruction of the RG1BG2 pattern illustrated in Figure 16 enables a 60fps output with a 120fps input. Each consecutive frame contains either the red or blue component from the previous frame. In Figure 16, each color component can be available for 8.3ms, and the resulting reconstructed frame has a duration of 16.67ms. Generally, in this pulsing scheme, the reconstructed frame has a duration of twice the duration of the incident colored frame, as shown in Figure 15. In other embodiments, different pulsing schemes can be employed. For example, an embodiment may be based on the timing (T1) of each color component or frame, and a reconstructed frame having a duration of twice the duration of the incident colored frame (2×T1). Different frames in sequence may have different frame durations, and the average capture rate may be a multiple of the final frame rate.

[0098] Figures 17-20 illustrate schematic diagrams of color correction methods and hardware for use with a segmented light system. In digital imaging, it is common to manipulate values ​​within image data to correct the output to meet user expectations or to emphasize specific aspects of the imaged object. Most commonly, this is done in satellite imagery where one data type is tuned and adjusted to emphasize the other. Often, satellite-acquired data contains the entire spectrum of available electromagnetic energy, i.e., the light source is uncontrolled, i.e., the sun. In contrast, there are imaging conditions where the light is controlled and even provided by the user. In such situations, image data calibration is still desirable, as without calibration, inappropriate emphasis may be applied to certain data compared to other data. In systems where the light is controlled by the user, it is advantageous to provide light emissions that are known to the user and can be only a portion of the electromagnetic spectrum or multiple portions of the entire electromagnetic spectrum. Calibration remains important to meet user expectations and to check for faults in the system. One calibration method may be a table of expected values ​​for a given imaging state that can be compared with data from the sensor. One embodiment may include a color-neutral scene having known values ​​to be output by the imaging device, and the device can be adjusted to satisfy such known values ​​when sampling the color-neutral scene.

[0099] During use and upon startup, the system can sample a color-neutral scene at 1710 (as illustrated in Figure 17) by operating a full cycle of multiple electromagnetic spectral partitions at 1702. At 1704, a table of values ​​1708 can be formed to generate a histogram of frames. At 1706, the frame values ​​can be compared to known or expected values ​​from a color-neutral scene. Then, at 1712, the imaging device can be adjusted to meet the desired output. In one embodiment illustrated in Figure 17, the system may include an image signal processor (ISP) that can adjust the imaging device to color-correct.

[0100] It should be noted that since each segmented spectrum of light may have different energy values, the sensor and / or optical emitter may be tuned to compensate for these energy differences. For example, in one embodiment, with respect to a silicon-based imaging device, since the blue light spectrum has a lower quantum efficiency than the red light spectrum, the sensor's response can be tuned to be less responsive during the red cycle and more responsive during the blue cycle. Conversely, the emitter can emit blue light at a higher intensity because the quantum efficiency of blue light is lower than that of red light for producing a properly exposed image.

[0101] In one embodiment illustrated in Figure 18, where a light source emission is provided and controllable by the system, the emission of such light can be adjusted in 1800 to perform color correction of the image. The adjustment can be made to any aspect of the emitted light, such as amplitude, duration (i.e., time-on), or range within the spectral partition. In addition, in some embodiments as shown in Figure 19, the emitter and sensor can be adjusted simultaneously.

[0102] As can be seen in Figure 20, subdivided adjustments can be made to the sensors or emitters within the system to reduce the amount of noise and artifacts in the output image stream or video. Figure 20 illustrates a system 2000 in which both the emitter 2006 and the sensor 2008 can be adjusted, but imaging devices in which either the emitter or the sensor is adjusted during use or over a portion of use are also conceivable and within the scope of this disclosure. It may be advantageous to adjust only the emitter during one portion of use, only the sensor during another portion of use, or both simultaneously during another portion of use. In any of the embodiments described above, improved image quality can be obtained by limiting the overall adjustments that the system can make during a frame cycle. In other words, in one embodiment, the emitter can be limited so that only a portion of its operating range can be adjusted at any time between frames. Similarly, the sensor can be limited so that only a portion of its operating range can be adjusted at any time between frames. Furthermore, in one embodiment, both the emitter and the sensor can be limited so that only a portion of their respective operating ranges can be adjusted together at any time between frames.

[0103] In one exemplary embodiment, adjustments to some components within the system may be made, for example, by about 0.1 dB of the component's operating range to compensate for the exposure of a previous frame. It should be noted that this 0.1 dB is merely an example, and in other embodiments, possible adjustments to components may be any part of their respective operating ranges. Components of the system can generally be modified by adjusting their intensity or duration, which is controlled by the number of bits (resolution) output by the component. The resolution of a component can typically range from about 10 to 24 bits, but should not be limited to this range, as it is intended to include resolutions of components to be further developed in addition to those currently available. For example, if, after a first frame, the scene is judged to be too blue upon observation, the emitter can be adjusted to reduce the amplitude or duration of the blue light pulses during the system's blue cycle by a partial adjustment, such as about 0.1 dB, as considered above.

[0104] In this exemplary embodiment, although it may have been necessary to exceed 10 percent, the system limited itself to a 0.1 dB adjustment of the operating range per system cycle. Thus, the blue light can then be readjusted as needed during the next system cycle. Subdivision adjustments between cycles can have a damping effect on the output image and reduce noise and artifacts when operating the emitters and sensors at their extreme operating limits. It can be determined that any small amount of adjustment range of the operation of any component can be used as a limiting factor, or that a particular embodiment of the system may have components that can be adjusted over the entire operating range of the component.

[0105] In addition, the optically black area of ​​any image sensor can be used to assist in image correction and noise reduction. In one embodiment, values ​​read from the optically black area can be compared with values ​​from the active pixel area of ​​the sensor to establish a reference point for use when processing image data. Figure 21 shows a type of sensor correction process that may be employed in a color pulsed system. CMOS image sensors typically have several non-idealities that negatively affect image quality, especially in low light. The main ones are fixed pattern noise and line noise. Fixed pattern noise (FPN) is variation in the offset of the sensing element. Typically, most FPN is inter-pixel variation, among other sources, caused by random fluctuations in the dark current between photodiodes. This looks very unnatural to the viewer. Even more severe is column FPN, which is caused by offsets in the readout chain associated with a particular row of pixels. This results in the perception of vertical stripes in the image.

[0106] Overall illumination control has the advantage that entire frames of dark data can be periodically acquired and used to correct pixel and column offsets. In the illustrated embodiment, a single frame buffer can be used to perform an operational average of the entire frame without illumination, for example, using simple exponential smoothing. This dark average frame is subtracted from every illuminated frame during normal operation.

[0107] Line noise is the probabilistic time variation of the pixel offset within each row. Since this is transient, a new correction must be calculated for each row and each frame. For this purpose, typically, a large number of optically blind (OB) pixels exist within each row of the array. These must be sampled first to evaluate the line offset before sampling the light-sensitive pixels. Then, during the line noise correction process, the line offset is simply subtracted.

[0108] In the embodiment shown in Figure 21, other corrections exist regarding acquiring data in the correct order, monitoring and controlling the voltage in the analog domain (black clamp), and identifying / correcting individual defective pixels.

[0109] Figures 22 and 23 illustrate schematics of methods and hardware for increasing dynamic range in a closed or restricted light environment. In one embodiment, the exposure input may be different levels over time and can be combined to produce a larger dynamic range. As can be seen in Figure 22, the imaging system can achieve a larger dynamic range by cycling at a first intensity over a first cycle in 2202, then at a second intensity over a second cycle in 2204, and then by combining the first and second cycles into a single frame in 2206. A larger dynamic range may be particularly desirable because the spatial environment in which the imaging device is used is restricted. In restricted spatial environments that are light-deficient or dark except for the light provided by a light source, and when the light source is close to the light emitter, exposure has an exponential relationship with distance. For example, objects close to the light source and the optical aperture of the imaging device tend to be overexposed, while distant objects tend to be very underexposed because there is little (or no) ambient light.

[0110] As can be seen in Figure 23, in 2300, the cycle of a system having electromagnetic energy emission at multiple partitions can be cycled continuously according to the partitions of the electromagnetic spectrum. For example, in an embodiment in which the emitter emits lasers at different red partitions, different blue partitions, and different green partitions, the two sets of cycle data to be combined may take the following forms. i.2302 is red with an intensity of 1. ii.2304, red with intensity 2, iii.2302, blue with intensity 1, In iv.2304, blue with intensity 2, In v.2302, the green color has an intensity of 1. In vi.2304, it's green with an intensity of 2.

[0111] Alternatively, the system can be cycled in the following ways: i.2302 is red with an intensity of 1. ii.2302, blue with intensity 1, iii.2302 is green with an intensity of 1. iv.2304 is red with an intensity of 2. In v.2304, blue with intensity 2. In vi.2304, it's green with an intensity of 2.

[0112] In one such embodiment, the first image can be derived from one intensity value, and the second image can be derived from two intensity values, which can then be combined or processed in 2310 as a complete image dataset rather than as components thereof.

[0113] It is conceivable that any number of emission partitions may be used in any order, which is within the scope of this disclosure. As seen in Figure 23, "n" is used as a variable to represent any number of electromagnetic partitions and "m", where "m" is used to represent any level of intensity of the "n" partitions. Such a system can be cycled as follows: i.2306, n of intensity m, ii. n+1 of intensity m+1, iii. n+2 of intensity m+2, In iv.2308, n+i is of intensity m+j.

[0114] Therefore, any pattern of serialized cycles can be used to generate the desired image correction, where "i" and "j" are additional values ​​within the operating range of the imaging system.

[0115] Digital color cameras incorporate image processing steps to maximize color fidelity. This is achieved through a 3x3 matrix known as the Color Correction Matrix (CCM).

[0116]

number

[0117] The terms in the CCM are adjusted using a set of reference colors (e.g., from a Macbeth chart) to provide the best overall match to the sRGB standard color space. The diagonal terms, a, e, and i, are effective white balance gains. However, typically, white balance is applied separately, and the sum of the horizontal rows is constrained to be singular so that the net gain is not applied by the CCM itself. The off-diagonal terms effectively address color crosstalk within the input channels. Thus, since the color filer array has a lot of response overlap between channels, Bayer sensors have higher off-diagonal than channels, as does a 3-chip camera.

[0118] There is a signal-to-noise ratio penalty for color correction that depends on the amplitude of off-diagonal terms. A virtual sensor with channels that perfectly match the sRGB components has the following identity matrix CCM:

[0119]

number

[0120] The signal-to-noise ratio evaluated in the green channel for a perfect white light signal of 10,000e / pixel (ignoring readout noise) in this case is as follows:

[0121]

number

[0122] Any deviation from this point forward will reduce the SNR. For example, considering a CCM that has a normal value for a Bayer CMOS sensor, the following occurs.

[0123]

number

[0124] In this case, the green SNR will be as follows:

[0125]

number

[0126] Figure 24 shows the results of a total SNR simulation using D65 illumination for a typical Bayer sensor CCM with an identity matrix-adjusted CCM. The SNR evaluated for the luminance component is approximately 6 dB worse as a result of color correction.

[0127] Since the system described in this disclosure uses monochrome illumination of multiple distinct wavelengths, there is naturally no color crosstalk. The X marks in Figure 25 indicate the locations of the three wavelengths available via the laser diode sources (465, 532, and 639 nm) compared to the sRGB range indicated by the triangles.

[0128] In this case, compared to a Bayer sensor, the off-diagonal term of the CCM is significantly reduced, providing a considerable SNR advantage.

[0129] Figure 26 illustrates an imaging system with increased dynamic range, provided by the pixel configuration of the image sensor's pixel array. As can be seen in the figure, adjacent pixels 2602 and 2604 can be set to different sensitivities, thereby each cycle containing data generated by pixels that are more and less sensitive to each other. Multiple sensitivities can be recorded in a single-cycle array, thus increasing the dynamic range, in contrast to the time-dependent continuous nature of other embodiments when recorded in parallel.

[0130] In one embodiment, the array may comprise rows of pixels that can be arranged within a row based on their sensitivity. In one embodiment, pixels of different sensitivities may be arranged throughout the array, alternating with respect to their nearest adjacent pixels within a row or column, based on their sensitivity, to form a checkerboard pattern. The above can be achieved through a shared arrangement of any pixel circuits or through an arrangement of any independent pixel circuits.

[0131] A wide dynamic range can be achieved by having multiple global TXs, each TX emitting to only one different set of pixels. For example, in global mode, global TX1 signals illuminate the first set of pixels, global TX2 signals illuminate the second set of pixels, ..., global TXn signals illuminate the nth set of pixels.

[0132] Based on Figure 11, Figure 27A shows an embodiment of timing for the sensitivity of two different pixels in a pixel array (dual-pixel sensitivity). In this case, the global TX1 signal illuminates half of the pixels in the array, and global TX2 illuminates the other half. Since global TX1 and global TX2 have different edge positions from "on" to "off", the integrated light is different between the TX1 and TX2 pixels. Figure 27B shows a different embodiment of the timing for dual-pixel sensitivity. In this case, the light pulse is modulated twice (pulse duration and / or amplitude). The TX1 pixel integrates the P1 pulse, and the TX2 pixel integrates the P1+P2 pulse. Separation of the global TX signal can be done in numerous ways. The following are examples. i. Distinguish the TX line from each other. ii. Multiple TX lines are transmitted per row, each addressing a different set of pixels.

[0133] One implementation describes a means for providing video with a wide dynamic range, utilizing the color pulsed system described in this disclosure. The basis of this means is having multiple flavor pixels, or pixels that can be adjusted differently, within the same monochrome array, which can integrate incident light over different durations within the same frame. An example of a pixel arrangement in such a sensor array is a uniform checkerboard pattern throughout, having two independent variable integration times. In this case, both red and blue information can be provided within the same frame. In practice, since the two integration times can be adjusted on a frame-by-frame basis, it is possible to do this simultaneously, for example, to extend the dynamic range of the green frame when most needed. The advantage is that color motion artifacts are not much of a problem when all data is derived from two frames for three. Of course, there is a subsequent loss of spatial resolution for the red and blue data, but this has less impact on image quality compared to green, as the green data occupies the luminance component.

[0134] An inherent characteristic of a monochrome wide-dynamic-range (WDR) array is that pixels with long integration times must integrate the upper set of light seen by pixels with short integration times. This is desirable for standard wide-dynamic-range operation in the green frame. For the red and blue frames, pulsing must be controlled in conjunction with the exposure period, for example, providing blue light from the start of a long exposure and switching to red when short-exposure pixels are turned on (both types of pixels have their charges transferred simultaneously).

[0135] In the color fusion stage, the two flavors of a pixel are separated into two buffers. Then, empty pixels are filled in, for example, using linear interpolation. At this point, one buffer contains the blue data and the other contains the entire red + blue image. The blue buffer can be subtracted from the second buffer to obtain the pure red data.

[0136] Figures 28A to 28C illustrate the use of white light emission, which is pulsed and / or synchronized by a corresponding color sensor, or kept constant. As can be seen in Figure 28A, a white light emitter can be configured to emit a beam of light during the blanking period of the corresponding sensor, providing a controlled light source within a controlled light environment. The light source can emit a beam with a constant amplitude and vary the pulse duration, as seen in Figure 28A, or, as illustrated in Figure 28B, can maintain a constant pulse while varying the amplitude to achieve correct exposure data. Figure 28C illustrates a graphical representation of a constant light source that can be adjusted by varying the current controlled and synchronized with the sensor.

[0137] In one embodiment, white light or multispectral light can be emitted as pulses, as desired, to provide data for use within the system (best illustrated in Figures 28A-28C). White emission, combined with an electromagnetic spectrum barrier, may be useful for highlighting and suppressing specific aspects of a scene. Such embodiments may utilize the following pulsing patterns. i. Green pulse, ii. Red pulse, iii. Blue pulse, iv. Green pulse, v. Red pulse, vi. Blue pulse, vii. White light (multispectral) pulses, viii. (Repeat)

[0138] Any system using an image sensor cycle at least twice as fast as the white light cycle is intended to be included within the scope of this disclosure. It will be understood that any combination of electromagnetic spectral dividers, whether from the visible or invisible spectrum of the entire electromagnetic spectrum, can be conceived herein.

[0139] Figures 29A and 29B illustrate perspective and side views, respectively, of an implementation of a monolithic sensor 2900 having multiple pixel arrays for generating a three-dimensional image, in accordance with the teachings and principles of this disclosure. Such an implementation may be desirable for three-dimensional image acquisition, and the two pixel arrays 2902 and 2904 may be offset during use. In another implementation, the first pixel array 2902 and the second pixel array 2904 may be dedicated to receiving electromagnetic radiation of a predetermined range of wavelengths, and the first pixel array may be dedicated to electromagnetic radiation of a different range of wavelengths than the second pixel array.

[0140] Figures 30A and 30B illustrate perspective and side views, respectively, of an image sensor 3000 mounted on multiple substrates. As illustrated, multiple pixel rows 3004 forming a pixel array are located on a first substrate 3002, and multiple circuit rows 3008 are located on a second substrate 3006. Electrical connections and communication between a pixel row and its associated or corresponding circuit row are also illustrated in the figures. In one mounting, the image sensor, which otherwise might be manufactured with the pixel array and supporting circuits on a single monolithic substrate / chip, can have the pixel array separated from all or most of the supporting circuits. The disclosure may use at least two substrates / chips stacked on top of each other using three-dimensional stacking techniques. The first of the two substrates / chips, 3002, can be processed using an image CMOS process. The first substrate / chip 3002 may consist solely of a pixel array, or a pixel array surrounded by limited circuits. The second or subsequent substrate / chip 3006 can be processed using any process and does not need to be an image CMOS process. The second substrate / chip 3006 may be a high-density digital process to integrate various and some functions within a very limited space or area on the substrate / chip, or it may be a mixed-mode or analog process to integrate, for example, precise analog functions, or it may be an RF process to implement wireless capabilities, or it may be a MEMS (Micro-Electro-Mechanical Systems) to integrate MEMS devices, but is not limited to these. The image CMOS substrate / chip 3002 can be stacked with the second or subsequent substrate / chip 3006 using any three-dimensional technique. The second substrate / chip 3006 can support most or most of the circuits that would otherwise be implemented in the first image CMOS chip 3002 as peripheral circuits (when mounted on a monolithic substrate / chip), thus increasing the overall system area while keeping the pixel array size constant and optimized to the greatest extent possible.Electrical connections between the two substrates / chips can be made through interconnects 3003 and 3005, which may be wire bonds, bumps, and / or TSVs (Through Silicon Vias).

[0141] Figures 31A and 31B illustrate perspective and side views, respectively, of an implementation configuration of an imaging sensor 3100 having multiple pixel arrays for generating a three-dimensional image. The three-dimensional image sensor can be constructed on multiple substrates and may include multiple pixel arrays and other associated circuits. Multiple pixel rows 3104a forming a first pixel array and multiple pixel rows 3104b forming a second pixel array are located on substrates 3102a and 3102b, respectively, while multiple circuit rows 3108a and 3108b are located on a separate substrate 3106. Electrical connections and communications between pixel rows and associated or corresponding rows of circuits are also shown.

[0142] It will be recognized that the teachings and principles of this disclosure may be used in reusable device platforms, limited-use device platforms, relocatable-use device platforms, or single-use / disposable device platforms without departing from the scope of this disclosure. In reusable device platforms, it will be recognized that the end user is responsible for cleaning and sterilizing the device. In limited-use device platforms, the device can be used a specified number of times before it becomes inoperable. A typical new device is supplied sterile for further use requiring cleaning and sterilization by the end user. In relocatable-use device platforms, a third party can reprocess the device into a single-use device (e.g., by cleaning, packaging, and sterilizing) for further use at a lower cost than a new unit. In single-use / disposable device platforms, the device is supplied sterile to the operating room and used only once before disposal.

[0143] One embodiment of the emitter may employ the use of a mechanical shutter and filters to create pulsed color light. As illustrated in Figure 32, an alternative method for generating pulsed color light uses a system 3200 of a white light source, mechanical color filters, and a shutter. The wheel may include a translucent color filter window and opaque sections of a certain pattern for shuttering. The opaque sections do not allow light to pass through and also create a dark period during which sensor readout occurs. The white light source can be based on any technology, such as a laser, LED, xenon, halogen, metal halide, or others. The white light can be projected through a series of color filters 3207, 3209, and 3211 of colored light pulses of a desired pattern. The pattern in one embodiment may be a red filter 3207, a green filter 3209, a blue filter 3211, and a green filter 3209. The filter and shutter system 3200 can be mounted on a wheel that rotates at a frequency necessary to synchronize with the sensor, such that knowledge of the arch length and rotational speed of the mechanical color filters 3207, 3209, and 3211, and the shutter 3205, provides timing information for the operation of the corresponding monochrome image sensor.

[0144] The embodiment illustrated in Figure 33 may include patterns of only translucent color filters 3307, 3309, and 3311 on the filter wheel 3300. Different shutters can be used in this configuration. The shutter can be mechanical and its size can be varied to dynamically adjust the "pulse" duration. Alternatively, the shutter can be electronic and incorporated into the sensor design. The motor that rotates the filter wheel 3300 needs to communicate with the sensor or be controlled in conjunction with the sensor so that knowledge of the arch length and rotation speed of the mechanical color filter systems 3307, 3309, and 3311 provides timing information for the operation of the corresponding monochrome image sensor. The control system needs to know the appropriate color filter for each frame captured by the sensor so that the full-color image can be properly reconstructed within the ISP. Although RGBG color patterns are shown, other colors and / or patterns can be used if advantageous. The relative sizes of the color sections are shown to be equal, but can be adjusted if advantageous. The mechanical structure of the filter is shown as a rotatably moving circle, but it can be a rectangle with linear movement, or a different shape with different movement patterns.

[0145] As illustrated in Figure 34, an embodiment for pulsing colored light may consist of a mechanical wheel or barrel holding electronics and heatsinks for red, green, blue, or white LEDs. The LEDs are spaced apart by a distance related to the rotational or torsional speed of the barrel or wheel to allow for time adjustment of the light pulsation, consistent with other embodiments in this patent. The wheel or barrel is rotated using an electric motor and a mechanical bracket to which the wheel or barrel is mounted. The motor is controlled using a microcontroller, FPGA, DSP, or other programmable device, including a control algorithm for appropriate timing as described in this patent. On one side, there is a mechanical opening that is optically coupled to an optical fiber to transport the fiber to the end of the scope in the manner described in this patent. This coupling may also have a mechanical opening that can be opened and closed to control the amount of light allowed to descend the optical fiber cable. This is a mechanical shutter device, and alternatively, an electronic shutter designed for a CMOS or CCD sensor can be used. This device is difficult to control and calibrate during manufacturing, but is another way in which pulsed light can pass through the system.

[0146] Figure 35 illustrates one embodiment of an emitter 3502 equipped with a linear filter 3504 and a shutter mechanism to provide pulsed electromagnetic radiation. The linear filter 3504 and shutter mechanism move horizontally at the required frequency to filter out light of the appropriate wavelength.

[0147] Figure 36 illustrates one embodiment of an emitter 3602 comprising a prism filter 3604 and a shutter mechanism for providing pulsed electromagnetic radiation. The prism filter 3604 filters light and delivers it to an output that may include a shutter. The prism filter 3604 moves at the required frequency to provide the correct color output pattern.

[0148] Furthermore, the teachings and principles of this disclosure may encompass all wavelengths of electron energy, including visible and invisible spectra, such as infrared (IR), ultraviolet (UV), and X-rays.

[0149] Figure 37 is a schematic diagram illustrating a system 3700 for providing illumination in low-light environments, such as for endoscopic imaging. System 3700 can be used in combination with any of the systems, methods, or apparatus disclosed herein. System 3700 includes a light source 3702, a controller 3704, a jumper waveguide 3706, a waveguide connector 3708, a lumen waveguide 3710, a lumen 3712, and an image sensor 3714 having associated optical components (such as lenses). The light source 3702 generates light that travels through the jumper waveguide 3706 and the lumen waveguide 3710 to illuminate the scene at the distal end of the lumen 3712. The light source 3700 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, and other wavelengths. The lumen 3712 can be inserted into the patient's body for imaging during a procedure or examination, etc. Light is output as illustrated by the dashed line 3716. The scene illuminated by the light can be captured using the image sensor 3714 and displayed to a physician or any other medical professional. The controller 3704 can provide control signals to the light source 3702 to control when illumination is provided to the scene. In one embodiment, the light source 3702 and the controller 3704 are located within a camera control unit (CCU) or external console to which the endoscope is connected. If the image sensor 3714 includes a CMOS sensor, light can be periodically provided to the scene in a series of illumination pulses during the readout period of the image sensor 3714, during a period known as the blanking period. Thus, the light can be pulsed in a controlled manner to avoid superimposing onto the readout period of image pixels in the pixel array of the image sensor 3714.

[0150] In one embodiment, the lumen waveguide 3710 includes one or more optical fibers. The optical fibers can be made of low-cost materials such as plastic to allow for the disposal of the lumen waveguide 3710 and / or other parts of the endoscope. In one embodiment, a single glass fiber having a diameter of 500 microns can be used. The jumper waveguide 3706 can be permanently attached to the light source 3702. For example, the jumper waveguide 3706 can receive light from an emitter in the light source 3702 and provide that light to the lumen waveguide 3710 at the location of the connector 3708. In one embodiment, the jumper waveguide 106 may include one or more glass fibers. The jumper waveguide may include any other type of waveguide for guiding the light to the lumen waveguide 3710. The connector 3708 selectively connects the jumper waveguide 3706 to the lumen waveguide 3710, allowing light in the jumper waveguide 3706 to pass through to the lumen waveguide 3710. In one embodiment, the lumen waveguide 3710 can be directly connected to a light source without any intervening jumper waveguide 3706.

[0151] Figures 38 to 40 are schematic block diagrams illustrating a light source 3800 having multiple emitters. With respect to Figure 38, the emitters include a first emitter 3802, a second emitter 3804, and a third emitter 3806. Further emitters may be included, as will be discussed further below. Emitters 3802, 3804, and 3806 may include one or more laser emitters that emit light having different wavelengths. For example, the first emitter 3802 may emit a wavelength matching a blue laser, the second emitter 3804 may emit a wavelength matching a green laser, and the third emitter 3806 may emit a wavelength matching a red laser. For example, the first emitter 3802 may include one or more blue lasers, the second emitter 3804 may include one or more green lasers, and the third emitter 3806 may include one or more red lasers. The emitters 3802, 3804, and 3806 emit laser beams toward a collection area 3808, which may be located at the position of a waveguide, lens, or other optical component for collecting and / or supplying light to a waveguide, such as the jumper waveguide 3706 or the lumen waveguide 3710 in Figure 37.

[0152] In one implementation where a patient is administered a reagent or dye to aid in the identification of a specific tissue, structure, chemical reaction, biological process, etc., emitters 3802, 3804, and 3806 may emit wavelengths(or wavelengths) to cause the reagent or dye to fluoresce. Such wavelengths(or wavelengths) may be determined based on the reagent or dye administered to the patient. In such an embodiment, the emitters may need to be highly precise in order to emit the desired wavelength(or wavelength) to cause a particular reagent or dye to fluoresce or activate them.

[0153] In the embodiment shown in Figure 38, emitters 3802, 3804, and 3806 each deliver laser light to the collection area 3808 at different angles. The variation in angle can result in variations in electromagnetic energy as it is located within the output waveguide. For example, if the light enters the fiber bundle (glass or plastic) directly in the collection area 3808, the angle variation can cause different amounts of light to enter different fibers. For instance, this angle can result in variations in intensity across the collection area 3808. Furthermore, since the light from different emitters cannot be mixed uniformly, some fibers may receive different amounts of light of different colors. Variations in the color or intensity of light in different fibers can lead to suboptimal illumination of the scene. For example, variations in delivered light or light intensity can occur in the scene and the captured image.

[0154] In one embodiment, an intervening optical element can be placed between the fiber bundle and emitters 3802, 3804, and 3806 to mix light of different colors (wavelengths) before it enters the fiber or other waveguide. Exemplary intervening optical elements include diffusers, mixing rods, one or more lenses, or other optical components that mix the light so that a given fiber receives an equal amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. While this mixing may result in each fiber having the same color, in some embodiments it may still result in different total luminances delivered to different fibers. In one embodiment, the intervening optical element may also diffuse or uniformize the light over the collection area so that each fiber carries the same total amount of light (for example, the light may be diffused in a top-hat profile). Diffusers or mixing rods may lead to light loss.

[0155] Although the collection region 3808 is represented as a physical component in Figure 38, the collection region 3808 may simply be the region through which light from emitters 3802, 3804, and 3806 is delivered. In some cases, the collection region 3808 may include optical components such as diffusers, mixing rods, lenses, or any other intervening optical components between emitters 3802, 3804, and 3806 and the output waveguide.

[0156] Figure 39 illustrates one embodiment of a light source 3800 having emitters 3802, 3804, and 3806 that provide light to a collection area 3808 at the same or substantially the same angle. The light is provided at an angle substantially perpendicular to the collection area 3808. The light source 3800 includes a plurality of dichroic mirrors, including a first dichroic mirror 3902, a second dichroic mirror 3904, and a third dichroic mirror 3906. The dichroic mirrors 3902, 3904, and 3906 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 3906 can reflect blue laser light provided by the third emitter, while transmitting red and green light provided by the first emitter 3802 and the second emitter 3804, respectively. The second dichroic mirror 3904 can transmit red light from the first emitter 3802 but reflect green light from the second emitter 3804. If other colors or wavelengths are present, the dichroic mirror can reflect light corresponding to at least one emitter and transmit light corresponding to other emitters. For example, the third dichroic mirror 3906 reflects light from the third emitter 3806 but reflects that light to "rear" emitters such as the first emitter 3802 and the second emitter 3804. In embodiments with tens or hundreds of emitters, each dichroic mirror can reflect light to one or more corresponding emitters in front of it, while transmitting it to emitters behind it. This allows tens or hundreds of emitters to emit electromagnetic energy into the collection area 3808 at substantially the same angle.

[0157] Since dichroic mirrors allow other wavelengths to be transmitted or passed through, each wavelength can reach the collection area 3808 from the same angle and / or at the same center or focal point. Providing light from the same angle and / or the same focal / center point can significantly improve the reception and color mixing in the collection area 3808. For example, a particular fiber can receive different colors in the same ratio as transmission / reflection by emitters 3802, 3804, 3806 and mirrors 3902, 3904, 3906. The mixing of light can be significantly improved in the collection area compared to the embodiment in Figure 38. In one embodiment, any optical component considered herein can be used in the collection area 3808 to collect light before providing it to a fiber or fiber bundle.

[0158] Figure 40 illustrates an embodiment of the light source 3800, which similarly has emitters 3802, 3804, and 3806 that provide light to the collection region 3808 at the same or substantially the same angle. However, the light incident on the collection region 3808 is offset from the vertical. Angle 4002 indicates the angular offset from the vertical. In one embodiment, the laser emitters 3802, 3804, and 3806 may have a cross-sectional intensity profile that is Gaussian. As discussed previously, an improved distribution of optical energy between fibers can be achieved by generating a flatter or top-hat shaped intensity profile. In one embodiment, as angle 4002 increases, the intensity across the collection region 3808 approaches a top-hat profile. For example, even a non-flat output beam can be approximated to a top-hat profile by increasing angle 4002 until the profile is sufficiently flat.

[0159] The top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intervening optical components between the emitters 3802, 3804, and 3806 and the output waveguide or fiber optic bundle.

[0160] Figure 41 is a schematic diagram illustrating a single optical fiber 4102 that outputs via a diffuser 4104 at the output. In one embodiment, the optical fiber 4102 may have a diameter of 500 microns and an numerical aperture of 0.65, and without the diffuser 4104, it emits an optical cone 4106 of about 70 or 80 degrees. With the diffuser 4104, the optical cone 4106 may have an angle of about 110 or 120 degrees. The optical cone 4106 may be the majority of the area where all light travels and is uniformly distributed. The diffuser 4104 can enable a more uniform distribution of electromagnetic energy in the scene observed by the image sensor.

[0161] In one embodiment, the lumen waveguide 4102 may include a single plastic or glass optical fiber approximately 500 microns wide. While plastic fibers may be less expensive, their width can, due to couplers, diffusers, or other losses, allow the fiber to carry a sufficient amount of light into the scene. For example, smaller fibers may not be able to carry as much light or power as larger fibers. The lumen waveguide 3710 may include one or more optical fibers. The lumen waveguide 3702 can receive light directly from a light source or via a jumper waveguide (see, for example, jumper waveguide 3706 in Figure 37). Diffusers can be used to widen the optical output 3706 for a desired field of view of an image sensor 3714 or other optical component.

[0162] Figures 38 to 40 show three emitters, but in some embodiments, one to hundreds or more emitters can be used. The emitters may have light of different wavelengths or spectra that they emit, and this light can be used to continuously cover a desired portion of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra).

[0163] In one embodiment, a light source having multiple emitters can be used for multispectral or hyperspectral imaging in light-deficient environments. For example, different chemicals, materials, or tissues may have different responses to electromagnetic energy of different colors or wavelengths. Some tissues have their own spectral signature (how the tissue responds to or varies in the reflected wavelengths of electromagnetic radiation). In one embodiment, certain types of tissue can be detected based on how the tissue responds to certain wavelengths or specific combinations of wavelengths. For example, vascular tissue can absorb and reflect electromagnetic energy of different wavelengths or spectra in a unique way that distinguishes it from muscle, fat, bone, nerves, ureters, or other tissues or materials in the body. Furthermore, certain types of muscle or other types of tissue can be distinguished based on their spectral responses. The disease state of tissue can also be determined based on spectral information. See U.S. Patent No. 8,289,503 and U.S. Patent No. 8,158,957.

[0164] In one embodiment, multispectral or hyperspectral image data can be acquired by filtering out all light or electromagnetic energy using one or more filters, except for those at a desired wavelength or spectrum. Figure 42 is a block diagram illustrating a filter 4202 for removing unwanted wavelengths before light 4208 (or other electromagnetic radiation) encounters the image sensor 4204 or another imaging medium (e.g., film). In one embodiment, white light 4208 passes through the filter 4202, and the filtered light 4210 passes through a lens 4206 that focuses it onto the image sensor 4204 for image acquisition and readout. The filters can be located anywhere in the system or can be accessories to the lens 4206 or the image sensor 4204.

[0165] In a light-deficient environment, light 4208 may include white light emitted by the emitter in a light-deficient environment. Filter 4202 can be selected for the desired inspection. For example, if it is desired to detect or highlight a specific tissue, filter 4202 may be selected to allow wavelengths corresponding to the spectral response of the specific tissue to pass through. Image sensor 4204, which may include a monochrome image sensor, can generate an image. Pixels of the captured image that are above or below a threshold can then be characterized as corresponding to a specific tissue. This data can then be used to generate an image showing the location of the specific tissue.

[0166] In another embodiment, fluorescent dyes can be used to image specific types of tissue, pathways, etc., within the body. For example, a fluorescent dye can be administered to a patient, and then an image of the dye can be captured. In one embodiment, the fluorescence of the dye can be triggered using electromagnetic energy of a specific wavelength. For example, the dye may only fluoresce when electromagnetic energy is present.

[0167] However, both filters and fluorescent dyes significantly limit the examination. For example, when using filters, the desired spectral response that can be detected, and therefore the materials or tissues that can be detected, is limited by the available filters. Furthermore, filters may need to be replaced or exchanged. Regarding dyes, they must be administered before imaging, and there may be competition between the administration of different dyes for different purposes during the same examination. Therefore, examinations using filters and dyes can be time-consuming and may require many different examinations to obtain the desired information.

[0168] In one embodiment, multispectral or hyperspectral imaging in a light-deficient environment can be achieved using a monochrome image sensor and emitter that emit electromagnetic energy of multiple different wavelengths or spectra. In one embodiment, a light source or other electromagnetic source (such as light source 3800 in any of Figures 38 to 40) may include multiple emitters to cover the desired spectrum.

[0169] Figure 43 illustrates a portion of the electromagnetic spectrum 4300 divided into 20 different subspectrals. The number of subspectrals is illustrative only. In at least one embodiment, the spectrum 4300 can be divided into hundreds of subspectrals, each having a small wavelength band. The spectrum may extend from the infrared spectrum 4302 through the visible spectrum 4304 to the ultraviolet spectrum 4306. Each subspectral has a wavelength band 4308 that covers a portion of the spectrum 4300. Each wavelength band can be defined by an upper wavelength and a lower wavelength.

[0170] In one embodiment, to provide complete and continuous coverage of the entire spectrum 4300, at least one emitter (such as a laser emitter) may be included within a light source (such as light sources 3702, 3800 in Figures 37-40) for each subspectral. For example, a light source for providing the illustrated subspectral coverage may include at least 20 different emitters, at least one for each subspectral. In one embodiment, each emitter may cover a spectrum covering 40 nanometers. For example, one emitter may emit light in the 500 nm to 540 nm wavelength band, while another emitter may emit light in the 540 nm to 580 nm wavelength band. In another embodiment, emitters may cover wavelength bands of other sizes depending on the type of emitter available or the imaging requirements. For example, multiple emitters may include a first emitter covering the 500-540 nm wavelength band, a second emitter covering the 540-640 nm wavelength band, and a third emitter covering the 640-650 nm wavelength band. Each emitter may cover different slices of the electromagnetic spectrum across far-infrared, mid-infrared, near-infrared, visible light, near-ultraviolet, and / or extreme ultraviolet. In some cases, multiple emitters of the same type or wavelength may be included to provide sufficient output for imaging. The number of emitters required for a particular wavelength band may depend on the sensitivity of the monochromator to that wavelength band and / or the power output capability of the emitters in that wavelength band.

[0171] The wavelength bandwidth and coverage provided by the emitter can be selected to provide any desired combination of spectra. For example, continuous spectral coverage using a very narrow wavelength bandwidth (e.g., less than 10 nm) can enable highly selective hyperspectral imaging. Since the wavelengths originate from an emitter that can operate selectively, maximum flexibility can be achieved when determining the spectral response of a material during inspection. Thus, further information about the spectral response, which would otherwise require multiple inspections or be delayed by the administration of dyes or stains, can be obtained in less time and within a single inspection. In one embodiment, the system can capture hyperspectral image data and process the data to identify what kind of tissue is present in each pixel.

[0172] Figure 44 is a schematic diagram illustrating a timing diagram 4400 for emission and readout to generate a multispectral or hyperspectral image according to one embodiment. The solid lines represent readout (peak 4402) and blanking periods (trough) for capturing a series of frames 4404-4414. The series of frames 4404-4414 may include a repeating series of frames that can be used to generate hyperspectral data for a video feed. The series of frames includes a first frame 404, a second frame 4406, a third frame 4408, a fourth frame 4410, a fifth frame 4412, and an nth frame 4426.

[0173] In one embodiment, each frame is generated based on at least one pulse of electromagnetic energy. The pulse of electromagnetic energy is reflected, detected by an image sensor, and then read out in a subsequent readout (4402). Thus, each blanking period and readout results in an image frame of a specific range of electromagnetic energy. For example, a first frame 404 can be generated based on the spectrum of one or more first pulses 4416, a second frame 4406 can be generated based on the spectrum of one or more second pulses 4418, a third frame 4408 can be generated based on the spectrum of one or more third pulses 4420, a fourth frame 4410 can be generated based on the spectrum of one or more fourth pulses 4422, a fifth frame 4412 can be generated based on the spectrum of one or more fifth pulses 4424, and an nth frame 4426 can be generated based on the spectrum of one or more nth pulses 4426.

[0174] Pulses 4416-4426 may include energy from a single emitter or from a combination of two or more emitters. For example, spectra contained within a single readout period or within multiple frames 4404-4414 can be selected for desired inspection or detection of a particular tissue or condition. According to one embodiment, one or more pulses may include visible spectral light to generate a color or black and white image, while one or more additional pulses may be used to obtain spectral responses and classify tissue types. For example, pulse 4416 may include red light, pulse 4418 may include blue light, pulse 4420 may include green light, while the remaining pulses 4422-4426 may include wavelengths and spectra for detecting a particular tissue type. As a further example, pulses within a single readout period may include spectra generated from multiple different emitters (e.g., electromagnetic spectra of different slices) that can be used to detect a particular tissue type. For example, if a combination of wavelengths results in a pixel having a value above or below a threshold, that pixel can be classified as corresponding to a particular type of tissue. Each frame can be used to further narrow down the type of tissue present in its pixel (e.g., each pixel in the image) and, based on its spectral response, provide a very specific classification of the tissue and / or the state of the tissue (pathological / healthy).

[0175] Multiple frames 4404-4414 are shown to have readout periods of varying length and pulses of different lengths or intensities. The blanking period, pulse length or intensity, etc., can be selected based on the sensitivity of the monochromator to a particular wavelength, the power output capability of the emitter(s), and / or the carrier capability of the waveguide.

[0176] Hyperspectral images or hyperspectral image data acquired in the manner illustrated in Figure 44 can yield multiple frames, each based on a different spectrum or combination of spectra. In some cases, tens or hundreds of different frames can be acquired. In other cases, such as for video streams, some frames may be limited to provide a viewable frame rate. Because different spectral combinations can be provided within a single readout period, useful and dynamic spectral information can still be obtained, even for video streams.

[0177] In one embodiment, the video or other image may include a black and white or color image superimposed with information derived from the spectral response of each pixel. For example, during an examination, pixels corresponding to a specific tissue or condition can be shown in light green or another color to assist a physician or other medical professional.

[0178] In one embodiment, dual image sensors can be used to acquire three-dimensional images or video feeds. Three-dimensional examination can enable an improved understanding of the three-dimensional structure of the area being examined, as well as the mapping of different types of tissues or materials within the area.

[0179] In one embodiment, multispectral or hyperspectral imaging can be used to see through materials or substances. For example, infrared wavelengths can pass through some tissues, such as muscle or fat, while being reflected by blood vessels. In one embodiment, infrared waves can penetrate 5, 8, or 10 mm or more into the tissue. By acquiring a series of frames including at least one infrared frame, the examination can be made possible to provide information about the location of blood vessels beneath the surface. This can be extremely useful in surgical procedures where it may be desirable to perform incisions that avoid blood vessels. In one embodiment, a color or grayscale image can be superimposed on green to indicate the location of blood vessels beneath the surface. Similarly, known spectral responses of blood can be used to see through the blood of a target tissue or structure during examination.

[0180] The assembly of subframes into a single frame for display on a monitor or other display device can be performed after capturing a series of frames 4404-4414. A color or grayscale image can be generated from one or more frames, and pixel overlay information can be determined based on all or the remaining frames. The color or grayscale image, combined with the overlay information, can generate a single frame. A single frame can be displayed as a single image or as an image within a video stream.

[0181] In one embodiment, the acquired hyperspectral data, as illustrated in Figure 44, can be provided to a third-party algorithm for analysis to classify tissues or materials captured in the image. In one embodiment, the third-party algorithm can be used to select the spectrum or wavelength band to be used during imaging so that a desired spectral response analysis can be performed. In one embodiment, the spectral response analysis can be performed in real time during a medical imaging procedure or other medical treatment. The spectral data can be overlaid on an RGB or black and white image so that a user can easily distinguish specific types of tissues, organs, chemical processes, diseases, etc. In one embodiment, the spectral data can be provided to a computer-operated system, such as a robotic system for automating medical imaging or medical treatment.

[0182] Figure 45 is a schematic diagram of an imaging system 4500 having a single-cut filter. The system 4500 includes an endoscope 4506 or other suitable imaging device having a light source 4508 for use in light-deficient environments. The endoscope 4506 includes an image sensor 4504 and a filter 4502 for filtering out unwanted wavelengths of light or other electromagnetic radiation before it reaches the image sensor 4504. The light source 4508 transmits light that can illuminate a surface 4512 in a light-deficient environment, such as a body cavity. The light 4510 is reflected off the surface 4512 and passes through the filter 4502 before hitting the image sensor 4504.

[0183] The filter 4502 can be used in a single implementation configuration in which a fluorescent reagent or dye is administered. In one such embodiment, the filter 4502 is configured to filter all light other than light of one or more desired wavelengths or spectral bands, or other electromagnetic radiation. In one embodiment, the filter 4502 is configured to filter electromagnetic radiation at the excitation wavelength that causes the reagent or dye to fluoresce, so as to allow only the expected relaxation wavelength of the fluorescent reagent or dye to pass through the filter 4502 and reach the image sensor 4504. In one embodiment, the filter 4502 filters the fluorescent reagent excitation wavelength at least from 770 nm to 790 nm. In one embodiment, the filter 4502 filters the fluorescent reagent excitation wavelength at least from 795 nm to 815 nm. In one embodiment, the filter 4502 filters the fluorescent reagent excitation wavelengths at least from 770 nm to 790 nm and from 795 nm to 815 nm. In these embodiments, the filter 4502 filters out the excitation wavelength of the reagent, allowing only the relaxation wavelength of the fluorescent reagent to be read by the image sensor 4504. The image sensor 4504 can be a wavelength-insensitive image sensor, and the filter 4502 can be configured to allow the image sensor 4504 to receive only the relaxation wavelength of the fluorescent reagent and not the emission excitation wavelength of the reagent. The data determined by the image sensor 4504 can then indicate the presence of important body structures, tissues, biological processes, or chemical processes determined by the location of the reagent or dye.

[0184] Filter 4502 can be further used in one implementation configuration in which no fluorescent reagent or dye is administered. Filter 4502 can be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read out by the image sensor 4504. The image sensor 4504 can be a monochrome image sensor so that pixels of the captured image above or below a threshold can be characterized to correspond to a specific spectral response or fluorescence emission. The spectral response or fluorescence emission determined by the pixels captured by the image sensor 4504 can indicate the presence of a specific body tissue or structure, a specific state, a specific chemical process, etc.

[0185] In one embodiment, the light source 4508 transmits white light that contacts the surface 4512, and the white light is reflected back to a location filtered by the filter 4502 before it hits the image sensor 4504. In one embodiment, the light source 4508 transmits white light such that filtered light of only one or more desired wavelengths passes through the filter 4502, emerges from the filter 4502, is reflected back to the surface 4512, and is read out by the image sensor 4504. For example, in one embodiment, the filter 4502 allows only light having a wavelength of 795 nm to pass through the filter 4502 and contact the image sensor 4504. In a further embodiment, the filter 4502 allows only light of a specific wavelength to be reflected back to the image sensor 4504 of the endoscope 4506 or other imaging device. The filter 4502 can be located anywhere in the system 4500 or can be an accessory to a lens or the image sensor 4504. The filter 4502 can be positioned in front of and / or behind the image sensor 4504. In one embodiment, light emitted by the light source 4508 is filtered before it reaches the surface 4512, and reflected light is filtered by an additional filter before it becomes available to the image sensor 4504.

[0186] The light source 4508 can be an emitter that can be configured to emit white light or electromagnetic radiation of one or more specific wavelengths. The light source 4508 may include a plurality of lasers configured to emit or pulse light of specified wavelengths. In one embodiment, the light source 4508 emits white light, and the filter 4502 is configured to filter out all unwanted light or other electromagnetic radiation other than light of one or more desired wavelengths. The filter 4502 may be selected for a specific inspection or purpose, for example, to highlight a certain type of body tissue or structure, or to highlight a specific state or chemical process.

[0187] Figure 46 is a schematic diagram of the 4600 imaging system having multiple cut filters. System 4600 includes an endoscope 4606 or other suitable imaging device having a light source 4608 for use in light-deficient environments. The endoscope 4606 includes an image sensor 4604 and two filters 4602a and 4602b. In alternative embodiments, it should be understood that system 4600 may include any number of filters, and that the number and type of filters may be selected for a particular purpose, for example, to collect imaging information such as specific body tissues, physical conditions, chemical processes, etc. Filters 4602a and 4602b are configured to filter unwanted wavelengths of light or other electromagnetic radiation. Filters 4602a and 4602b may be configured to filter unwanted wavelengths from white light or other electromagnetic radiation that can be emitted by the light source 4608. The filtered light can strike a surface 4612 (e.g., body tissue) and be reflected back to the image sensor 4604.

[0188] Continuing the disclosure with respect to Figure 45, filters 4602a and 4602b can be used in one implementation configuration with a fluorescent reagent or dye applied. Filters 4602a and 4602b can be configured to block the emission excitation wavelength of the reagent or dye, allowing the image sensor 4604 to read only the relaxation wavelength of the reagent or dye. Furthermore, filters 4602a and 4602b can be used in one implementation configuration without a fluorescent reagent or dye applied. In such a configuration, filters 4602a and 4602b can be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read by the image sensor 4604.

[0189] Multiple filters 4602a, 4602b can each be configured to filter different wavelength ranges of the electromagnetic spectrum. For example, one filter can be configured to filter wavelengths beyond a desired wavelength range, and additional filters can be configured to filter wavelengths shorter than the desired wavelength range. A combination of two or more filters can result in only specific wavelengths or bands of wavelengths read out by the image sensor 4604.

[0190] In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 513nm to 545nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 565nm to 585nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 900nm to 1000nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 425nm to 475nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 520nm to 545nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 625nm to 645nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 760nm to 795nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 795nm to 815nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 370nm to 420nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b can be customized so that electromagnetic radiation in the range of 600nm to 670nm contacts the image sensor 4604. In one embodiment, filters 4602a and 4602b are configured to allow only specific fluorescence relaxation emissions to pass through filters 4602a and 4602b and come into contact with the image sensor 4604.

[0191] In one embodiment, the system 4600 includes a plurality of image sensors 4604, and in particular, may include two image sensors for use in generating a three-dimensional image. The image sensor(s) 4604 may be color / wavelength-insensitive and may be configured to read out any wavelength of electromagnetic radiation reflected off the surface 4612. In one embodiment, each image sensor 4604 may be color-dependent or wavelength-dependent and may be configured to read out electromagnetic radiation of a specific wavelength that is reflected off the surface 4612 and returns to the image sensor 4604. Alternatively, the image sensor 4604 may include a single image sensor having a plurality of different pixel sensors configured to read out different wavelengths or colors of light, such as a Bayer filter color filter array. Alternatively, the image sensor 4604 may include one or more color-insensitive image sensors that can be configured to read out different wavelengths of electromagnetic radiation according to a pulsed schedule, such as those illustrated in Figures 5-7E and 15-16.

[0192] Figure 47 is a schematic diagram illustrating a system 4700 for mapping a surface and / or tracking an object in a light-deficient environment. In one embodiment, an endoscope 4702 in a light-deficient environment pulses a grid array 4706 (which may also be referred to as a laser map pattern) of a surface 4704. In one embodiment illustrated in Figure 47, the grid array 4706 may include vertical hashing 4708 and horizontal hashing 4710. It should be understood that the grid array 4706 may include any suitable array for mapping the surface 4704, including, for example, a raster grid of individual locations, an occupied grid map, a dot array, etc. In addition, the endoscope 4702 can pulse multiple grid arrays 4706, and can also pulse one or more individual grid arrays of each of multiple objects or structures in a light-deficient environment, for example.

[0193] In one embodiment, the system 4700 can pulse the grid array 4706, which can be used to determine a three-dimensional surface and / or to track the position of an object such as a tool or another device in a light-deficient environment. In one embodiment, the system 4700 can provide data to a third-party system or computer algorithm for determining the dimensions and configuration of a surface by light detection and ranging (LIDAR) mapping. The system 4700 can pulse any suitable wavelength of light or electromagnetic radiation, including, for example, ultraviolet light, visible light, and / or infrared light or near-infrared light, into the grid array 4706. Surfaces 4704 and / or objects in the environment can be mapped and tracked with very high resolution and with very high accuracy and precision.

[0194] In one embodiment, system 4700 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to one or more image sensors. System 4700 may include a light engine having an illumination source that generates one or more pulses of electromagnetic radiation, and a lumen that transmits one or more pulses of electromagnetic radiation to the distal tip of the endoscope in a light-deficient environment such as a body cavity. In one embodiment, at least a portion of one or more pulses of electromagnetic radiation includes a laser map pattern emitted to a surface in a light-deficient environment, such as the surface of body tissue and / or the surface of a tool or other device in a body cavity. The endoscope 4702 may include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations in a light-deficient environment.

[0195] In one embodiment, system 4700 includes a processor for determining the distance from an object, such as a surface 4704, to an endoscope or tool. The processor can further determine the angle between the endoscope or tool and the object. The processor can further determine surface area information about the object, including, for example, the size of the surgical instrument, the size of the structure, the size of the anatomical structure, positional information, and other positional data and metrics. System 4700 may include one or more image sensors that provide image data, which is an output to a control system for determining the distance from the endoscope or tool to an object, such as a surface 4704. The image sensors can output information to the control system for determining the angle between the endoscope or tool and the object. In addition, the image sensors can output information to the control system for determining surface area information about the object, the size of the surgical instrument, the size of the structure, the size of the anatomical structure, positional information, and other positional data and metrics.

[0196] In one embodiment, the grid array 4706 is pulsed at a sufficiently fast rate by the illumination source of the endoscope 4702 so that the grid array 4706 cannot be seen by the user. In various implementations, the user may be distracted by seeing the grid array 4706 during endoscopic imaging and / or endoscopic surgical procedures. The grid array 4706 can be pulsed for a sufficiently short period of time so that the grid array 4706 cannot be detected by the human eye. In an alternative embodiment, the endoscope 4702 pulses the grid array 4706 at a sufficiently high repetition rate so that the grid array 4706 can be seen by the user. In such an embodiment, the grid array 4706 can be superimposed on an image of the surface 4704 on a display. The grid array 4706 can be superimposed on a grayscale or RGB image of the surface 4704 so that the user can see the grid array 4706 while using the system 4700. The user of system 4700 can instruct whether the grid array 4706 should be superimposed on the image of surface 4704 and / or whether the user should be able to see the grid array 4706. System 4700 may include a display that provides real-time measurements of the distance from the endoscope 4702 to surface 4704 or another object in a low-light environment. The display may further provide real-time surface area information for surface 4704 and / or any object, structure, or tool in a low-light environment. The accuracy of the measurements can be less than 1 millimeter.

[0197] The endoscope 4702 can pulse electromagnetic radiation according to a pulse schedule, such as those illustrated in Figures 5-7E and 15-16, which may further include pulsing a grid array 4706 along with pulsing red, green, and blue light to generate an RGB image, and further generating a grid array 4706 that can be superimposed on the RGB image and / or used to map and track surfaces 4704 and objects in light-deficient environments.

[0198] In one embodiment, the endoscope 4702 includes one or more color-insensitive image sensors. In one embodiment, the endoscope 4702 includes two color-insensitive image sensors for generating a three-dimensional image or map of a light-deficient environment. The image sensors can generate an RGB image of the light-deficient environment according to a pulsation schedule disclosed herein. In addition, the image sensors can map the light-deficient environment and determine data for tracking one or more objects within the light-deficient environment, based on data determined when the grid array 4706 is pulsed. In addition, the image sensors can determine spectral data or hyperspectral data along with fluorescence imaging data according to a pulsation schedule that can be modified by the user to meet specific needs of the imaging procedure. In one embodiment, the pulsation schedule includes red pulses, green pulses, and blue pulses, along with pulsation of the grid array 4706 and / or pulsation for generating hyperspectral image data and / or fluorescence image data. In various implementations, the pulsation schedule may include any suitable combination of pulses of electromagnetic radiation according to the user's needs. The repetition frequency of electromagnetic radiation of different wavelengths can be determined based on factors such as the energy of a particular pulse, user needs, and whether it is necessary to continuously update certain data (e.g., hyperspectral data and / or fluorescence imaging data) or not update them very frequently.

[0199] The pulsation schedule can be modified in any preferred manner, and specific pulses of electromagnetic radiation can be repeated at any preferred frequency according to user needs or the program of the computer implementation for a particular imaging procedure. For example, in one embodiment where surface tracking data generated based on the grid array 4706 is provided to a program of a computer implementation for use, for example, in a robotic surgical procedure, the grid array 4706 can be pulsed more frequently than when the surface tracking data is provided to a user visualizing the scene during the imaging procedure. In such an embodiment where the surface tracking data is used in a robotic surgical procedure, the surface tracking data may need to be updated more frequently or may need to be extremely accurate so that the program of the computer implementation can perform the robotic surgical procedure precisely and accurately.

[0200] In one embodiment, the system 4700 is configured to generate an occupied grid map comprising an array of cells divided into a grid. The system 4700 is configured to store the respective height values ​​of each grid cell to determine the surface mapping of a three-dimensional environment in a light-deficient environment.

[0201] Figure 48 is a schematic flowchart of Method 4800 for hyperspectral imaging in light-deficient environments. Method 4800 can be performed by an imaging system such as the endoscopic imaging system illustrated in Figure 37.

[0202] Method 4800 includes, in 4802, emitting a plurality of narrowband pulses during the readout period of a monochrome image sensor. In 4802, the pulses can be emitted using a light source that includes a plurality of emitters that emit electromagnetic energy within a narrow frequency band. For example, the light source may include at least one emitter for a plurality of frequency bands covering a desired spectrum. In 4804, the monochrome image sensor reads pixel data from the monochrome image sensor and generates a plurality of frames after the readout period. Each frame may include different spectral components. These frames may include a plurality of repeating frames that can be used to generate a digital video stream. Each frame may be based on the energy emitted by one or more emitters of the light source. In one embodiment, the frames may be based on a combination of light emitted by the light source to generate a combination of frequencies that match the frequency response of a desired tissue or material. In 4806, a controller, CCU, or other system determines the spectral response of a tissue for one or more pixels based on the plurality of frames. For example, the frequency response of a particular pixel can be determined using pixel values ​​and knowledge regarding the frequencies of light emitted for each frame, based on the pixel values ​​in the plurality of frames. In the 4808, the system can generate a composite image based on multiple frames, which includes an overlay showing the spectral response of one or more pixels. For example, the composite image may be grayscale or a color image in which pixels corresponding to a specific tissue or classification are shown in light green.

[0203] Figure 49 is a schematic flowchart of Method 4900 for fluorescence imaging in light-deficient environments. Method 4900 can be performed by an imaging system such as the endoscopic imaging system illustrated in Figure 37.

[0204] Method 4900 includes, in 4902, emitting a plurality of narrowband pulses during the readout period of a monochrome image sensor. In 4902, the pulses can be emitted using a light source that includes a plurality of emitters that emit electromagnetic energy within a narrow frequency band. For example, the light source may include at least one emitter for a plurality of frequency bands covering a desired spectrum. In 4904, the monochrome image sensor reads pixel data from the monochrome image sensor and generates a plurality of frames after the readout period. Each frame may include different spectral components. These frames may include a plurality of repeating frames that can be used to generate a digital video stream. Each frame may be based on the energy emitted by one or more emitters of the light source. In one embodiment, the frames may be based on a combination of light emitted by the light source to generate a combination of frequencies that match the frequency response of a desired tissue or substance. In 4906, a controller, CCU, or other system determines the fluorescence relaxation emission of a reagent for one or more pixels based on the plurality of frames. For example, the frequency response of a particular pixel can be determined using pixel values ​​and knowledge regarding the frequency of light emitted for each frame, based on the pixel values ​​in the plurality of frames. In 4908, the system can generate a composite image based on multiple frames, which includes an overlay showing the fluorescence relaxation emission of one or more pixels. For example, the composite image may be grayscale or a color image in which pixels corresponding to a specific tissue or classification are shown in light green. [Examples]

[0205] The following examples relate to further embodiments.

[0206] Example 1 is an endoscope system for use in light-deficient environments. The system includes a tube, one or more image sensors, and a lens assembly having at least one optical element corresponding to one or more image sensors. The system includes a display for the user to visualize the scene. The system includes a light engine which includes an illumination source that generates one or more pulses of electromagnetic radiation. The light engine may further include a lumen which transmits one or more pulses of electromagnetic radiation to the distal tip of the endoscope, wherein at least a portion of one or more pulses of electromagnetic radiation comprises electromagnetic radiation in the 513 nm to 545 nm range, at least a portion of one or more pulses of electromagnetic radiation comprises electromagnetic radiation in the 565 nm to 585 nm range, and at least a portion of one or more pulses of electromagnetic radiation comprises electromagnetic radiation in the 900 nm to 1000 nm range.

[0207] Example 2 is the same system as in Example 1, wherein the duration of one or more pulses of electromagnetic radiation is variable.

[0208] Example 3 is a system according to any one of Examples 1 to 2, comprising multiple pulses of electromagnetic radiation, the pulse duration having a duration shorter than the total pulse duration.

[0209] Example 4 is a system according to any one of Examples 1 to 3, wherein multiple pulses of electromagnetic radiation having a duration shorter than the total pulse duration have a duration that provides a single frame exposure.

[0210] Example 5 is a system according to any one of Examples 1 to 4, wherein multiple subpulses of electromagnetic radiation having sub-durations are emitted during the pulse duration, and the sub-durations are shorter than the pulse duration.

[0211] Example 6 is a system described in any of Examples 1 to 5, wherein the pulse sub-duration within the pulse duration differs for each sub-pulse of electromagnetic radiation.

[0212] Example 7 is a system according to any of Examples 1 to 6, wherein the pulse sub-duration within the pulse duration is the same for each sub-pulse of electromagnetic radiation.

[0213] Example 8 is a system according to any one of Examples 1 to 7, wherein the pulse sub-durations within the pulse duration include two or more sub-pulses of electromagnetic radiation having equal durations.

[0214] Example 9 is a system according to any of Examples 1 to 8, wherein one or more of the subpulses may include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single subpulse.

[0215] Example 10 is a system according to any one of Examples 1 to 9, wherein the subdurations of multiple subpulses are variable in both the wavelength of the emitted electromagnetic radiation and the number of emissions per pulse duration.

[0216] Example 11 is a system according to any of Examples 1 to 10, wherein the subdurations of multiple subpulses are equal to the duration of a single pulse that provides a single frame exposure.

[0217] Example 12 is a system according to any of Examples 1 to 11, wherein a series of pulses of various wavelengths of electromagnetic radiation can be adjusted by the user of the endoscope light engine.

[0218] Example 13 is a system according to any of Examples 1 to 12, wherein the system includes a toggle button, and the user changes or adjusts the order of pulses by operating the toggle button.

[0219] Example 14 is a system according to any of Examples 1 to 13, wherein the system comprises a control system for properly illuminating a scene, the control system automatically adjusts or changes the sequence of pulses of electromagnetic radiation emitted from the light engine based on a threshold, and the threshold deductively determines the proper illumination of the scene.

[0220] Example 15 is a system according to any of Examples 1 to 14, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on a display.

[0221] Example 16 is a system according to any of Examples 1 to 15, wherein a single image is assigned a visible color for use on a display, and the visible color is 8-bit, 16-bit, or n-bit.

[0222] Example 17 is a system according to any of Examples 1 to 16, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on a display.

[0223] Example 18 is a system according to any of Examples 1 to 17, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit.

[0224] Example 19 is a system according to any of Examples 1 to 18, wherein the illumination source generates one or more pulses of electromagnetic radiation using one or more laser emitters.

[0225] Example 20 is a system according to any of Examples 1 to 19, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on the display.

[0226] Example 21 is a system according to any of Examples 1 to 20, wherein a single image is assigned a visible color for use on a display, and the visible color is 8-bit, 16-bit, or n-bit.

[0227] Example 22 is the system according to any one of Examples 1 to 21, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on the display.

[0228] Example 23 is the system according to any one of Examples 1 to 22, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit or 16-bit or n-bit.

[0229] Example 24 is the system according to any one of Examples 1 to 23, wherein the illumination source generates one or more pulses of electromagnetic radiation at a wavelength of 425 nm to 475 nm.

[0230] Example 25 is the system according to any one of Examples 1 to 24, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on the display.

[0231] Example 26 is the system according to any one of Examples 1 to 25, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit or 16-bit or n-bit.

[0232] Example 27 is the system according to any one of Examples 1 to 26, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on the display.

[0233] Example 28 is the system according to any one of Examples 1 to 27, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit or 16-bit or n-bit.

[0234] Example 29 is the system according to any one of Examples 1 to 28, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 520 nm to 545 nm.

[0235] Example 30 is the system according to any one of Examples 1 to 29, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on a display.

[0236] Example 31 is the system according to any one of Examples 1 to 30, wherein a visible color for use on a display is assigned to the single image, and the visible color is 8-bit or 16-bit or n-bit.

[0237] Example 32 is the system according to any one of Examples 1 to 31, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on a display.

[0238] Example 33 is the system according to any one of Examples 1 to 32, wherein a visible color for use on a display is assigned to the overlay image, and the visible color is 8-bit or 16-bit or n-bit.

[0239] Example 34 is the system according to any one of Examples 1 to 33, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 625 nm to 645 nm.

[0240] Example 35 is the system according to any one of Examples 1 to 34, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on a display.

[0241] Example 36 is a system according to any of Examples 1 to 35, wherein a single image is assigned a visible color for use on a display, and the visible color is 8-bit, 16-bit, or n-bit.

[0242] Example 37 is a system according to any of Examples 1 to 36, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on a display.

[0243] Example 38 is a system according to any of Examples 1 to 37, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit.

[0244] Example 39 is a system according to any of Examples 1 to 38, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 760 nm to 795 nm.

[0245] Example 40 is a system according to any of Examples 1 to 39, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on a display.

[0246] Example 41 is a system according to any of Examples 1 to 40, wherein a single image is assigned a visible color for use on a display, and the visible color is 8-bit, 16-bit, or n-bit.

[0247] Example 42 is a system according to any of Examples 1 to 41, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on a display.

[0248] Example 43 is a system according to any of Examples 1 to 42, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit.

[0249] Example 44 is a system according to any of Examples 1 to 43, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 795 nm to 815 nm.

[0250] Example 45 is a system according to any of Examples 1 to 44, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on a display.

[0251] Example 46 is a system according to any of Examples 1 to 45, wherein a single image is assigned a visible color for use on a display, and the visible color is 8-bit, 16-bit, or n-bit.

[0252] Example 47 is a system according to any of Examples 1 to 46, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on a display.

[0253] Example 48 is a system according to any of Examples 1 to 47, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit.

[0254] Example 49 is a system according to any of Examples 1 to 48, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 370 nm to 420 nm.

[0255] Example 50 is the system according to any one of Examples 1 to 49, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on a display.

[0256] Example 51 is the system according to any one of Examples 1 to 50, wherein a visible color for use on a display is assigned to the single image, and the visible color is 8 bits or 16 bits or n bits.

[0257] Example 52 is the system according to any one of Examples 1 to 51, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on a display.

[0258] Example 53 is the system according to any one of Examples 1 to 52, wherein a visible color for use on a display is assigned to the overlay image, and the visible color is 8 bits or 16 bits or n bits.

[0259] Example 54 is the system according to any one of Examples 1 to 53, wherein the illumination source generates one or more pulses of electromagnetic radiation at a wavelength of 600 nm to 670 nm.

[0260] Example 55 is the system according to any one of Examples 1 to 54, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as a single image on a display.

[0261] Example 56 is the system according to any one of Examples 1 to 55, wherein a visible color for use on a display is assigned to the single image, and the visible color is 8 bits or 16 bits or n bits.

[0262] Example 57 is a system according to any of Examples 1 to 56, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to the user as an overlay image on a display.

[0263] Example 58 is a system according to any of Examples 1 to 57, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit.

[0264] Example 59 is a system according to any of Examples 1 to 58, wherein the light engine includes a polarizing filter.

[0265] Example 60 is a system according to any of Examples 1 to 59, wherein the polarizing filter is placed in the path of electromagnetic radiation.

[0266] Example 61 is a system according to any of Examples 1 to 60, wherein the polarizing filter is positioned at the proximal end of the lumen.

[0267] Example 62 is a system according to any of Examples 1 to 61, wherein the polarizing filter is placed at the distal end of the lumen.

[0268] Example 63 is a system according to any of Examples 1 to 62, wherein the lens assembly comprises an electromagnetic radiation filter.

[0269] Example 64 is a system according to any of Examples 1 to 63, wherein the lens assembly comprises a polarizing filter.

[0270] Example 65 is a system according to any of Examples 1 to 64, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are supplied to a corresponding system that provides the location of important tissue structures.

[0271] Example 66 is a system according to any of Examples 1 to 65, wherein the location of important structures is received by an endoscope system, superimposed on a display, and the important structures are encoded in any color selected by either an algorithm or a user.

[0272] It will be understood that the various features described herein represent significant advantages and advancements in the art. The following claims exemplify some of these features.

[0273] In the aforementioned "Modes for Carrying Out the Invention," various features of the Disclosure are grouped together into a single embodiment for the purpose of streamlining the Disclosure. The method of the Disclosure shall not be construed as reflecting an intention that the subject matter to be claimed requires more features than those explicitly described in each claim. Rather, the aspects of the Invention consist of fewer features than all the features of the single embodiment of the Disclosure described above.

[0274] It should be understood that the above-described configurations are merely illustrative of the application of the principles of this disclosure. Numerous variations and alternative configurations can be devised by those skilled in the art without departing from the spirit and scope of this disclosure, and the appended disclosure is intended to encompass such variations and configurations.

[0275] Therefore, although this disclosure has been described above with specific and detailed illustrations, it will be apparent to those skilled in the art that numerous variations, including but not limited to variations in size, material, shape, form, function and operation, assembly and use, can be made without departing from the principles and concepts described herein.

[0276] Furthermore, where appropriate, the functions described herein can be performed by one or more hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) can be programmed to perform one or more of the systems and procedures described herein. Certain terms are used below and throughout this disclosure to refer to specific system components. Components may be referred to by different names as will be understood by those skilled in the art. This specification does not intend to distinguish between components that have different names but the same function.

[0277] The foregoing descriptions are provided for illustrative and explanatory purposes only. They are not intended to be exhaustive or to limit the invention to the exact forms disclosed. Many variations and modifications are possible in light of the above teachings. Furthermore, it should be noted that any or all of the alternative embodiments described above may be used in any combination desired to form further hybrid embodiments of the disclosure.

[0278] Furthermore, while specific embodiments of this disclosure have been described and illustrated, this disclosure should not be limited to the specific forms and configurations of the portions described and illustrated in this manner. The scope of this disclosure should be defined by the claims appended to this specification, this specification and any future claims filed in other applications and their equivalents.

[0279] [Implementation Method] (1) An endoscope system for use in light-deficient environments, An imaging device, Tube and, One or more image sensors, An imaging device comprising: a lens assembly including at least one optical element corresponding to one or more image sensors; A display for the user to visualize the scene, Image signal processing controller, It is a light engine, A lighting source that generates one or more pulses of electromagnetic radiation, An endoscope system comprising: a light engine; a lumen for transmitting one or more pulses of electromagnetic radiation to the distal tip of an endoscope, wherein at least a portion of the one or more pulses of electromagnetic radiation includes electromagnetic radiation in the range of 513 nm to 545 nm, at least a portion of the one or more pulses of electromagnetic radiation includes electromagnetic radiation in the range of 565 nm to 585 nm, and at least a portion of the one or more pulses of electromagnetic radiation includes electromagnetic radiation in the range of 900 nm to 1000 nm. (2) The endoscope system according to Embodiment 1, wherein the duration of one or more pulses of electromagnetic radiation is variable. (3) The endoscope system according to Embodiment 2, comprising a plurality of pulses of electromagnetic radiation, the pulse duration having a duration shorter than the total pulse duration. (4) The endoscope system according to Embodiment 3, wherein the plurality of pulses of electromagnetic radiation having a duration shorter than the total pulse duration have a duration that provides a single frame exposure. (5) The endoscope system according to Embodiment 1, wherein during the pulse duration, a plurality of subpulses of electromagnetic radiation having sub-durations are emitted, the sub-durations of which are shorter than the pulse duration.

[0280] (6) The endoscope system according to Embodiment 5, wherein the pulse sub-duration within the pulse duration is different for each sub-pulse of electromagnetic radiation. (7) The endoscope system according to Embodiment 5, wherein the pulse sub-duration within the pulse duration is the same for each sub-pulse of electromagnetic radiation. (8) The endoscope system according to Embodiment 5, wherein the pulse subduration within the pulse duration includes two or more subpulses of electromagnetic radiation having equal durations. (9) The endoscope system according to Embodiment 5, wherein one or more of the plurality of subpulses may include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single subpulse. (10) The endoscope system according to Embodiment 5, wherein the subduration of the plurality of subpulses is variable in both the wavelength of the emitted electromagnetic radiation and the number of emissions per pulse duration.

[0281] (11) The endoscope system according to Embodiment 10, wherein the subdurations of the plurality of subpulses are equal to the duration of a single pulse that provides a single frame exposure. (12) The endoscope system according to Embodiment 1, wherein the sequence of pulses of various wavelengths of electromagnetic radiation can be adjusted by the user of the endoscope light engine. (13) The endoscope system according to Embodiment 1, wherein the system includes a toggle button, and a user changes or adjusts the order of pulses by operating the toggle button. (14) The endoscope system according to Embodiment 1, wherein the system comprises a control system for appropriately illuminating the scene, the control system automatically adjusts or changes the sequence of pulses of electromagnetic radiation emitted from the light engine based on a threshold, the threshold deductively determines appropriate illumination of the scene. (15) The endoscope system according to Embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0282] (16) The endoscope system according to Embodiment 15, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (17) The endoscope system according to Embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (18) The endoscope system according to Embodiment 17, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (19) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation using one or more laser emitters. (20) The endoscope system according to Embodiment 19, wherein each pulse of electromagnetic radiation results in an exposure frame created by the one or more image sensors, and the one or more exposure frames are displayed to the user as a single image on the display.

[0283] (21) The endoscope system according to Embodiment 20, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (22) The endoscope system according to Embodiment 19, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (23) The endoscope system according to Embodiment 22, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (24) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 425 nm to 475 nm. (25) The endoscope system according to Embodiment 24, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0284] (26) The endoscope system according to embodiment 25, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (27) The endoscope system according to Embodiment 24, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (28) The endoscope system according to Embodiment 27, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (29) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 520 nm to 545 nm. (30) The endoscope system according to Embodiment 29, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0285] (31) The endoscope system according to Embodiment 30, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (32) The endoscope system according to Embodiment 29, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (33) The endoscope system according to embodiment 32, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (34) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 625 nm to 645 nm. (35) The endoscope system according to embodiment 34, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0286] (36) The endoscope system according to embodiment 35, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (37) The endoscope system according to Embodiment 34, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (38) The endoscope system according to Embodiment 37, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (39) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 760 nm to 795 nm. (40) The endoscope system according to embodiment 39, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0287] (41) The endoscope system according to Embodiment 40, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (42) The endoscope system according to Embodiment 39, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (43) The endoscope system according to Embodiment 42, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (44) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 795 nm to 815 nm. (45) The endoscope system according to embodiment 44, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0288] (46) The endoscope system according to embodiment 45, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (47) The endoscope system according to Embodiment 44, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (48) The endoscope system according to Embodiment 47, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (49) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 370 nm to 420 nm. (50) The endoscope system according to Embodiment 49, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0289] (51) The endoscope system according to embodiment 50, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (52) The endoscope system according to Embodiment 49, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (53) The endoscope system according to embodiment 52, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (54) The endoscope system according to Embodiment 1, wherein the illumination source generates one or more pulses of electromagnetic radiation at wavelengths of 600 nm to 670 nm. (55) The endoscope system according to embodiment 54, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as a single image on the display.

[0290] (56) The endoscope system according to embodiment 55, wherein the single image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (57) The endoscope system according to embodiment 54, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are displayed to the user as an overlay image on the display. (58) The endoscope system according to embodiment 57, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8-bit, 16-bit, or n-bit. (59) The endoscope system according to Embodiment 1, wherein the light engine comprises a polarizing filter. (60) The endoscope system according to embodiment 59, wherein the polarizing filter is located within the path of the electromagnetic radiation.

[0291] (61) The endoscope system according to embodiment 60, wherein the polarizing filter is located at the proximal end of the lumen. (62) The endoscope system according to embodiment 60, wherein the polarizing filter is located at the distal end of the lumen. (63) The endoscope system according to Embodiment 1, wherein the lens assembly comprises an electromagnetic radiation filter. (64) The endoscope system according to Embodiment 1, wherein the lens assembly comprises a polarizing filter. (65) The endoscope system according to Embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by one or more image sensors, and one or more exposure frames are supplied to a corresponding system that provides the location of important tissue structures.

[0292] (66) The endoscope system according to embodiment 65, wherein the location of the important structure is received by the endoscope system and superimposed on a display, and the important structure is encoded in any color selected by either an algorithm or a user.

Claims

1. An endoscope system for use in environments with insufficient light, An imaging device, An endoscope tube consisting of a proximal end and a distal end, One image sensor located at the distal end and An imaging device comprising: a lens assembly including at least one optical element corresponding to the image sensor; A display for the user to visualize the scene and Image signal processing controller, It is a light engine, A lighting source including an emitter that generates multiple electromagnetic radiation pulses, A light engine comprising: a collection region configured to supply a plurality of electromagnetic radiation pulses generated by the emitter to a waveguide, The waveguide is configured to transmit the plurality of electromagnetic radiation pulses to the distal end, The emitter comprises a visible light-emitting element and a plurality of hyperspectral emitters, each configured to pulse electromagnetic radiation covering a specific wavelength band. The aforementioned multiple hyperspectral emitters, A first hyperspectral emitter configured to pulse electromagnetic radiation covering a wavelength band from 500 nm to 540 nm, A second hyperspectral emitter configured to pulse electromagnetic radiation covering a wavelength band from 540 nm to 640 nm, It comprises a third hyperspectral emitter configured to pulse electromagnetic radiation covering a wavelength band from 900 nm to 1000 nm, The emitter is configured to generate the plurality of electromagnetic radiation pulses during the blanking period of the image sensor. The image sensor captures the reflection of electromagnetic radiation from the emitter and generates hyperspectral image data. The hyperspectral image data is read out and one or more hyperspectral exposure frames are generated. The aforementioned image sensor is a monochrome sensor, The image signal processing controller is configured to assign the colors of a given dataset to the pixel array of the image sensor based on prior information from the emitter. Endoscopic system.

2. The endoscopic system according to claim 1, characterized in that the image signal processing controller is configured to instruct one or more of the emitters to modulate the intensity and / or duration of one or more of the plurality of electromagnetic radiation pulses.

3. The endoscope system according to claim 1, wherein the plurality of electromagnetic radiation pulses include subpulses having a sub-duration shorter than the duration of the plurality of electromagnetic radiation pulses, and the subpulses have a duration that provides a single-frame exposure.

4. The endoscopic system according to claim 3, characterized in that the sub-duration is variable for each sub-pulse of electromagnetic radiation.

5. The endoscopic system according to claim 3, wherein the sub-duration includes two or more subpulses of electromagnetic radiation with equal durations.

6. The endoscope system according to claim 3, wherein the subpulse is a plurality of subpulses, and one or more of the plurality of subpulses includes electromagnetic radiation emitted simultaneously as a single pulse or a single subpulse at two or more wavelengths.

7. The endoscopic system according to claim 3, characterized in that the subpulse is a plurality of subpulses, and the subduration of the plurality of subpulses is variable in both the emitted electromagnetic radiation wavelength and the number of emissions generated per pulse duration.

8. The endoscopic system according to claim 1, characterized in that the sequence of pulses of various wavelengths of electromagnetic radiation can be adjusted by the user of the light engine.

9. The endoscope system according to claim 1, further comprising a control system for appropriately illuminating the scene, wherein the control system automatically adjusts or modifies a sequence of multiple electromagnetic radiation pulses emitted from the light engine based on a threshold, and the threshold deductively determines appropriate illumination of the scene.

10. The endoscope system according to claim 1, wherein each pulse of the plurality of electromagnetic radiation pulses results in one or more exposure frames created by one or more image sensors, the one or more exposure frames consist of one or more hyperspectral exposure frames, and the one or more exposure frames are displayed to the user as a single image on a display or as an overlay image on a display.

11. The endoscope system according to claim 10, characterized in that the single image or the overlay image is assigned visible colors for use on a display, and the visible colors are 8-bit, 16-bit, or n-bit.

12. The endoscopic system according to claim 1, characterized in that the illumination light source generates the plurality of electromagnetic radiation pulses using one or more laser emitters.

13. The plurality of hyperspectral emitters of the illumination light source are further, A fourth hyperspectral emitter that generates one or more electromagnetic radiation pulses covering a wavelength band of 425 nm to 475 nm; A fifth hyperspectral emitter that generates one or more electromagnetic radiation pulses covering a wavelength band of 520 nm to 545 nm; A sixth hyperspectral emitter that generates one or more electromagnetic radiation pulses covering a wavelength band of 625 nm to 645 nm; A seventh hyperspectral emitter that generates one or more electromagnetic radiation pulses covering a wavelength band of 760 nm to 795 nm; An eighth hyperspectral emitter that generates one or more electromagnetic radiation pulses covering a wavelength band of 795 nm to 815 nm; A ninth hyperspectral emitter that generates one or more electromagnetic radiation pulses covering a wavelength band of 370 nm to 420 nm, or A tenth hyperspectral emitter that generates one or more electromagnetic radiation pulses covering a wavelength band of 600 nm to 670 nm, The endoscopic system according to claim 1, comprising one or more of the following.

14. The endoscopic system according to claim 1, characterized in that the light engine includes a polarizing filter, and the polarizing filter is located in the path of the plurality of electromagnetic radiation pulses, at the proximal end of the waveguide, or at the distal end of the waveguide.

15. The endoscope system according to claim 1, characterized in that the lens assembly comprises an electromagnetic radiation filter and / or a polarizing filter.

16. The endoscopic system according to claim 1, characterized in that the blanking period consists of one or more different blanking periods, and the one or more different blanking periods are variable.

17. The endoscopic system according to claim 1, characterized in that the plurality of hyperspectral emitters are configured to extract one or more spectral responses from tissue, and the hyperspectral image data includes the one or more spectral responses.

18. The endoscopic system according to claim 1, wherein the one or more hyperspectral exposure frames are supplied to a corresponding system that provides the location of one or more tissue structures.

19. The endoscopic system according to claim 18, characterized in that the location of one or more tissue structures is received by the endoscopic system and superimposed on a display, and the one or more tissue structures are encoded in any color selected by either an algorithm or a user.

20. The endoscopic system according to claim 1, wherein the plurality of hyperspectral emitters emit electromagnetic radiation that penetrates into tissue by 5 nm, 8 nm, or 10 nm.

Images