Hyperspectral imaging with tool tracking in low-light environments

Abstract

To solve such a problem that a medical endoscope inevitably becomes small and cannot be adapted to multiple different imaging and ranging systems.SOLUTION: An endoscopic imaging system for use in a light deficient environment includes an imaging device having a tube, one or more image sensors, and a lens assembly including at least one optical element that corresponds to the one or more image sensors. The endoscopic system includes a display for a user to visualize a scene and an image signal processing controller. The endoscopic system includes a light engine having an illumination source generating one or more pulses of electromagnetic radiation and a lumen transmitting one or more pulses of electromagnetic radiation to a distal tip of an endoscope.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Advances in technology have led to advances in medical imaging capabilities. Endoscopes can be used to see inside the body and examine the interior of body organs or cavities. Endoscopes can be used to investigate patient symptoms, confirm diagnoses, or provide medical treatment. Medical endoscopes can be used to view various body systems and parts, such as the digestive tract, airway, urinary tract, and abdominal cavity, through small incisions. Endoscopes can also be used for surgical procedures, such as plastic surgery procedures, procedures performed on joints or bones, procedures performed on the nervous system, and procedures performed within the abdominal cavity.

[0002] Endoscopes have also been used in non-medical fields to view and inspect spaces that may be inaccessible or difficult to see. For example, endoscopes can be used by planners or architects to visualize scale models of proposed buildings or cities. Endoscopes can be used to visualize the interior spaces of complex systems such as computers. Endoscopes can even be used by law enforcement or military personnel to monitor tight spaces or inspect explosive devices. Furthermore, endoscopic imaging can be used by computer-implemented programs to perform robotic medical procedures. In particular, in such implementations, it can be beneficial to track the position of an object during endoscopic imaging. The object may include the endoscope itself, a medical device, tissue or a structure within the body, etc.

[0003] Among various applications of endoscopy, it can be beneficial to view space in color. A digital color image can include at least three layers, or "color channels," for each pixel in the image. Each color channel measures the intensity and chrominance of light in a spectral band. Typically, a digital color image includes color channels for the red, green, and blue spectral bands of light (sometimes referred to as an RGB image). Each of the red, green, and blue color channels contains luminance information for the red, green, or blue spectral band of light. The luminance information from the separate red, green, and blue layers can be combined to create a digital color image. Because color images are composed of separate layers, digital camera image sensors typically include a color filter array that allows red, green, and blue visible light wavelengths to impinge on selected pixel sensors. Each individual pixel sensor element is sensitive to red, green, or blue wavelengths and returns image data for only those wavelengths. Image data from the entire array of pixel sensors is combined to create an RGB image.

[0004] In the case of endoscopic imaging for medical diagnosis or medical treatment, it may be beneficial, or even necessary, to view a body cavity with a color image. For example, when using an endoscope to view the abdominal cavity of a body, a color image can provide useful information to aid in identifying different organs or tissues within the abdomen or identifying specific conditions or diseases within the space. As discussed above, a digital camera capable of capturing color images may have at least three different types of pixel sensors to individually capture the red, green, and blue layers of the color image. At least three different types of pixel sensors may consume a relatively large physical space (when compared to a color-agnostic pixel array), and thus the entire pixel array cannot be fitted into the small distal end of an endoscope that is inserted into the body. Because a color digital camera may include at least three different types of pixel sensors, the entire pixel array (i.e., the image sensor) is typically located within the endoscope handpiece unit, which is held by the endoscope operator and is not positioned within the body cavity. In such endoscopes, light is transmitted along the length of the endoscope from a handpiece unit to the distal tip of the endoscope, which is positioned within a body cavity. This endoscope configuration has significant limitations. Endoscopes with this configuration are delicate and can be prone to displacement or damage when bumped or impacted during normal use. This can significantly degrade the quality of the images produced by the endoscope and can require frequent repair or replacement of the endoscope.

[0005] Endoscopic imaging can be used to guide medical practitioners during medical procedures, such as diagnostic imaging or surgical procedures. Additionally, in some implementations, it may be desirable to use endoscopic imaging and endoscopic medical devices to perform computer-implemented robotic surgery. In either case, it may be desirable to determine precise measurements indicating, for example, distances and / or angles between structures within the body, devices or tools within body cavities, and critical structures within the body. Such measurements can improve the results of endoscopic procedures and may be necessary in the case of robotic endoscopic procedures.

[0006] Various measurement systems and methods exist in the art for applications such as archaeology, geography, atmospheric physics, and autonomous vehicles. One such system is light detection and ranging (LIDAR), a three-dimensional laser scanning system. LIDAR has been applied to navigation systems, such as aircraft or satellites, to determine the position and orientation of the sensor in combination with other systems and sensors. LIDAR can use active sensors to illuminate an object and detect the energy that reflects off the object and returns to the sensor. Laser scanning technology has been applied to airborne and terrestrial environments. Airborne laser scanning has been used by aircraft in flight to generate three-dimensional point clouds of the aircraft's surroundings. Terrestrial laser scanning has been used to survey stationary or moving objects on the Earth's surface for use in topography, surveillance, and positioning.

[0007] However, applications of laser scanning technology known in the art typically require highly specialized equipment that may not be useful for additional applications. Furthermore, laser scanning technology provides a limited view of the environment and typically must be used in conjunction with multiple separate systems. For example, autonomous vehicles deploy LIDAR systems in conjunction with multiple imaging cameras, radar sensors, ultrasonic sensors, etc. The many sensors and imaging systems required by autonomous vehicles or other systems can be very costly and consume significant physical space. In the context of medical imaging procedures, such as medical endoscopic imaging, all sensors must fit within a small physical area within a body cavity. In some cases, for example, imaging of a joint or organ, the geographic area may be exceptionally small, allowing only a very small tip of the endoscope to accommodate. Therefore, medical endoscopes are necessarily small and cannot accommodate multiple different imaging and ranging systems.

[0008] In certain implementations, it may be desirable to deploy an endoscopic imaging system for generating images of the body cavity and also for determining measurements within the body cavity. Such images and measurements can be provided to a user for viewing the body cavity and / or can be used by a computer-implemented program for running the robotic system. Summary of the Invention [Means for solving the problem]

[0009] The present disclosure relates generally to electromagnetic sensing and sensors applicable to endoscopic imaging. The disclosure also relates to low-energy electromagnetic input conditions and low-energy electromagnetic throughput conditions. The present disclosure more specifically, but not necessarily, relates to systems for generating images in low-light environments, and related structures, methods, and features, which may include controlling a light source through duration, intensity, or both, pulsing a content-controlled light source during blanking periods of an image sensor, maximizing blanking periods to allow optimal light, and maintaining color balance.

[0010] The features and advantages of the present disclosure will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the disclosure without undue experimentation. The features and advantages of the present disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. [Brief explanation of the drawings]

[0011] Non-limiting and non-exhaustive implementations of the present disclosure are described with reference to the following figures, in which like reference numerals refer to like parts throughout the various views unless otherwise specified. Advantages of the present disclosure will be better understood with reference to the following description and the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of an embodiment of a system of a pair of sensors and electromagnetic emitters in operation for use in generating images in low light environments, according to one embodiment. [Figure 2] FIG. 1 is a schematic diagram of complementary system hardware. [Figure 2A] FIG. 1 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Figure 2B] FIG. 1 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Figure 2C] FIG. 1 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Figure 2D] FIG. 1 is an illustration of the operational cycle of a sensor used to construct one image frame, according to an embodiment of the present disclosure. [Figure 3] 1 is a graphical representation of the operation of an embodiment of an electromagnetic emitter, according to one embodiment. [Figure 4] 1 is a graphical representation of varying the duration and amplitude of an emitted electromagnetic pulse to provide exposure control, according to one embodiment. [Figure 5] 5 is a graphical representation of one embodiment of the present disclosure combining the operational cycles of the sensors, electromagnetic emitters, and emitted electromagnetic pulses of FIGS. 2A-4 showing an imaging system in operation, according to one embodiment. [Figure 6] FIG. 1 is a schematic diagram of two different processes over the period t(0) to t(1) for recording video frames of full spectrum and split spectrum light, according to one embodiment. [Figure 7A] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7B] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7C]FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7D] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 7E] FIG. 1 is a schematic diagram of a process over a time interval for recording video frames of both full spectrum and split spectrum light in accordance with the principles and teachings of the present disclosure. [Figure 8] FIG. 1 illustrates the adjustment of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 9] FIG. 1 illustrates the adjustment of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 10] FIG. 1 illustrates the adjustment of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 11] FIG. 1 illustrates the adjustment of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 12] FIG. 1 illustrates the adjustment of both an electromagnetic emitter and a sensor, which in some embodiments can occur simultaneously in accordance with the principles and teachings of the present disclosure. [Figure 13] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 14] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 15] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 16] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 17] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 18] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 19] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 20] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 21] FIG. 1 is a schematic diagram of a sensor correction method and hardware for use with a split optical system, according to an embodiment of the present disclosure. [Figure 22] 1 is a schematic diagram of a method and hardware for increasing dynamic range in closed or limited light environments, according to an embodiment of the present disclosure. FIG. [Figure 23] 1 is a schematic diagram of a method and hardware for increasing dynamic range in closed or limited light environments, according to an embodiment of the present disclosure. FIG. [Figure 24] FIG. 1 illustrates the effect on the signal-to-noise ratio of color correction of a typical Bayer-based sensor compared to one without color correction. [Figure 25] FIG. 1 illustrates the chromaticity of three monochrome lasers compared to the sRGB gamut. [Figure 26] 1 is a schematic diagram of a method and hardware for increasing dynamic range in closed or limited light environments, according to an embodiment of the present disclosure. FIG. [Figure 27A] 1 is a schematic diagram of a method and hardware for increasing dynamic range in closed or limited light environments, according to an embodiment of the present disclosure. FIG. [Figure 27B]1 is a schematic diagram of a method and hardware for increasing dynamic range in closed or limited light environments, according to an embodiment of the present disclosure. FIG. [Figure 28A] 10A-10C illustrate the use of white light emission pulsed and / or synchronized with a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 28B] 10A-10C illustrate the use of white light emission pulsed and / or synchronized with a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 28C] 10A-10C illustrate the use of white light emission pulsed and / or synchronized with a corresponding color sensor, according to an embodiment of the present disclosure. [Figure 29A] FIG. 1 illustrates an implementation having multiple pixel arrays for generating a three-dimensional image, according to an embodiment of the present disclosure. [Figure 29B] FIG. 1 illustrates an implementation having multiple pixel arrays for generating a three-dimensional image, according to an embodiment of the present disclosure. [Figure 30A] FIG. 1 is a perspective view of an implementation of an image sensor constructed on multiple substrates, where multiple pixel columns forming a pixel array are located on a first substrate and multiple circuit columns are located on a second substrate, showing the electrical connections and communication between one pixel column and its associated or corresponding circuit column. [Figure 30B] FIG. 1 is a side view of an implementation of an image sensor constructed on multiple substrates, with multiple pixel columns forming a pixel array located on a first substrate and multiple circuit columns located on a second substrate, illustrating the electrical connections and communication between one pixel column and its associated or corresponding circuit column. [Figure 31A] 1 is a perspective view of an implementation of an image sensor having multiple pixel arrays for generating a three-dimensional image, where the multiple pixel arrays and image sensors are constructed on multiple substrates. [Figure 31B] FIG. 1 is a side view of one implementation of an imaging sensor having multiple pixel arrays for generating a three-dimensional image, where the multiple pixel arrays and image sensors are constructed on multiple substrates. [Figure 32]10A-10C illustrate embodiments of emitters with various mechanical filter and shutter configurations, according to embodiments of the present disclosure. [Figure 33] 10A-10C illustrate embodiments of emitters with various mechanical filter and shutter configurations, according to embodiments of the present disclosure. [Figure 34] 10A-10C illustrate embodiments of emitters with various mechanical filter and shutter configurations, according to embodiments of the present disclosure. [Figure 35] 10A-10C illustrate embodiments of emitters with various mechanical filter and shutter configurations, according to embodiments of the present disclosure. [Figure 36] 10A-10C illustrate embodiments of emitters with various mechanical filter and shutter configurations, according to embodiments of the present disclosure. [Figure 37] FIG. 1 is a schematic diagram illustrating a system for providing illumination to a low-light environment, according to one embodiment. [Figure 38] FIG. 1 is a schematic block diagram illustrating a light source having multiple emitters, according to one embodiment. [Figure 39] FIG. 10 is a schematic block diagram illustrating a light source having multiple emitters, according to another embodiment. [Figure 40] FIG. 10 is a schematic block diagram illustrating a light source having multiple emitters, according to yet another embodiment. [Figure 41] FIG. 1 is a schematic diagram illustrating a single optical fiber outputting through a diffuser at the output to illuminate a scene, according to one embodiment. [Figure 42] FIG. 2 is a block diagram illustrating the generation of a filtered image using a filter, according to one embodiment. [Figure 43] FIG. 1 illustrates a portion of the electromagnetic spectrum divided into multiple different subspectra that may be emitted by an emitter of a light source, according to one embodiment. [Figure 44] FIG. 1 is a schematic diagram illustrating an emission and readout timing diagram for generating a multispectral or hyperspectral image, according to one embodiment. [Figure 45] FIG. 2 is a block diagram illustrating the generation of a filtered image using a filter, according to one embodiment. [Figure 46] FIG. 2 is a block diagram illustrating the generation of a filtered image using multiple filters, according to one embodiment. [Figure 47] FIG. 1 is a schematic diagram illustrating a grid array for tracking objects and / or surfaces, according to one embodiment. [Figure 48] FIG. 1 is a schematic flow chart diagram illustrating a method of emission and readout for generating a multispectral or hyperspectral image, according to one embodiment. [Figure 49] FIG. 1 is a schematic flow chart diagram illustrating a method of emission and readout for generating a fluorescence image, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present disclosure extends to methods, systems, and computer-based products for digital imaging that may be suitable primarily for medical applications such as medical endoscopic imaging. Such methods, systems, and computer-based products disclosed herein can provide imaging or diagnostic capabilities for use in medical robotics applications, such as the use of robotics to perform imaging procedures, surgical procedures, and the like. In the following description of the disclosure, reference is made to the accompanying drawings, which form a part hereof, and in which is shown, by way of illustration, specific implementations in which the present disclosure may be practiced. It will be understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure.

[0013] Endoscopes have a wide variety of uses and can provide significant benefits in the medical field. Endoscopy is used in medicine to see inside the body and, in some cases, can provide imaging that is otherwise impossible to see or that would require invasive surgical procedures. Endoscopes can be used for medical diagnosis, investigation, or research, and can also be used to perform medical procedures in a minimally invasive manner. Medical endoscopes can provide significant benefits to patients and medical practitioners by negating the need for painful and invasive corrective or exploratory surgery.

[0014] As disclosed herein, an endoscopic system for use in low-light environments, such as body cavities, may include an imaging device and a light engine. The light engine may include an illumination source for generating pulses of electromagnetic radiation and may further include a lumen at the distal tip of the endoscope for transmitting the pulses of electromagnetic radiation. The lumen may transmit the pulses of electromagnetic radiation at a specific wavelength or band of wavelengths in the electromagnetic spectrum. The lumen may transmit such pulses in a timed sequence, and imaging data may be captured by a sensor during each pulse. The imaging data associated with pulses of different wavelengths may be used to generate red-green-blue (RGB) images and / or fluorescence images. In one embodiment, the fluorescence imaging may be overlaid on a black-and-white image or an RGB image.

[0015] As disclosed herein, systems, methods, and devices for endoscopic imaging systems can provide specialized image data for low-light environments. The specialized image data can be used to generate fluorescent images and / or identify specific materials, tissues, components, or processes within low-light environments. In certain embodiments, the fluorescent images can be provided to a practitioner or a computer-implemented program to enable identification of specific structures or tissues within the body. Such fluorescent imaging data can be overlaid on black-and-white or RGB images to provide additional information and context.

[0016] Furthermore, such systems, methods, and devices for endoscopic imaging systems can be used in concert with specific reagents or dyes. In one implementation of medical imaging, a specific reagent or dye can be administered to a patient, and the reagent or dye can fluoresce or respond to electromagnetic radiation of a specific wavelength. The endoscopic imaging systems disclosed herein can transmit electromagnetic radiation of a specific wavelength to cause the reagent or dye to fluoresce. The fluorescence of the reagent or dye can be captured by an image sensor to generate an image to aid in the identification of tissue or structures and / or to aid in diagnosis or investigation. In one implementation, a patient can be administered multiple reagents or dyes, each configured to fluoresce at a different wavelength and / or to provide an indication of a different structure, tissue, chemical reaction, biological process, etc. In such an implementation, the endoscopic systems disclosed herein can cause each applicable reagent or dye to fluoresce by emitting each applicable wavelength. This can negate the previous need to perform individual imaging procedures for each of the multiple reagents or dyes.

[0017] Medical endoscopes can provide a continuous digital image stream of the interior space of the body into which the distal end of the endoscope is inserted. In various implementations, it may be beneficial or even necessary for the digital image stream to provide full-color imaging so that medical practitioners can better distinguish between tissues and structures within the body. In further implementations, it may be beneficial to provide hyperspectral imaging data to distinguish between structures, tissues, processes, and conditions with increased accuracy. Additionally, hyperspectral imaging can enable medical practitioners or computer programs to receive information about conditions within the human body that cannot be seen by the human eye or identified in RGB color images.

[0018] Disclosed herein are systems, methods, and devices for generating color image data and / or fluorescence image data via an endoscope. The disclosed system includes an imaging device having a tube, one or more image sensors, and a lens assembly. The lens assembly can include at least one optical element corresponding to at least one of the one or more image sensors. The system can further include a display for visualizing the scene and an image signal processing controller. The system can further include a light engine. The light engine includes an illumination source configured to generate one or more pulses of electromagnetic radiation and a lumen for transmitting the one or more pulses of electromagnetic radiation to the distal tip of the endoscope. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation include an excitation wavelength of electromagnetic radiation between 770 nm and 790 nm that causes one or more reagents to fluoresce at a wavelength different from the excitation wavelength of a portion of the one or more pulses of electromagnetic radiation.

[0019] In one embodiment of the present disclosure, an endoscopic system illuminates a source and pulses electromagnetic radiation at a specific wavelength to excite electrons in a reagent or dye. In one embodiment, the reagent or dye is configured to emit fluorescence in response to the specific wavelength of electromagnetic radiation emitted by the endoscopic system. An image sensor of the endoscopic system can read the fluorescent relaxation emission of the reagent or dye, which can be of lower energy than the pulsed electromagnetic radiation to excite the reagent or dye. The reagent or dye can be specialized to classify specific tissues, structures, biological processes, and / or chemical processes.

[0020] Imaging reagents, such as fluorescent reagents, can enhance imaging capabilities in the pharmaceutical, medical, biotechnology, diagnostic, and medical treatment industries. Many imaging techniques, such as X-ray, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI), and nuclear medicine, primarily analyze anatomy and morphology and cannot detect changes at the molecular level. Fluorescent reagents, dyes, and probes, including quantum dot nanoparticles and fluorescent proteins, can assist medical imaging techniques by providing additional information about specific tissues, structures, chemical processes, and / or biological processes present within the imaging area. Imaging using fluorescent reagents can enable cell tracking and / or tracking of specific molecular biomarkers. Fluorescent reagents can be applied to image cancer, infectious diseases, inflammation, stem cell biology, and more. Numerous fluorescent reagents and dyes have been developed and applied to visualize and track biological processes in a non-destructive manner. Such fluorescent reagents can be excited by electromagnetic radiation of specific wavelengths or bands of wavelengths. Similarly, such fluorescent reagents, when fluorescing, can emit relaxation energy at a particular wavelength or band of wavelengths, which can be read by a sensor to determine the location and / or boundaries of the reagent or dye.

[0021] In one embodiment of the present disclosure, an endoscopic system pulses electromagnetic radiation to excite electrons in a fluorescent reagent or dye. The wavelength or band of wavelengths of the electromagnetic radiation can be specifically selected to cause a particular reagent or dye to emit fluorescence. In one embodiment, the endoscopic system can pulse electromagnetic radiation of multiple different wavelengths to cause multiple different reagents or dyes to emit fluorescence during a single imaging session. A sensor in the endoscopic system can determine the location and / or boundaries of the reagent or dye based on the relaxation emission of the reagent or dye. The endoscopic system can also pulse electromagnetic radiation in the red, green, and blue bands of visible light. The endoscopic system can determine RGB image and fluorescence image data according to a pulsing schedule of the electromagnetic radiation pulses.

[0022] In one embodiment of the present disclosure, an endoscopic system illuminates a source for spectral or hyperspectral imaging and pulses electromagnetic radiation. Spectral imaging uses multiple bands across the electromagnetic spectrum. This differs from traditional cameras, which only capture light across three wavelengths based on the visible spectrum that are distinguishable by the human eye, including red, green, and blue wavelengths to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, the visible spectrum, the ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can overlay imaging based on invisible bands (e.g., infrared) on top of imaging based on visible bands (e.g., standard RGB images) to provide additional information that can be easily discerned by humans or computer algorithms.

[0023] Hyperspectral imaging is a subcategory of spectral imaging. Hyperspectral imaging includes spectroscopy and digital photography. In one embodiment of hyperspectral imaging, a full spectrum or some spectral information is collected at every pixel in the image plane. Using specialized hardware, a hyperspectral camera can capture any suitable number of wavelength bands for each pixel, which can be interpreted as a full spectrum. The goal of hyperspectral imaging can be different for different applications. In one application, the goal of hyperspectral imaging is to obtain the entire electromagnetic spectrum for each pixel in an image scene. This can make it possible to find specific objects that may otherwise be indistinguishable under visible light wavelength bands. This can make it possible to accurately identify specific materials or tissues when they may be indistinguishable under visible light wavelength bands. Furthermore, this can make it possible to detect specific processes by capturing images across all wavelengths of the electromagnetic spectrum.

[0024] Hyperspectral imaging can offer certain advantages over conventional imaging in medical applications. Information obtained by hyperspectral imaging can enable medical practitioners and / or computer-implemented programs to accurately identify specific tissues or conditions that may lead to diagnoses that may be impossible or less accurate when using conventional imaging, such as RGB imaging. Additionally, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, which can allow medical practitioners, for example, to view tissues located behind specific tissues or fluids, identify atypical cancer cells as opposed to typical healthy cells, identify specific tissues or conditions, identify critical structures, etc. Hyperspectral imaging can provide specialized diagnostic information regarding tissue physiology, morphology, and composition that cannot be generated using conventional imaging.

[0025] Endoscopic hyperspectral imaging may offer advantages over conventional imaging in various applications and implementations of the present disclosure. In medical implementations, endoscopic hyperspectral imaging allows a practitioner or computer-implemented program to identify, for example, nerve tissue, muscle tissue, various vascular systems, the direction of blood flow, and the like. Hyperspectral imaging can precisely distinguish atypical cancerous tissue from typical healthy tissue, thus enabling a practitioner or computer-implemented program to identify the boundaries of cancerous tumors during surgery or investigative imaging. Additionally, the hyperspectral imaging in low-light environments disclosed herein can be combined with the use of reagents or dyes to further differentiate specific tissues or materials. In one such embodiment, the reagent or dye can fluoresce with specific wavelength bands within the electromagnetic spectrum, thus providing information specific to the purpose of the reagent or dye. The systems, methods, and devices disclosed herein can enable pulsing of any number of wavelength bands, such that one or more reagents or dyes can fluoresce at different times. In certain embodiments, this can enable the identification or investigation of many medical conditions during a single imaging procedure.

[0026] Medical endoscopes can pulse electromagnetic radiation in wavelength bands outside the visible light spectrum, enabling the production of hyperspectral images. Endoscopic hyperspectral imaging is a non-contact, non-invasive means of medical imaging that does not require patients to undergo harmful radiation exposure common in other imaging methods.

[0027] Conventional endoscopes used in robotic endoscopic procedures, such as arthroscopy and laparoscopy, are designed so that the image sensor is located in a handpiece unit that is typically held by the endoscope operator and not inserted into the cavity. In this configuration, the endoscope unit transmits incident light along the length of the endoscope tube toward the sensor with minimal loss and distortion via a complex set of precisely coupled optical components. Because optical components are expensive and the manufacturing process for them is labor-intensive, the optical components dominate the cost of the endoscope unit. Furthermore, this type of endoscope is mechanically delicate, and relatively minor impacts can easily damage components or upset their relative alignment. Even slight misalignment of endoscope components (such as precisely coupled optical components) can cause a significant degradation in image quality or render the endoscope unusable. When components are misaligned, the incident light traveling along the length of the endoscope is reduced, resulting in little or no light at the distal end of the endoscope, rendering the endoscope unusable. Because conventional endoscopes require such precise and complex optical components that can easily become misaligned, such conventional endoscopes require frequent and expensive repair cycles to maintain image quality.

[0028] One solution to this problem is to place the image sensor at the distal tip within the endoscope itself. Such a solution could eliminate the need for a complex, precision-linked collection of optical components that can easily become displaced and / or damaged. This solution potentially addresses the optical simplicity, robustness, and economy commonly realized in, for example, cell phone cameras. However, it should be understood that many of the advantages offered by endoscopes result from the compactness of the endoscope's distal tip. If the distal tip of an endoscope were enlarged to accommodate multiple different wavelength-sensitive pixel sensors traditionally used for color or hyperspectral imaging, the pixel array could become too large, and the endoscope could no longer fit into tight spaces or could become obstructive or invasive when used in medical implementations. Because the distal tip of an endoscope must remain very small, placing more than one image sensor at the distal tip is difficult. An acceptable solution to this method is far from trivial and presents its own set of engineering challenges, not least of which is the fact that sensors for color and / or hyperspectral imaging must be fitted into a very limited area. This is particularly difficult when the pixel array of a conventional camera includes separate pixel sensors for each of the red, green, and blue visible light bands, along with additional pixel sensors for the other wavelength bands used for hyperspectral imaging. While the area at the distal tip of the endoscope may be limited left and right, particularly in the X and Y dimensions, there is more space along the length of the endoscope tube in the Z dimension.

[0029] Because many of the benefits of endoscopes derive from the miniaturization of their distal tip, challenging limitations must be placed on the image sensor area when the image sensor is located at the distal tip. These challenging limitations on sensor area inevitably result in fewer and / or smaller pixels in the pixel array. Reducing the number of pixels can directly affect spatial resolution, while reducing pixel area and optimizing the original number of pixels using pixel count, minimum pixel resolution, and maximum pixel quality and pitch to maximize image quality can reduce the available signal capacity and therefore pixel sensitivity, resulting in resolution becoming less of an issue and reducing the signal-to-noise ratio (SNR) of each pixel. Reducing signal capacity reduces the dynamic range—i.e., the ability of an imaging device or camera to simultaneously capture all useful information from a scene with a wide range of luminosity. Various methods exist for extending the dynamic range of an imaging system beyond the dynamic range of the pixels themselves. However, they all may have some penalty (e.g., in resolution or frame rate) and may introduce undesirable artifacts that become problematic in extreme cases. Reducing sensitivity results in darker areas of the scene requiring more optical power to bring them to an acceptable signal level. Lowering the F-number (larger aperture) can compensate for the loss of sensitivity, but at the expense of spatial distortion and a shallower depth of focus.

[0030] In the sensor industry, complementary metal-oxide-semiconductor ("CMOS") image sensors have largely replaced traditional charge-coupled device ("CCD") image sensors in modern camera applications. Compared to CCD image sensors, CMOS image sensors are easier to integrate and operate, offer better or equivalent image quality, are more versatile, and are less expensive. Typically, CMOS image sensors can include the circuitry necessary to convert image information into digital data, which can then incorporate various levels of digital processing. This can range from basic algorithms to correct for non-idealities that may arise from, for example, variations in amplifier behavior, to a complete image signal processing (ISP) chain, providing video data in, for example, the standard red-green-blue ("RGB") color space (camera-on-chip).

[0031] The control unit for the endoscope or image sensor may be located remotely from the image sensor and may be a significant physical distance from the image sensor. When the control unit is remote from the sensor, it may be desirable to communicate data in the digital domain, as this is less susceptible to interference noise and signal degradation when compared to communicating an analog data stream. It will be appreciated that various electrical digital signaling standards may be used, such as low voltage differential signaling (LVDS), sub-LVDS, scalable low voltage signaling (SLVS), or other electrical digital signaling standards.

[0032] There may be a strong desire to minimize the number of electrical conductors, reducing the number of space-consuming pads on the sensor and reducing the complexity and cost of sensor manufacturing. It may be advantageous to add analog-to-digital conversion to the sensor, but the additional area occupied by the conversion circuitry is offset by the significant reduction in analog buffering power required by the initial conversion to a digital signal.

[0033] With respect to area consumption, given the typical feature sizes available in CMOS image sensor (CIS) technology, in some implementations it may be preferable to have all internal logic signals generated on the same chip as the pixel array via a set of control registers and a simple command interface.

[0034] Some implementations of the present disclosure can include combined sensor and system design aspects that enable high-resolution imaging with reduced pixel counts in highly controlled lighting environments. This can be achieved by pulsing a single color wavelength frame by frame, and by switching or alternating each frame between a single different color wavelength using a controlled light source in conjunction with a high frame capture rate and a corresponding specially designed monochrome sensor. In addition, electromagnetic radiation outside the visible light spectrum can be pulsed to enable the generation of hyperspectral images. The pixels can be color-independent, such that each pixel can generate data for each pulse of electromagnetic radiation, including pulses of red, green, and blue visible light wavelengths, along with other wavelengths that can be used for hyperspectral imaging.

[0035] As used herein, a monochrome sensor refers to an unfiltered imaging sensor. Because the pixels are color-independent, the effective spatial resolution is significantly higher than their color (usually Bayer-pattern filtered) counterparts in conventional single-sensor cameras. The pixels may also have higher quantum efficiency because far fewer incident photons are wasted between individual pixels. Furthermore, Bayer-based spatial color modulation requires a reduction in the modulation transfer function (MTF) of the accompanying optics compared to the monochromatic case in order to blur the color artifacts associated with the Bayer pattern. This has a negative impact on the actual spatial resolution that can be achieved with a color sensor.

[0036] This disclosure also relates to a system solution for endoscopic applications where the image sensor resides at the distal end of the endoscope. In the pursuit of the smallest area sensor-based system, there are other design aspects that can be exploited besides reducing the number of pixels. The area of ​​the digital portion of the chip can be minimized. Additionally, the number of connections to the chip (pads) can also be minimized. This disclosure describes a novel method to achieve these goals for realizing such a system. This involves the design of a fully custom CMOS image sensor with several novel features.

[0037] For the purposes of promoting an understanding of the principles according to the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is thereby intended. Any alterations and further modifications of the features of the invention illustrated herein, and any additional applications of the principles of the present disclosure as illustrated herein, which would occur to one of ordinary skill in the art and possessor of this disclosure, are deemed to be within the scope of the disclosure as claimed.

[0038] Before structures, systems, and methods for generating images in low-light environments are disclosed and described, it is to be understood that the disclosure is not limited to the particular structures, configurations, process steps, and materials disclosed herein, as such structures, configurations, process steps, and materials may vary somewhat. It is also to be understood that the terminology used herein is used for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present disclosure will be limited only by the appended claims and equivalents thereof.

[0039] In describing and claiming the subject matter of the present disclosure, the following terminology will be used in accordance with the definitions set out below.

[0040] It must be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0041] As used herein, the terms "comprising," "including," "containing," "characterized by," and their grammatical equivalents are inclusive or broad terms that do not exclude additional, unrecited elements or method steps.

[0042] As used herein, the word "consisting of" and its grammatical equivalents exclude any element or step not specified in the claim.

[0043] As used herein, the phrase "consisting essentially of" and its grammatical equivalents limit the scope of a claim to the specified materials or steps and those that do not materially affect the basic and novel feature or features of the claimed disclosure.

[0044] As used herein, the term "proximal" refers broadly to the concept of the part closest to the origin.

[0045] As used herein, the term "distal" generally refers to the opposite of proximal, and thus to the concept of being further or furthest from the origin, depending on the context.

[0046] As used herein, a color sensor or multispectral sensor is a sensor known to have a color filter array (CFA) thereon to filter incident electromagnetic radiation into its separate components. In the electromagnetic spectrum, such a CFA can be structured in a Bayer pattern or a modification thereof to separate the green, red, and blue spectral components of light.

[0047]

[0013] Referring now to Figures 1-5, a system and method for generating images in low light environments will now be described. Figure 1 illustrates a schematic diagram of a pair of sensors and electromagnetic emitters in operation for use in generating images in low light environments. Such a configuration allows for enhanced functionality in light-controlled or low ambient light environments.

[0048] As used herein, the term "light" refers to electromagnetic radiation, both particles and wavelengths, that is detectable by the pixel array, and it should be noted that this can include wavelengths in the visible and non-visible spectrum of electromagnetic radiation. The term "segment" is used herein to refer to a range of wavelengths in the electromagnetic spectrum that is less than the entire spectrum, in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that may be controllable with respect to the physical properties of its components, the intensity of its emission, or the duration of its emission, or the portion of the electromagnetic spectrum that it emits or can operate on with respect to all of the above. Emitters can emit light with any dithered, diffused, or collimated emission and can be controlled digitally or via analog methods or systems. As used herein, an electromagnetic emitter is a source of a burst of electromagnetic energy and includes light sources such as lasers, LEDs, incandescent lights, or any light source that can be digitally controlled.

[0049] The pixel array of the image sensor may be electrically paired with the emitter, so that they are synchronized to receive emissions and adjustments made within the system during operation. As can be seen in FIG. 1 , the emitter 100 can be adjusted to emit electromagnetic radiation in the form of a laser, which can be pulsed to illuminate the object 110. The emitter 100 can be pulsed at intervals corresponding to the operation and functionality of the pixel array 122. The emitter 100 can pulse light in multiple electromagnetic sections 105, so that the pixel array receives electromagnetic energy and generates a data set corresponding (in time) to each specific electromagnetic section 105. For example, FIG. 1 shows a system having a monochrome sensor 120 with a pixel array (black and white) 122 and supporting circuitry, where the pixel array 122 is sensitive to electromagnetic radiation of any wavelength. The light emitter 100 illustrated in the figure may be a laser emitter capable of emitting red electromagnetic segments 105a, blue electromagnetic segments 105b, and green electromagnetic segments 105c in any desired order. In an embodiment capable of producing hyperspectral images, the light emitter 100 may pulse electromagnetic radiation at any wavelength within the electromagnetic spectrum, thereby producing a hyperspectral image. It will be understood that other light emitters 100, such as digital or analog-based emitters, may be used in FIG. 1 without departing from the scope of the present disclosure.

[0050] During operation, the data produced by the monochrome sensor 120 for any individual pulse can be assigned a particular color or wavelength segment, based on the timing of the color or wavelength segment pulsed from the emitter 100. Even if the pixels 122 are not dedicated to a color, they can be assigned a color for any given data set based on prior information about the emitter.

[0051] In one exemplary embodiment of the present disclosure, emitter 100 pulses electromagnetic radiation at specialized wavelengths. Such pulses can enable the generation of specialized fluorescence images that are particularly suited to certain medical or diagnostic applications. In an exemplary embodiment, at least a portion of the electromagnetic radiation emitted by emitter 100 includes electromagnetic radiation at excitation wavelengths between 770 nm and 790 nm and between 795 nm and 815 nm that cause one or more reagents to emit fluorescence at wavelengths different from the excitation wavelength of the portion of the electromagnetic radiation.

[0052] In one embodiment, three data sets representing red, green, and blue electromagnetic pulses may be combined to form a single image frame. One or more additional data sets representing other wavelength intervals may be overlaid onto the single image frame based on the red, green, and blue pulses. The one or more additional data sets may represent, for example, fluorescence imaging responsive to excitation wavelengths of 770 nm to 790 nm and 795 nm to 815 nm. The one or more additional data sets may represent fluorescence imaging and / or hyperspectral imaging that may be overlaid onto the single image frame based on the red, green, and blue pulses.

[0053] It will be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic zone, and that any color combination or any electromagnetic zone may use cyan, magenta, and yellow instead of red, green, and blue, ultraviolet, infrared, any combination of the foregoing, or any other color combination, including all visible and non-visible wavelengths, without departing from the scope of the present disclosure. In the figure, the object 110 being imaged includes a red portion 110a, a green portion 110b, and a blue portion 110c. As shown, the reflected light from the electromagnetic pulse contains data only for portions of the object having the particular color corresponding to the pulsed color zone. These separate color (or color interval) data sets can then be used to reconstruct an image at 130 by combining the data sets.

[0054] In one embodiment, multiple data sets representing red, green, and blue electromagnetic pulses, along with additional wavelength segments along the electromagnetic spectrum, can be combined to form a single image frame in which hyperspectral image data is overlaid on an RGB image. Depending on the application or situation, different combinations of wavelength data sets may be desirable. For example, in some implementations, data sets representing specific wavelength segments can be used to generate specialized hyperspectral images for diagnosing specific medical conditions, studying specific body tissues, etc.

[0055] As illustrated in FIG. 2 , embodiments of the present disclosure may comprise or utilize a special-purpose or general-purpose computer, including computer hardware such as one or more processors and system memory, as discussed in detail below. Embodiments within the scope of the present disclosure may also include physical and other computer-readable media for supporting or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that support computer-executable instructions are transmission media. Thus, by way of non-limiting example, implementations of the present disclosure may include at least two distinctly different types of computer-readable media: computer storage media (devices) and transmission media.

[0056] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSD") (e.g., RAM compatible), flash memory, phase change memory ("PCM"), other types of memory, other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computer.

[0057] A "network" is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules and / or other electronic devices. In one implementation, sensors and camera control units can be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transferred or provided to a computer over a network or another communications connection (hardwired, wireless, or a combination of hardwired or wireless), the computer properly views the connection as a transmission medium. Transmission media can include networks and / or data links that can be used to convey desired program code means in the form of computer-executable instructions or data structures and that are accessible by a general-purpose or special-purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0058] Furthermore, upon reaching various components of a computer system, program code means in the form of computer-executable instructions or data structures may be automatically transferred from transmission media to computer storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to computer system RAM and / or less volatile computer storage media (devices) within the computer system. RAM may also include solid-state drives (SSDs or PCIx-compatible real-time memory tiered storage, such as Fusion IO). Thus, it should be understood that computer storage media (devices) may be included in computer system components that also (or even primarily) utilize transmission media.

[0059] Computer-executable instructions include, for example, instructions and data that, when executed on a processor, cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions. Computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. While the subject matter has been described in language specific to structural features and / or method acts, it should be understood that the subject matter defined within the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as exemplary forms of implementing the disclosure.

[0060] Those skilled in the art will appreciate that the present disclosure can be practiced in networked computing environments having many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, control units, camera control units, handheld devices, handpieces, multiprocessor systems, microprocessor-based or programmable consumer electronics products, network PCs, minicomputers, mainframe computers, cell phones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the computing devices described above can be provided by or located within a brick-and-mortar location. The present disclosure can also be practiced in distributed system environments where both local and remote computer systems, linked via a network (by hardwired data links, wireless data links, or a combination of hardwired and wireless data links), perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0061] Additionally, where appropriate, the functions described herein may be implemented by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) may be programmed to perform one or more of the systems and procedures described herein. Certain terms are used throughout the following specification and claims to refer to particular system components. As will be understood by those skilled in the art, components may be referred to by different names. This document does not intend to distinguish between components that differ in name but not function.

[0062] 2 is a block diagram illustrating an exemplary computing device 150. Computing device 150 can be used to perform various procedures, such as those discussed herein. Computing device 150 can function as a server, a client, or any other computing entity. Computing device 150 can perform various monitoring functions as discussed herein and can execute one or more application programs, such as those described herein. Computing device 150 can be any of a wide variety of types of computing devices, such as a desktop computer, a notebook computer, a server computer, a portable computer, a camera control unit, a tablet computer, etc.

[0063] Computing device 150 includes one or more processors 152, one or more memory devices 154, one or more interfaces 156, one or more mass storage devices 158, one or more input / output (I / O) devices 160, and a display device 180, all coupled to a bus 162. Processor(s) 152 include one or more processors or controllers that execute instructions stored in memory(s) 154 and / or mass storage device(s) 158. Processor(s) 152 may also include various types of computer-readable media, such as cache memory.

[0064] The storage device(s) 154 may include a variety of computer-readable media, such as volatile memory (e.g., random access memory (RAM) 164) and / or non-volatile memory (e.g., read-only memory (ROM) 166). The storage device(s) 154 may also include rewritable ROM, such as flash memory.

[0065] Mass storage device(s) 158 include various computer-readable media, such as magnetic tape, magnetic disks, optical disks, solid-state memory (e.g., flash memory), etc. As shown in Figure 2, a particular mass storage device is a hard disk drive 174. Various drives may also be included in mass storage device(s) 158 to enable reading from and / or writing to the various computer-readable media. Mass storage device(s) 158 include removable media 176 and / or non-removable media.

[0066] I / O device(s) 160 include a variety of devices that allow data and / or other information to be input to or retrieved from computing device 150. Exemplary I / O device(s) 160 include digital imaging devices, electromagnetic sensors and emitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs, or other image capture devices, etc.

[0067] Display device 180 includes any type of device capable of displaying information to one or more users of computing device 150. Examples of display device 180 include a monitor, a display terminal, a video projection device, etc.

[0068] The interface(s) 106 include various interfaces that enable the computing device 150 to interact with other systems, devices, or computing environments. Exemplary interface(s) 156 may include any number of different network interfaces 170, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet. Other interface(s) include a user interface 168 and a peripheral interface 172. The interface(s) 156 may also include one or more user interface elements 168. The interface(s) 156 may also include one or more peripheral interfaces, such as interfaces to a printer, a pointing device (mouse, trackpad, etc.), a keyboard, etc.

[0069] Bus 162 allows processor(s) 152, storage device(s) 154, interface(s) 156, mass storage device(s) 158, and I / O device(s) 160 to communicate with each other and with other devices or components coupled to bus 162. Bus 162 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, etc.

[0070] For purposes of illustration, programs and other executable program components are illustrated herein as separate blocks, with the understanding that such programs and components may reside at various times in different storage components of computing device 150 and be executed by processor(s) 152. Alternatively, the systems and procedures described herein may be implemented in hardware or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to execute one or more of the systems and procedures described herein.

[0071] FIG. 2A illustrates an operational cycle of a sensor used in a rolling readout mode, or during sensor readout 200. A frame readout may begin with and be represented by a vertical line 210. A readout period is represented by a diagonal or oblique line 202. The sensor may be read out row by row, with the top of the downward diagonal edge being the top row of sensors 212 and the bottom of the downward diagonal edge being the bottom row of sensors 214. The time between the readout of the last row and the next readout cycle may be referred to as a blanking time 216. Note that some of the sensor pixel rows may be covered with a light shield (e.g., a substantially black layer of metal coating or any other type of material). These covered pixel rows may be referred to as optical black rows 218 and 220. The optical black rows 218 and 220 may be used as input for a correction algorithm. As shown in FIG. 2A, these optical black rows 218 and 220 can be located at the top of the pixel array, the bottom of the pixel array, or both the top and bottom of the pixel array. FIG. 2B illustrates a process for controlling the amount of electromagnetic radiation, e.g., light, exposed to a pixel and thereby integrated or stored by the pixel. It is understood that photons are elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or stored by each pixel and converted into an electrical charge or current. An electronic shutter, or rolling shutter (shown by dashed line 222), can be used to start the integration time by resetting the pixel. Light can then be integrated until the next readout stage. The position of the electronic shutter 222 can be moved between two readout cycles 202 to control pixel saturation for a given amount of light. Note that this technique allows for a constant integration time between two different lines, but introduces a delay when moving from the top row to the bottom row. FIG. 2C illustrates the case where the electronic shutter 222 is removed. In this configuration, integration of the incident light can begin during readout 202 and can end with the next readout cycle 202, which also defines the start of the next integration. Figure 2D shows a configuration without an electronic shutter 222, but with controlled and pulsed light 230 during blanking time 216.This ensures that all rows see the same light emitted from the same light pulse 230. In other words, each row begins its integration in a dark environment, which may be in the optical black trailing row 220 of the readout frame (m) with the maximum light pulse width, then receives the light strobe, and ends its integration in a dark environment, which may be in the optical black leading row 218 of the next subsequent readout frame (m+1) with the maximum light pulse width. In the example of Figure 2D, the image generated from the light pulse is available only during the readout of frame (m+1), without any interference with frames (m) and (m+2). Note that a condition for having a light pulse that is readout in only one frame and does not interfere with adjacent frames is to have a given light pulse emitted during blanking time 216. Because the optical black rows 218, 220 are insensitive to light, the frame (m) time of the trailing optical black row 220 and the frame (m+1) time of the leading optical black row 218 can be added to the blanking time 216 to determine the maximum range of emission times for the light pulses 230. As illustrated in FIG. 2A , the sensor can be cycled multiple times to receive data for each pulsed color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum). Each cycle can be timed. In one embodiment, the cycles can be timed to operate within a 16.67 ms interval. In another embodiment, the cycles can be timed to operate within an 8.3 ms interval. It will be understood that other time intervals are contemplated by and intended to be within the scope of this disclosure.

[0072] FIG. 3 graphically illustrates the operation of one embodiment of an electromagnetic emitter. The emitter can be timed to correspond to a sensor cycle, so that electromagnetic radiation is emitted within and / or during portions of the sensor operating cycle. FIG. 3 illustrates pulse 1 at 302, pulse 2 at 304, and pulse 3 at 306. In one embodiment, the emitter can be pulsed during the readout portion 202 of the sensor operating cycle. In one embodiment, the emitter can be pulsed during the blanking portion 216 of the sensor operating cycle. In one embodiment, the emitter can be pulsed for a duration during portions of two or more sensor operating cycles. In one embodiment, the emitter can begin pulsing during the blanking portion 216 or during the optical black portion 220 of the readout portion 202, and can end pulsing during the readout portion 202 or during the optical black portion 218 of the readout portion 202 of a subsequent cycle. It will be understood that any combination of the above is intended to fall within the scope of the present disclosure, so long as the emitter pulses and sensor cycles are matched.

[0073] 4 graphically illustrates the variation of the duration and amplitude of emitted electromagnetic pulses (e.g., pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406) to control exposure. An emitter with a constant output amplitude can be pulsed during any of the cycles described above in connection with FIGS. 2D and 3 at intervals to provide the required electromagnetic energy to the pixel array. An emitter with a constant output amplitude can be pulsed at longer intervals to provide more electromagnetic energy to the pixel, or the emitter can be pulsed at shorter intervals to provide less electromagnetic energy. Whether a longer or shorter interval is required depends on the operating conditions.

[0074] As opposed to adjusting the time interval at which the emitter pulses a constant output amplitude, the amplitude of the emission itself can be increased to provide more electromagnetic energy to the pixel. Similarly, decreasing the amplitude of the pulse provides less electromagnetic energy to the pixel. Note that one embodiment of the system may have the ability to simultaneously adjust both amplitude and duration, if desired. Furthermore, the sensor may be adjusted to increase sensitivity and duration as desired for optimal image quality. FIG. 4 illustrates varying the amplitude and duration of the pulses. In the illustration, pulse 1 at 402 has a greater amplitude or intensity than either pulse 2 at 404 or pulse 3 at 406. Additionally, pulse 1 at 402 has a shorter duration than pulse 2 at 404 or pulse 3 at 406, resulting in a reduction in the electromagnetic energy provided by the pulses, as illustrated by the area under the pulses shown in the illustration. In the illustration, pulse 2 at 404 has a relatively smaller amplitude or intensity and a longer duration when compared to either pulse 1 at 402 or pulse 3 at 406. Finally, in the figure, pulse 3 at 406 has an intermediate amplitude or intensity and duration when compared to pulse 1 at 402 and pulse 2 at 404 .

[0075] 5 is a graphical representation of one embodiment of the present disclosure that combines the operating cycle, electromagnetic emitter, and emitted electromagnetic pulses of FIGS. 2-4 to demonstrate an imaging system in operation in accordance with the principles and teachings of the present disclosure. As can be seen, the electromagnetic emitter pulses emissions primarily during the sensor blanking period 216, which fills the pixels and makes them ready for readout during the readout portion 202 of the sensor cycle. The dashed portions of the pulses (from FIG. 3) illustrate the potential or ability to emit electromagnetic energy during the optical black portions 220 and 218 of the read cycle (sensor cycle) 200 when additional time is needed or desired to pulse the electromagnetic energy.

[0076] Referring now to FIGS. 6-9, FIG. 6 shows a schematic diagram of two different processes over the period t(0) to t(1) for recording video frames of full-spectrum and split-spectrum light. Note that color sensors have a color filter array (CFA) for filtering specific wavelengths of light per pixel, which is typically used for full-spectrum light reception. An example of a CFA is the Bayer pattern. Because color sensors may include pixels in an array made sensitive to a single color from the full spectrum, a low-resolution image results because the pixel array has pixel space dedicated to only a single color of light within the full spectrum. Typically, such an array is formed in a checkerboard-type pattern across the array.

[0077] In contrast, when a split spectrum of light is used, a sensor can be made sensitive or responsive to all light energy amplitudes because the pixel array is instructed to sense electromagnetic energy from a predetermined section of the full spectrum of electromagnetic energy in each cycle. Therefore, to form an image, the sensor needs to cycle through different sections from within the full spectrum of light and then reassemble the image to display a predetermined mixture of color values ​​for all pixels in the array. This also provides a higher resolution image because the distance between pixel centers of the same color sensitivity is reduced for each color pulse compared to a Bayer sensor. As a result, the color image formed has a higher modulation transfer function (MTF). Because the image from each color split frame cycle has higher resolution, the resulting image produced when the split light frames are combined into a full-color frame also has higher resolution. In other words, because every pixel in the array (instead of at most every second pixel in the sensor with a color filter) senses the energy amplitude for a given pulse and a given scene only a short time apart, a high resolution image is created for each scene with less acquired (less accurate) data that needs to be introduced.

[0078] For example, white light or full spectrum visible light is a combination of red, green, and blue light. In the embodiment shown in FIG. 6, it can be seen that in both the split-spectrum process 620 and the full-spectrum process 610, the time of image capture is t(0) to t(1). In the full-spectrum process 610, white light or full spectrum electromagnetic energy is emitted at 612. At 614, the white or full spectrum electromagnetic energy is sensed. At 616, the image is processed and displayed. Thus, between times t(0) and t(1), the image is processed and displayed. Conversely, in the split-spectrum process 620, a first segment is emitted at 622 and sensed at 624. At 626, a second segment is emitted and then sensed at 628. At 630, a third segment is emitted and sensed at 632. At 634, the image is processed and displayed. It will be understood that any system that uses an image sensor cycle that is at least twice as fast as the white light cycle is intended to fall within the scope of this disclosure.

[0079] 6, as can be seen in graphical format between times t(0) and t(1), the sensors of split-spectrum system 620 are cycled three times per full spectrum system. In split-spectrum system 620, the first of the three sensor cycles is for green spectra 622 and 624, the second of the three is for red spectra 626 and 628, and the third is for blue spectra 630 and 632. Thus, in one embodiment where the display device (LCD panel) operates at 50-60 frames per second, the split-spectrum system should operate at 150-180 frames per second to maintain continuity and smoothness of the displayed image.

[0080] In other embodiments, there may be different capture and display frame rates. Additionally, the average capture rate may be any multiple of the display rate.

[0081] In one embodiment, it may be desired that not all segments be equally represented within the system frame rate. In other words, not all light sources need be pulsed with the same regularity, to accentuate and de-emphasize aspects of the recorded scene, as desired by the user. It should also be understood that non-visible and visible segments of the electromagnetic spectrum may be pulsed together within the system, with their respective data values ​​incorporated into the video output as desired for display to the user.

[0082] One embodiment may include a pulse cycle pattern as follows: i. Green pulse, ii. red pulse, iii. Blue pulse, iv. green pulse, v. Red pulse, vi. Blue pulse, vii. Infrared (IR) pulses; viii. (Repeat)

[0083] As can be seen in the examples, infrared segments or special wavelength segments (e.g., 513-545 nm, 565-585 nm, and / or 900-100 nm) can be pulsed at a different rate than the other segments. This may be done to emphasize particular aspects of a scene; the IR data can simply be overlaid with other data in the video output to achieve the desired emphasis. Note that adding an electromagnetic segment on top of the red, green, and blue segments does not necessarily require a serialized system to operate at four times the speed of a full-spectrum, non-serialized system, since not all segments need to be equally represented in the pulse pattern. As can be seen in this embodiment, adding a segment pulse that is less represented in the pulse pattern (infrared in the above example) results in less than a 20% increase in the sensor's cycle rate to accommodate the irregular segment sampling.

[0084] In one embodiment, an electromagnetic section may be emitted that is sensitive to a dye or material used to highlight aspects of a scene. In this embodiment, high resolution may not be required, which may be sufficient to highlight the location of the dye or material. In such an embodiment, the dye-sensitive electromagnetic section may be cycled much less frequently than other sections in the system to contain the highlighted data.

[0085] In various embodiments, the pulse cycle pattern can include any of the following wavelengths in any suitable order, which may be particularly suitable for determining multispectral or hyperspectral image data, or for determining image data based on fluorescent reagent relaxation emissions: i.465±5nm, ii.533±4nm, iii.638±5nm, iv.780±5nm, v.805±5nm, vi.975±5nm, vii. 577±2 nm, or viii.523±4nm.

[0086] The partition cycle may be divided to correspond to or approximate various imaging and video standards. In one embodiment, as best illustrated in Figures 7A-7D below, the partition cycle may include pulses of electromagnetic energy in the red, green, and blue spectrums. In Figure 7A, different light intensities are achieved by adjusting the width or duration of the light pulse within the operating range indicated by the gray vertical dashed lines. In Figure 7B, different light intensities are achieved by adjusting the light power or the power of the electromagnetic emitter, which may be a laser or LED emitter, but keeping the pulse width or duration constant. Figure 7C illustrates the case where both the light power and the light pulse width are adjusted, leading to greater flexibility. The partition cycle may use CMY, IR, and ultraviolet light using an invisible pulse source mixed with a visible pulse source and any other color space needed to generate an image or to approximate a desired video standard, now known or yet to be developed. It should also be understood that the system may be capable of switching between color spaces on the fly to provide the desired image output quality.

[0087] In embodiments using the color space green-blue-green-red (as seen in FIG. 7D ), it may be desirable to pulse the luminance component more frequently than the chrominance components, since users are typically more sensitive to differences in light amplitude than to differences in light color. This principle can be exploited using a monochrome sensor, as illustrated in FIG. 7D . In FIG. 7D , green, which contains the most luminance information, may be pulsed more frequently or with greater intensity in a (GBGRGBGR...) scheme to obtain luminance data. Such a configuration produces a video stream with perceptibly more detail without creating and conveying imperceptible data.

[0088] In one embodiment, duplicating a weaker portion of the pulse can be used to generate an output tailored for weaker pulses. For example, blue laser light is considered weaker relative to the sensitivity of silicon-based pixels and is more difficult to generate compared to red or green light, so it can be pulsed more frequently during a frame cycle to compensate for the weaker light. These additional pulses can be done sequentially over time or by using multiple lasers pulsing simultaneously to create the desired compensation effect. Note that by pulsing during blanking periods (times when the sensor is not reading out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same type and simply accumulates light for the desired output. In another embodiment, the maximum light pulse range can vary between frames. This is shown in Figure 7E, where the light pulses vary between frames. The sensor can be constructed so that different blanking times can be programmed in a repeating pattern of two, three, four, or n frames. In Figure 7E, four different light pulses are illustrated, and pulse 1 can be repeated, for example, after pulse 4, resulting in a pattern of four frames with different blanking times. This technique can be used to place the most powerful section for the smallest blanking time, thus allowing the weakest section to have a wider pulse in one of the following frames without having to increase the readout speed. The reconstructed frame, when composed of many pulsed frames, can still have a regular pattern between frames.

[0089] As can be seen in FIG. 8 , each split spectrum of light can have a different energy value, so the sensor and / or light emitter can be adjusted to compensate for the difference in energy values. At 810, data obtained from a histogram from the previous frame can be analyzed. At 820, the sensor can be adjusted as described below. Additionally, at 830, the emitter can be adjusted. At 840, an image can be obtained from the sensor with an adjusted sampling time, or an image can be obtained with adjusted (increased or decreased) emitted light, or a combination of the above. For example, because red light spectrums are more easily detected by sensors in the system than blue light spectrums, the sensor can be adjusted to be less sensitive during the red segment cycle and more sensitive during the blue segment cycle because the blue segment has a lower quantum efficiency with respect to silicon (best illustrated in FIG. 9 ). Similarly, the emitter can be adjusted to provide adjusted segments (e.g., higher or lower intensity and duration). Furthermore, adjustments can be made at both the sensor and emitter levels. Emitters can also be designed to emit at one specific frequency, or can be modified to emit a specific range of frequencies to broaden the spectrum of the light being emitted, if desired for a particular application.

[0090] Figure 10 shows a schematic diagram of a non-shared 4T pixel. The TX signal is used to transfer the accumulated charge from the photodiode (PPD) to the floating diffusion (FD). The reset signal is used to reset the FD to the reset bus. If the reset and TX signals are "on" simultaneously, the PPD is always reset (each photocharge generated in the PPD is collected directly on the reset bus) and the PPD is always empty. A typical pixel array implementation includes a horizontal reset line that attaches the reset signal for all pixels in a row, and a horizontal TX line that attaches the TX signal for all pixels in a row.

[0091] In one embodiment, the timing of sensor sensitivity adjustment is illustrated, which can be achieved using a global reset mechanism (i.e., a means of firing all pixel array reset signals at the same time) and a global TX mechanism (i.e., a means of firing all pixel array TX signals at the same time). This is shown in Figure 11. In this case, the light pulse is constant in duration and amplitude, but the light integrated within all pixels starts with the global TX "on," transitions to "off," and ends with the light pulse. Therefore, modulation is achieved by moving the falling edge of the global TX pulse.

[0092] Conversely, the emitter can emit red light at a lower intensity than the blue light to produce a properly exposed image (best illustrated in FIG. 12). At 1210, data obtained from a histogram from a previous frame can be analyzed. At 1220, the emitter can be adjusted. At 1230, an image can be obtained from the adjusted emitted light. Additionally, in one embodiment, both the emitter and sensor can be adjusted simultaneously.

[0093] In some embodiments, reconstructing the split-spectrum frame into a full-spectrum frame for later output can be as simple as blending the sensed values ​​for each pixel in the array. Additionally, the blending and mixing of values ​​can be a simple average or can be matched to a predetermined lookup table (LUT) of values ​​for the desired output. In one embodiment of a system using split light spectrum, the sensed values ​​can be post-processed or further refined by an image or secondary processor remote from the sensor and just before being output to a display.

[0094] FIG. 13 illustrates a basic implementation at 1300 of a monochrome ISP and how the ISP chain can be assembled for the purpose of generating an sRGB image sequence from raw sensor data acquired in the presence of an GRGB light pulsing scheme.

[0095] The first stage involves making corrections (see 1302, 1304, and 1306 in FIG. 13) to compensate for any non-idealities in the sensor technology that is best suited to function in the raw data domain (see FIG. 21).

[0096] The next step is to buffer two frames (see 1308 and 1310 in FIG. 13) since each final frame derives data from three raw frames. At 1314, reconstruction of the frame begins by sampling data from the current frame and the two buffered frames (1308 and / or 1310). The reconstruction process results in a full-color frame in linear RGB color space.

[0097] In this embodiment, white balance factors at 1318 and a color correction matrix at 1320 are applied before converting to YCbCr space at 1322 for subsequent edge enhancement at 1324. After edge enhancement at 1324, the image is converted back to linear RGB at 1326 for scaling at 1328, if applicable.

[0098] Finally, a gamma transfer function is applied at 1330 to translate the data into the sRGB domain at 1332.

[0099] FIG. 14 is an example of one embodiment of color fusion hardware. At 1402, the color fusion hardware takes an RGBGRGBGRGBG video data stream and converts it into a parallel RGB video data stream at 1405. The input bit width can be, for example, 12 bits per color. The output width of the example is 36 bits per pixel. Other embodiments may have a different initial bit width and an output width that is three times that number. At 1402, a memory writer block takes the RGBG video stream as its input and writes each frame to its correct frame memory buffer at 1404 (the memory writer triggers the same pulse generator 1410 that operates the laser light source). As illustrated at 1404, writing to memory follows a pattern of red, green 1, blue, green 2, and then starting again with red. At 1406, a memory reader reads three frames simultaneously to construct RGB pixels. Each pixel is three times the bit width of an individual color component. At 1410, the reader also triggers the laser pulse generator. The reader waits until the Red, Green 1, and Blue frames have been written, and then proceeds to read them in parallel, while the writer continues writing Green 2 and starts again with Red. Once Red is complete, the reader begins reading from Blue, Green 2, and Red. This pattern continues indefinitely.

[0100] Referring now to FIGS. 15 and 16, in one embodiment, reconstruction of the RG1BG2RG1BG2 pattern illustrated in FIG. 16 allows for a 60 fps output with a 120 fps input. Each successive frame includes either the red or blue component from the previous frame. In FIG. 16, each color component is available for 8.3 ms, and the resulting reconstructed frame has a duration of 16.67 ms. Generally, for this pulsing scheme, the reconstructed frame has a duration twice that of the incident color frame, as shown in FIG. 15. Other embodiments can employ different pulsing schemes. For example, an embodiment can be based on the timing of each color component or frame (T1) and the reconstructed frame having a duration twice that of the incident color frame (2×T1). Different frames in the sequence can have different frame durations, and the average capture rate can be any multiple of the final frame rate.

[0101] 17-20 illustrate schematic diagrams of color correction methods and hardware for use with split-light systems. In digital imaging, it is common to manipulate values ​​within image data to correct the output to meet user expectations or to emphasize particular aspects of the imaged object. Most commonly, this is done in satellite imagery, which is tuned and adjusted to emphasize one data type over another. Often, satellite-acquired data does not have a controlled light source, i.e., the sun is the light source, so the entire spectrum of available electromagnetic energy is present. In contrast, there are imaging conditions where the light is controlled and even provided by the user. In such situations, calibration of the image data is still desirable, as without calibration, inappropriate emphasis may be given to certain data over others. In systems where the light is user-controlled, it is advantageous to provide light emissions that are known to the user and may be only a portion of the electromagnetic spectrum or multiple portions of the entire electromagnetic spectrum. Calibration remains important to meet user expectations and to identify faults within the system. One method of calibration can be a table of expected values ​​for a given imaging condition that can be compared to data from the sensor. One embodiment can include a color-neutral scene with known values ​​to be output by the imaging device, and the device can be adjusted to meet those known values ​​when it samples the color-neutral scene.

[0102] During use, and upon start-up, the system may sample a color-neutral scene at 1710 (as illustrated in FIG. 17) by running a full cycle of multiple electromagnetic spectrum segments at 1702. At 1704, a table of values ​​1708 may be formed to generate a histogram of the frame. At 1706, the values ​​of the frame may be compared to known or expected values ​​from a color-neutral scene. The imager may then be adjusted at 1712 to meet the desired output. In one embodiment illustrated in FIG. 17, the system may include an image signal processor (ISP) that may adjust the imager to color correct.

[0103] It should be noted that because each split spectrum of light may have a different energy value, the sensor and / or light emitter may be adjusted to compensate for the difference in energy values. For example, in one embodiment, for a silicon-based imager, the sensor's response may be adjusted to be less responsive during the red cycle and more responsive to the blue cycle because the blue light spectrum has a lower quantum efficiency than the red light spectrum. Conversely, the emitter may emit blue light at a higher intensity because the quantum efficiency of blue light is lower than that of red light to produce a properly exposed image.

[0104] In one embodiment illustrated in Figure 18 where light source emissions are provided and controllable by the system, adjustments of such light emissions can be made at 1800 to provide color correction for the image. Adjustments can be made to any aspect of the emitted light, such as amplitude, duration (i.e., on-time), or range within a spectral section. Additionally, in some embodiments, such as shown in Figure 19, both the emitter and sensor can be adjusted simultaneously.

[0105] As can be seen in FIG. 20 , granular adjustments can be made to sensors or emitters in the system to reduce the amount of noise and artifacts in the output image stream or video. While FIG. 20 illustrates a system 2000 in which both the emitter 2006 and the sensor 2008 can be adjusted, imaging devices in which either the emitter or the sensor is adjusted during use or over a portion of use are also contemplated and within the scope of this disclosure. It may be advantageous to adjust only the emitter during a portion of use and only the sensor during another portion of use, while also adjusting both simultaneously during a portion of use. In any of the above embodiments, improved image quality can be obtained by limiting the overall adjustments the system can make between frame cycles. In other words, one embodiment can limit the emitter to only be able to adjust a portion of its operating range at any time between frames. Similarly, the sensor can be limited to only be able to adjust a portion of its operating range at any time between frames. Furthermore, one embodiment can limit both the emitter and the sensor to only be able to adjust a portion of their respective operating ranges together at any time between frames.

[0106] In one exemplary embodiment, adjustments to a portion of a component in the system may be performed, for example, by approximately 0.1 dB of the component's operating range to compensate for the exposure of a previous frame. Note that this 0.1 dB is merely one example; in other embodiments, possible adjustments to components may be any portion of their respective operating ranges. The components of the system may be altered by intensity or duration adjustments, typically governed by the number of bits (resolution) output by the component. Component resolution may typically range from approximately 10 to 24 bits, but should not be limited to this range, as it is intended to include resolutions of components that are currently available as well as those yet to be developed. For example, if, after the first frame, it is determined that the scene is too blue when observed, the emitter may be adjusted to reduce the amplitude or duration of the blue light pulse during the system's blue cycle by a portion adjustment, such as approximately 0.1 dB, as discussed above.

[0107] In this exemplary embodiment, the system limits itself to adjusting 0.1 dB of the operating range per system cycle, although more than 10 percent may be necessary. Therefore, the blue light can then be adjusted again as needed during the next system cycle. Finer adjustments between cycles may have the effect of damping the output image, reducing noise and artifacts when operating the emitter and sensor at their operating extremes. It may be determined that any small amount of component adjustment range of operation can be used as a limiting factor, or that a particular embodiment of the system may include components that can be adjusted over the entire operating range.

[0108] Additionally, optical black areas of any image sensor can be used to assist in image correction and noise removal. In one embodiment, values ​​read out from the optical black areas can be compared to values ​​in the active pixel areas of the sensor to establish a reference point for use in processing image data. Figure 21 illustrates a type of sensor correction process that can be employed in a color-pulsed system. CMOS image sensors typically have several non-idealities that adversely affect image quality, especially in low light. The main ones are fixed pattern noise and line noise. Fixed pattern noise (FPN) is the variation in the offset of the sensing elements. Typically, most FPN is pixel-to-pixel variation caused by random fluctuations in dark current between photodiodes, among other sources. This appears very unnatural to the viewer. Even more egregious is column FPN, which is caused by the offset of the readout chain associated with a particular pixel column. This results in the perception of vertical stripes in the image.

[0109] Full control of the illumination has the advantage that an entire frame of dark data can be periodically acquired and used to correct for pixel and column offsets. In the illustrated embodiment, a single frame buffer can be used to perform a running average of the entire frame without illumination, for example, using a simple exponential smoothing method. This dark average frame is subtracted from all illuminated frames during normal operation.

[0110] Line noise is the stochastic time variation of the offset of pixels within each row. Because it is transient, the correction must be calculated anew for each line and each frame. For this purpose, there are typically a number of optically blind (OB) pixels in each row in the array, which must be sampled first to estimate the line offset before sampling the light-sensitive pixels. Then, during the line noise correction process, the line offset is simply subtracted.

[0111] In the embodiment of FIG. 21, there are other corrections related to acquiring data in the proper order, monitoring and controlling voltage offsets in the analog domain (black clamp), and identifying / correcting individual bad pixels.

[0112] 22 and 23 illustrate schematics of a method and hardware for increasing dynamic range in a closed or limited-light environment. In one embodiment, exposure inputs can be input at different levels over time and combined to generate a larger dynamic range. As can be seen in FIG. 22 , the imaging system can be cycled by cycling at a first intensity for a first cycle at 2202, then subsequently cycling at a second intensity for a second cycle at 2204, and then combining the first and second cycles into a single frame at 2206, thereby achieving a larger dynamic range. A larger dynamic range can be particularly desirable because the spatial environment in which the imaging device is used is limited. In limited spatial environments that are poorly lit or dark except for the light provided by the light source, and when the light source is close to the light emitter, exposure has an exponential relationship with distance. For example, objects near the light source and optical aperture of the imaging device tend to be overexposed, while objects farther away tend to be underexposed because there is little, if any, ambient light.

[0113] 23, in 2300, the cycle of a system having emission of electromagnetic energy in multiple segments can be cycled continuously according to segments of the electromagnetic spectrum. For example, in one embodiment where an emitter emits lasers in a unique red segment, a unique blue segment, and a unique green segment, the two sets of cycle data to be combined can be in the form: i.2302 red with intensity 1, ii.2304 red with intensity 2, iii.2302 with blue intensity 1, iv.2304 is blue with intensity 2, v.2302 green with intensity 1, vi.2304 green intensity 2

[0114] Alternatively, the system can be cycled in the following manner. i.2302 red with intensity 1, ii.2302 with blue intensity 1, iii.2302 with green intensity 1, iv.2304 is red with intensity 2, v.2304 blue with intensity 2, vi.2304 green intensity 2

[0115] In one such embodiment, a first image may be derived from Intensity 1 values ​​and a second image may be derived from Intensity 2 values, which may then be combined or processed at 2310 as a complete image data set rather than their constituent parts.

[0116] It is contemplated within the scope of this disclosure that any number of emission segments may be used in any order. As seen in Figure 23, "n" is used as a variable to represent any number of electromagnetic segments, and "m" is used to represent any level of intensity for the "n" segments. Such a system may be cycled as follows: n of intensity m in i.2306, ii. n+1 of intensity m+1, iii. n+2 with intensity m+2; iv.2308 n+i with intensity m+j.

[0117] Thus, any pattern of serialized cycles can be used to produce the desired image correction, where "i" and "j" are additional values ​​within the operating range of the imaging system.

[0118] Digital color cameras incorporate an image processing stage to maximize the fidelity of color reproduction, which is achieved by a 3x3 matrix known as the Color Correction Matrix (CCM).

[0119]

number

[0120] The terms in the CCM are adjusted using a set of reference colors (e.g., from the Macbeth chart) to provide the best overall match to the sRGB standard color space. The diagonal terms, a, e, and i, are essentially white balance gains. However, white balance is typically applied separately, and the sum of the horizontal rows is constrained to be unity so that no net gain is applied by the CCM itself. The off-diagonal terms effectively address color crosstalk within the input channels. Therefore, Bayer sensors have higher off-diagonal gains than 3-chip cameras because color filer arrays have more response overlap between channels.

[0121] There is a signal-to-noise ratio penalty to color correction that depends on the amplitude of the off-diagonal terms. A hypothetical sensor with channels perfectly matched to the sRGB components has the identity matrix CCM:

[0122]

number

[0123] The signal to noise ratio estimated in the green channel is, for a perfect white light signal of 10,000 e / pixel (neglecting readout noise) in this case, as follows:

[0124]

number

[0125] Any deviation from this will reduce the SNR. For example, consider the CCM with non-anomalous values ​​for a Bayer CMOS sensor:

[0126]

number

[0127] In this case, the SNR for green is:

[0128]

number

[0129] Figure 24 shows the overall SNR simulation results using D65 illumination for a typical Bayer sensor CCM when using the identity matrix compared to the adjusted CCM. The SNR evaluated for the luminance component is approximately 6 dB worse as a result of the color correction.

[0130] Because the system described in this disclosure uses monochromatic illumination of multiple discrete wavelengths, there is essentially no color crosstalk. The Xs in Figure 25 indicate the locations of the three wavelengths available via the laser diode source (465, 532, and 639 nm) compared to the sRGB range indicated by the triangles.

[0131] In this case, the off-diagonal terms of the CCM are significantly reduced compared to the Bayer sensor, providing a significant SNR advantage.

[0132] 26 illustrates an imaging system with increased dynamic range provided by the pixel configuration of the image sensor's pixel array. As can be seen, adjacent pixels 2602 and 2604 can be set to different sensitivities, such that each cycle contains data generated by pixels that are more and less sensitive to each other. Because multiple sensitivities can be recorded in a single cycle array, the dynamic range can be increased when recorded in parallel, as opposed to the time-dependent series nature of other embodiments.

[0133] In one embodiment, the array can include rows of pixels that can be arranged within the rows based on their sensitivity. In one embodiment, pixels of different sensitivity can be staggered within rows or columns relative to their nearest neighbors across the array based on their sensitivity to form a checkerboard pattern. This can be achieved through any shared pixel circuit arrangement or with any independent pixel circuit arrangement.

[0134] A wide dynamic range can be achieved by having multiple global TXs, each firing only on a different set of pixels. For example, in global mode, the global TX1 signal lights up the first set of pixels, the global TX2 signal lights up the second set of pixels, ..., the global TXn signal lights up the nth set of pixels.

[0135] Based on FIG. 11, FIG. 27A shows an example of timing for two different pixel sensitivities (dual pixel sensitivity) in a pixel array. In this case, a global TX1 signal lights up half of the pixels in the array, and global TX2 lights the other half. Because global TX1 and global TX2 have different "on" to "off" edge positions, the integrated light differs between the TX1 and TX2 pixels. FIG. 27B shows a different embodiment of timing for dual pixel sensitivity. In this case, the light pulse is modulated twice (pulse duration and / or amplitude). The TX1 pixel integrates the P1 pulse, and the TX2 pixel integrates the P1+P2 pulse. Separating the global TX signals can be done in a number of ways. Below are some examples: i. Distinguish TX lines from each row, ii. Send multiple TX lines per row, each corresponding to a different set of pixels.

[0136] In one implementation, a means for providing wide dynamic range video is described that utilizes the color pulsing system described in this disclosure. The basis for this means is having multiple flavors of pixels, or pixels that can be adjusted differently, within the same monochrome array, capable of integrating incident light for different durations within the same frame. One example of pixel arrangement within such a sensor's array is a uniform checkerboard pattern throughout, with two independently variable integration times. In such a case, both red and blue information can be provided within the same frame. In fact, since the two integration times can be adjusted on a frame-by-frame basis, this can be done simultaneously to extend the dynamic range of the green frame, if necessary. The advantage is that color motion artifacts are less of an issue when all data is derived from two frames versus three. Naturally, there is a subsequent loss of spatial resolution for the red and blue data, but this has less of an impact on image quality compared to green, since the green data accounts for the luminance component.

[0137] An inherent property of monochrome wide-dynamic range (WDR) arrays is that pixels with long integration times must integrate a superset of the light seen by pixels with short integration times. This is desirable for standard wide-dynamic range operation in the green frame. For red and blue frames, this means that pulsing must be controlled in conjunction with the exposure period, providing blue light from the beginning of the long exposure, for example, and switching to red once the short exposure pixels are turned on (both pixel types have their charge transferred simultaneously).

[0138] The color blending step separates the two flavors of pixels into two buffers. Empty pixels are then filled, for example using linear interpolation. At this point, one buffer contains the full image of blue data, and the other contains the full image of red + blue. The blue buffer can be subtracted from the second buffer to get pure red data.

[0139] Figures 28A-28C illustrate the use of white light emission that is pulsed and / or synchronized with a corresponding color sensor, or held constant. As can be seen in Figure 28A, the white light emitter can be configured to emit a beam of light during the blanking period of the corresponding sensor to provide a controlled light source within a controlled lighting environment. As can be seen in Figure 28A, the light source can emit a beam with a constant amplitude and vary the duration of the pulse, or, as illustrated in Figure 28B, the pulse can be kept constant while varying the amplitude to achieve properly exposed data. Figure 28C illustrates a graphical representation of a constant light source that can be adjusted with varying current controlled and synchronized by a sensor.

[0140] In one embodiment, white or multi-spectral light can be emitted as pulses, as desired, to provide data for use within the system (best illustrated in Figures 28A-28C). White light emission in combination with sections of the electromagnetic spectrum can be useful for highlighting and de-emphasizing particular aspects within a scene. Such an embodiment can use the following pulsing patterns: i. Green pulse, ii. red pulse, iii. Blue pulse, iv. green pulse, v. Red pulse, vi. Blue pulse, vii. White light (multispectral) pulses, viii. (Repeat)

[0141] Any system that uses an image sensor cycle that is at least twice as fast as the white light cycle is intended to fall within the scope of this disclosure. It will be understood that any combination of sections of the electromagnetic spectrum, whether from the visible or invisible spectrum of the total electromagnetic spectrum, is contemplated herein.

[0142] 29A and 29B illustrate perspective and side views, respectively, of one implementation of a monolithic sensor 2900 having multiple pixel arrays for generating a three-dimensional image in accordance with the teachings and principles of the present disclosure. Such an implementation may be desirable for three-dimensional image capture, where the two pixel arrays 2902 and 2904 may be offset during use. In another implementation, the first pixel array 2902 and the second pixel array 2904 may be dedicated to receiving electromagnetic radiation in a predetermined range of wavelengths, with the first pixel array dedicated to a different range of wavelengths than the second pixel array.

[0143] 30A and 30B illustrate perspective and side views, respectively, of one implementation of an image sensor 3000 constructed on multiple substrates. As illustrated, a plurality of pixel columns 3004 forming a pixel array are disposed on a first substrate 3002, and a plurality of circuit columns 3008 are disposed on a second substrate 3006. The figures also illustrate electrical connections and communications between one pixel column and its associated or corresponding circuit columns. In one implementation, the pixel array of an image sensor that might otherwise be fabricated with the pixel array and support circuitry on a single monolithic substrate / chip may be separated from all or most of the support circuitry. The present disclosure may use at least two substrates / chips stacked together using three-dimensional stacking techniques. The first of the two substrates / chips 3002 may be processed using an imaging CMOS process. The first substrate / chip 3002 may consist of either exclusively a pixel array or a pixel array surrounded by limited circuitry. The second or subsequent substrate / chip 3006 can be processed using any process and does not have to be an image CMOS process. The second substrate / chip 3006 can be a high-density digital process to integrate a variety and many functions into a very limited space or area on the substrate / chip, or a mixed-mode or analog process, for example to integrate precise analog functions, or an RF process to implement wireless capabilities, or a Micro-Electro-Mechanical Systems (MEMS) process to integrate MEMS devices, but is not limited to this. The image CMOS substrate / chip 3002 can be stacked with the second or subsequent substrate / chip 3006 using any three-dimensional technique. The second substrate / chip 3006 can support most or most of the circuitry that would otherwise be implemented in the first image CMOS chip 3002 as peripheral circuitry (if implemented on a monolithic substrate / chip), thus increasing the overall system area while keeping the pixel array size constant and optimized to the greatest extent possible.Electrical connections between the two substrates / chips can be made through interconnects 3003 and 3005, which can be wirebonds, bumps and / or TSVs (Through Silicon Vias).

[0144] 31A and 31B illustrate perspective and side views, respectively, of one implementation of an imaging sensor 3100 having multiple pixel arrays for generating a three-dimensional image. The three-dimensional image sensor can be constructed on multiple substrates and can include multiple pixel arrays and other associated circuitry, with multiple pixel columns 3104a forming a first pixel array and multiple pixel columns 3104b forming a second pixel array disposed on corresponding substrates 3102a and 3102b, respectively, and multiple circuit columns 3108a and 3108b disposed on a separate substrate 3106. Also illustrated are electrical connections and communications between the pixel columns and associated or corresponding circuit columns.

[0145] It will be appreciated that the teachings and principles of the present disclosure may be used in a reusable device platform, a limited-use device platform, a reposable-use device platform, or a single-use / disposable device platform without departing from the scope of the present disclosure. It will be appreciated that in a reusable device platform, the end user is responsible for cleaning and sterilizing the device. In a limited-use device platform, the device can be used a specified number of times before becoming inoperable. A typical new device is supplied sterile and requires cleaning and sterilization by the end user before further use. In a reposable-use device platform, a third party can reprocess (e.g., clean, package, and sterilize) the single-use device for further use at a lower cost than a new unit. In a single-use / disposable device platform, the device is provided sterile to the operating room and is used only once and then discarded.

[0146] One embodiment of the emitter can employ the use of mechanical shutters and filters to create pulsed color light. As illustrated in FIG. 32, an alternative method for generating pulsed color light uses a white light source, mechanical color filters, and a shutter system 3200. The wheel can include a pattern of translucent color filter windows and opaque sections for shuttering. The opaque sections do not allow light to pass through, creating dark periods during which sensor readout occurs. The white light source can be based on any technology: laser, LED, xenon, halogen, metal halide, or others. The white light can be projected through a series of color filters 3207, 3209, and 3211 in a desired pattern of colored light pulses. In one embodiment, the pattern can be red filter 3207, green filter 3209, blue filter 3211, green filter 3209. The filter and shutter system 3200 can be arranged on a wheel that rotates at the required frequency to synchronize with the sensor, such that knowledge of the arch length and rotational speed of the system of mechanical color filters 3207, 3209, and 3211 and shutter 3205 provides timing information for the operation of the corresponding monochrome image sensor.

[0147] The embodiment illustrated in FIG. 33 can include a pattern of only semi-transparent color filters 3307, 3309, and 3311 on the filter wheel 3300. Different shutters can be used in this configuration. The shutters can be mechanical and dynamically adjust the "pulse" duration by varying their size. Alternatively, the shutters can be electronic and integrated into the sensor design. The motor that rotates the filter wheel 3300 needs to communicate with or be controlled in conjunction with the sensor, so that knowledge of the arc length and rotational speed of the mechanical color filter 3307, 3309, and 3311 system provides timing information for the operation of the corresponding monochrome image sensor. The control system needs to know the appropriate color filter for each frame captured by the sensor so that a full-color image can be properly reconstructed within the ISP. While an RGBG color pattern is shown, other colors and / or patterns can be used if advantageous. The relative sizes of the color sections are shown as equal, but can be adjusted if advantageous. The mechanical structure of the filter is shown as a rotating and moving circle, but it can be a rectangle with linear movement, or a different shape with a different movement pattern.

[0148] As illustrated in FIG. 34, an embodiment for pulsing colored light can consist of a mechanical wheel or barrel holding the electronics and heat sinks for red, green, blue, or white LEDs. The LEDs are spaced a distance related to the rotational or twisting speed of the barrel or wheel to allow for timing of the light pulsation consistent with other embodiments in this patent. The wheel or barrel is rotated using an electric motor and a mechanical bracket that attaches the wheel or barrel to the electric motor. The motor is controlled using a microcontroller, FPGA, DSP, or other programmable device that includes a control algorithm for appropriate timing as described in this patent. On one side is a mechanical opening that is optically coupled to the optical fiber to transport the fiber to the end of the scope in the manner described in this patent. This connection can also have a mechanical opening that can be opened and closed to control the amount of light allowed down the fiber optic cable. This is a mechanical shuttering device; alternatively, an electronic shutter designed into a CMOS or CCD-type sensor can be used. This device is difficult to control and calibrate during manufacturing, but it is another way to obtain pulsed light in this system.

[0149] Figure 35 illustrates one embodiment of an emitter 3502 that includes a linear filter 3504 and shutter mechanism to provide pulsed electromagnetic radiation. The linear filter 3504 and shutter mechanism move horizontally at the required frequency to filter out the appropriate wavelength of light.

[0150] Figure 36 illustrates one embodiment of an emitter 3602 that includes a prism filter 3604 and shutter mechanism to provide pulsed electromagnetic radiation. The prism filter 3604 filters the light and delivers an output that may include a shutter. The prism filter 3604 shifts at the required frequency to provide the correct color output pattern.

[0151] Additionally, the teachings and principles of the present disclosure may include any and all wavelengths of electromagnetic energy, including the visible and non-visible spectrum, such as infrared (IR), ultraviolet (UV), and x-rays.

[0152] FIG. 37 is a schematic diagram illustrating a system 3700 for providing illumination in a low-light environment, such as for endoscopic imaging. The system 3700 can be used in combination with any of the systems, methods, or devices disclosed herein. The system 3700 includes a light source 3702, a controller 3704, a jumper waveguide 3706, a waveguide connector 3708, a lumen waveguide 3710, a lumen 3712, and an image sensor 3714 with associated optical components (such as a lens). The light source 3702 generates light that travels through the jumper waveguide 3706 and the lumen waveguide 3710 to illuminate a scene at the distal end of the lumen 3712. The light source 3700 can be used to emit electromagnetic energy of any wavelength, including visible, infrared, ultraviolet, and other wavelengths. The lumen 3712 can be inserted into a patient's body for imaging, such as during a procedure or examination. Light is output as illustrated by dashed line 3716. The scene illuminated by the light can be captured using image sensor 3714 and displayed to a physician or some other medical personnel. To control when illumination is provided to the scene, controller 3704 can provide control signals to light source 3702. In one embodiment, light source 3702 and controller 3704 are located in a camera control unit (CCU) or external console to which the endoscope is connected. If image sensor 3714 includes a CMOS sensor, light can be periodically provided to the scene in a series of illumination pulses during the readout period of image sensor 3714, during what is known as the blanking period. Thus, the light can be pulsed in a controlled manner to avoid overlapping with the readout period of image pixels in the pixel array of image sensor 3714.

[0153] In one embodiment, the lumen waveguide 3710 includes one or more optical fibers. The optical fibers can be made of a low-cost material, such as plastic, to allow for disposal of the lumen waveguide 3710 and / or other portions of the endoscope. In one embodiment, a single glass fiber having a diameter of 500 micrometers can be used. The jumper waveguide 3706 can be permanently attached to the light source 3702. For example, the jumper waveguide 3706 can receive light from an emitter in the light source 3702 and provide that light to the lumen waveguide 3710 at the connector 3708. In one embodiment, the jumper waveguide 106 can include one or more glass fibers. The jumper waveguide can include any other type of waveguide for guiding light to the lumen waveguide 3710. Connector 3708 selectively couples jumper waveguide 3706 to lumen waveguide 3710 and allows light in jumper waveguide 3706 to pass to lumen waveguide 3710. In one embodiment, lumen waveguide 3710 can be directly coupled to the light source without any intervening jumper waveguide 3706.

[0154] 38-40 are schematic block diagrams illustrating a light source 3800 having multiple emitters. With reference to FIG. 38, the emitters include a first emitter 3802, a second emitter 3804, and a third emitter 3806. As discussed further below, additional emitters may be included. The emitters 3802, 3804, and 3806 may include one or more laser emitters that emit light having different wavelengths. For example, the first emitter 3802 may emit a wavelength corresponding to a blue laser, the second emitter 3804 may emit a wavelength corresponding to a green laser, and the third emitter 3806 may emit a wavelength corresponding to a red laser. For example, the first emitter 3802 may include one or more blue lasers, the second emitter 3804 may include one or more green lasers, and the third emitter 3806 may include one or more red lasers. Emitters 3802, 3804, 3806 emit laser beams toward collection area 3808, which may be the location of a waveguide, lens, or other optical component for collecting and / or providing light to a waveguide such as jumper waveguide 3706 or lumen waveguide 3710 in FIG. 37.

[0155] In one implementation, in which the patient has been administered a reagent or dye to aid in the identification of particular tissues, structures, chemical reactions, biological processes, etc., emitters 3802, 3804, and 3806 can emit wavelength(s) that cause the reagent or dye to fluoresce. Such wavelength(s) can be determined based on the reagent or dye administered to the patient. In such an embodiment, the emitters may need to be very precise to emit the desired wavelength(s) to cause or activate the particular reagent or dye to fluoresce.

[0156] In the embodiment of FIG. 38 , emitters 3802, 3804, 3806 each deliver laser light to collection region 3808 at different angles. Variations in angle can result in variations when electromagnetic energy is disposed in an output waveguide. For example, if the light immediately enters a fiber bundle (glass or plastic) at collection region 3808, variations in angle can cause different amounts of light to enter different fibers. For example, this angle can result in variations in intensity across collection region 3808. Furthermore, because light from different emitters cannot mix uniformly, some fibers may receive different amounts of light of different colors. Variations in the color or intensity of light in different fibers can lead to suboptimal illumination of a scene. For example, variations in the delivered light or light intensity can occur in the scene and in the captured image.

[0157] In one embodiment, intervening optical elements can be placed between the fiber bundles and the emitters 3802, 3804, 3806 to mix light of different colors (wavelengths) before entering the fibers or other waveguides. Exemplary intervening optical elements include a diffuser, a mixing rod, one or more lenses, or other optical components that mix light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in a fiber bundle may have the same color. This mixing may result in the same color for each fiber, but in some embodiments, may still result in different total intensities being delivered to different fibers. In one embodiment, intervening optical elements can also diffuse or uniform the light over the collection area (e.g., the light can be diffused with a top-hat profile) so that each fiber carries the same total amount of light. A diffuser or mixing rod may result in light loss.

[0158] 38 as a physical component, the collection region 3808 may simply be the region through which light from the emitters 3802, 3804, and 3806 is delivered. In some cases, the collection region 3808 may include optical components such as a diffuser, mixing rod, lens, or any other intervening optical component between the emitters 3802, 3804, 3806 and the output waveguide.

[0159] 39 illustrates one embodiment of a light source 3800 having emitters 3802, 3804, and 3806 that provide light to a collection region 3808 at the same or substantially the same angle. The light is provided at a substantially perpendicular angle to the collection region 3808. The light source 3800 includes multiple dichroic mirrors, including a first dichroic mirror 3902, a second dichroic mirror 3904, and a third dichroic mirror 3906. The dichroic mirrors 3902, 3904, and 3906 include mirrors that reflect light of a first wavelength but transmit (i.e., are transmissive) light of a second wavelength. For example, the third dichroic mirror 3906 may reflect blue laser light provided by the third emitter while transmitting red and green light provided by the first and second emitters 3802 and 3804, respectively. The second dichroic mirror 3904 can transmit red light from the first emitter 3802 but reflect green light from the second emitter 3804. If other colors or wavelengths are included, the dichroic mirrors can be selected to reflect light corresponding to at least one emitter and transmit light corresponding to other emitters. For example, the third dichroic mirror 3906 reflects light from the third emitter 3806 but reflects that light to "rear" emitters, such as the first emitter 3802 and the second emitter 3804. In embodiments where there are tens or hundreds of emitters, each dichroic mirror can reflect light to a corresponding emitter or emitters in front of it while transmitting light to emitters behind it. This allows tens or hundreds of emitters to emit electromagnetic energy at substantially the same angle into the collection region 3808.

[0160] The dichroic mirror transmits or allows other wavelengths to pass, so that each of the wavelengths can arrive at the collection region 3808 from the same angle and / or at the same center point or focal point. Providing light from the same angle and / or the same focal point / center point can significantly improve light reception and color mixing in the collection region 3808. For example, a particular fiber can receive different colors in the same proportions as transmitted / reflected by the emitters 3802, 3804, 3806 and mirrors 3902, 3904, 3906. Light mixing can be significantly improved in the collection region compared to the embodiment of FIG. 38 . In one embodiment, any of the optical components discussed herein can be used in the collection region 3808 to collect light before providing it to the fiber or fiber bundle.

[0161] FIG. 40 also illustrates an embodiment of a light source 3800 having emitters 3802, 3804, and 3806 that provide light to a collection region 3808 at the same or substantially the same angle. However, the light incident on the collection region 3808 is offset from normal. Angle 4002 indicates the angular offset from normal. In one embodiment, the laser emitters 3802, 3804, and 3806 may have a cross-sectional intensity profile that is Gaussian. As previously discussed, improved distribution of optical energy among the fibers may be achieved by creating a flatter or top-hat shaped intensity profile. In one embodiment, as angle 4002 increases, the intensity across the collection region 3808 approaches a top-hat profile. For example, even a non-flat output beam can be made to approximate a top-hat profile by increasing angle 4002 until the profile is sufficiently flat.

[0162] A top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intervening optical components between the emitters 3802, 3804, 3806 and the output waveguide, fiber, or fiber optic bundle.

[0163] 41 is a schematic diagram illustrating a single optical fiber 4102 outputting through a diffuser 4104 at its output. In one embodiment, the optical fiber 4102 has a diameter of 500 micrometers and a numerical aperture of 0.65, and can emit a light cone 4106 of approximately 70 or 80 degrees without the diffuser 4104. With the diffuser 4104, the light cone 4106 can have an angle of approximately 110 or 120 degrees. The light cone 4106 can be the majority of where all light travels and is uniformly distributed. The diffuser 4104 can allow for a more uniform distribution of electromagnetic energy in the scene observed by the image sensor.

[0164] In one embodiment, the lumen waveguide 4102 can comprise a single plastic or glass optical fiber of approximately 500 micrometers. While plastic fiber can be low cost, its width can limit the fiber's ability to carry a sufficient amount of light to the scene due to couplers, diffusers, or other losses. For example, a smaller fiber may not be able to carry as much light or power as a larger fiber. The lumen waveguide 3710 can comprise a single or multiple optical fibers. The lumen waveguide 3702 can receive light from a light source directly or through a jumper waveguide (see, for example, jumper waveguide 3706 in FIG. 37). A diffuser can be used to widen the light output 3706 for a desired field of view of the image sensor 3714 or other optical component.

[0165] Although three emitters are shown in Figures 38-40, in some embodiments, anywhere from one to hundreds or more emitters can be used. The emitters can have different wavelengths or spectrums of light that they emit, which can be used to continuously cover desired portions of the electromagnetic spectrum (e.g., the visible spectrum and the infrared and ultraviolet spectrum).

[0166] In one embodiment, a light source with multiple emitters can be used for multispectral or hyperspectral imaging in low-light environments. For example, different chemicals, materials, or tissues may have different responses to different colors or wavelengths of electromagnetic energy. Some tissues have their unique spectral signatures (how tissues react or vary in reflecting wavelengths of electromagnetic radiation). In one embodiment, specific types of tissue can be detected based on how they respond to specific wavelengths or combinations of wavelengths. For example, vascular tissue can absorb and reflect electromagnetic energy of different wavelengths or spectra in unique ways, distinguishing it from muscle, fat, bone, nerves, ureters, or other tissues or materials in the body. Furthermore, specific types of muscle or other types of tissue can be distinguished based on their spectral response. Disease states of tissues can also be determined based on spectral information. See U.S. Patent No. 8,289,503. See also U.S. Patent No. 8,158,957.

[0167] In one embodiment, fluorescence image data and / or multispectral or hyperspectral image data can be acquired using one or more filters to filter out all light or electromagnetic energy except for those of desired wavelengths or spectrums. Figure 42 is a block diagram illustrating a filter 4202 for filtering out unwanted wavelengths before light 4208 (or other electromagnetic radiation) encounters an imaging sensor 4204 or other imaging medium (e.g., film). In one embodiment, white light 4208 passes through filter 4202 and the filtered light 4210 passes through a lens 4206 that focuses it onto the imaging sensor 4204 for image capture and readout. The filter can be located anywhere in the system or can be an attachment to the lens 4206 or image sensor 4204.

[0168] In low light environments, the light 4208 can include white light emitted by an emitter in a low light environment. The filter 4202 can be selected for the desired examination. For example, if it is desired to detect or highlight a particular tissue, the filter 4202 can be selected to pass wavelengths corresponding to the spectral response of the particular tissue or the fluorescent emission of a particular reagent. The image sensor 4204, which can include a monochrome image sensor, can generate an image. Pixels of the captured image that are above or below a threshold can then be characterized as corresponding to the particular tissue. This data can then be used to generate an image that indicates the location of the particular tissue.

[0169] In another embodiment, fluorescent dyes or reagents can be used to image specific tissue types, pathways, etc. within the body. For example, a fluorescent dye can be administered to a patient, and then an image of the dye can be captured. In one embodiment, causing the dye to fluoresce can be triggered using electromagnetic energy of a specific wavelength. For example, the dye can fluoresce only in the presence of electromagnetic energy.

[0170] However, both filters and fluorescent dyes significantly limit the examination. For example, when using filters, the desired spectral response that can be detected, and therefore the materials or tissues that can be detected, are limited by the available filters. Furthermore, filters may need to be replaced or exchanged. With dyes, the dye must be administered before imaging, and there may be competition between the administration of different dyes for different purposes during the same examination. Therefore, examinations using filters and dyes may take a long time and may require many different examinations to obtain the desired information.

[0171] In one embodiment, multispectral or hyperspectral imaging in low-light environments can be achieved using a monochrome image sensor and emitters that emit electromagnetic energy at multiple different wavelengths or spectrums. In one embodiment, a light source or other electromagnetic source (such as light source 3800 in any of Figures 38-40) can include multiple emitters to cover the desired spectrum.

[0172] Figure 43 illustrates a portion of the electromagnetic spectrum 4300 divided into 20 different subspectra. The number of subspectra is exemplary only. In at least one embodiment, the spectrum 4300 can be divided into hundreds of subspectra, each having a small wavelength band. The spectrum can range from the infrared spectrum 4302, through the visible spectrum 4304, to the ultraviolet spectrum 4306. The subspectra each have wavelength bands 4308 that cover a portion of the spectrum 4300. Each wavelength band can be defined by an upper wavelength and a lower wavelength.

[0173] In one embodiment, at least one emitter (e.g., a laser emitter) can be included in a light source (e.g., light sources 3702, 3800 in Figures 37-40) for each subspectrum to provide complete and continuous coverage of the entire spectrum 4300. For example, a light source for providing the illustrated subspectral coverage can include at least 20 different emitters, at least one for each subspectrum. In one embodiment, each emitter can cover a spectrum covering 40 nanometers. For example, one emitter can emit light within a wavelength range from 500 nm to 540 nm, while another emitter can emit light within a wavelength range from 540 nm to 580 nm. In other embodiments, the emitters can cover wavelength ranges of other sizes depending on the type of emitter available or the imaging needs. For example, the multiple emitters may include a first emitter covering a wavelength range of 500-540 nm, a second emitter covering a wavelength range of 540-640 nm, and a third emitter covering a wavelength range of 640-650 nm. Each emitter may cover a different segment of the electromagnetic spectrum spanning the far-infrared, mid-infrared, near-infrared, visible, near-ultraviolet, and / or extreme ultraviolet. In some cases, multiple emitters of the same type or wavelength may be included to provide sufficient power for imaging. The number of emitters required for a particular wavelength range may depend on the sensitivity of the monochromatic sensor to that wavelength range and / or the power output capabilities of the emitters in that wavelength range.

[0174] The width and coverage of the wavelength bands provided by the emitters can be selected to provide any desired combination of spectra. For example, continuous coverage of the spectrum using very narrow wavelength bands (e.g., 10 nm or less) can enable highly selective hyperspectral imaging. Because the wavelengths are derived from emitters that can be selectively activated, maximum flexibility can be achieved in determining the spectral response of materials during an examination. Thus, more information about spectral response that might require multiple examinations, be delayed by the administration of dyes or stains, etc. can be achieved in less time and with a single examination. In one embodiment, the system can capture hyperspectral image data and process the data to identify what type of tissue is present at each pixel.

[0175] Figure 44 is a schematic diagram illustrating a timing diagram 4400 for emission and readout to generate a multispectral or hyperspectral image, according to one embodiment. The solid lines represent readouts (peaks 4402) and blanking periods (valleys) for capturing a series of frames 4404-4414. The series of frames 4404-4414 may include a repeating series of frames that can be used to generate hyperspectral data for a video feed. The series of frames includes a first frame 404, a second frame 4406, a third frame 4408, a fourth frame 4410, a fifth frame 4412, and an nth frame 4426.

[0176] In one embodiment, each frame is generated based on at least one pulse of electromagnetic energy. The pulse of electromagnetic energy is reflected, detected by an image sensor, and then read out in a subsequent readout (4402). Thus, each blanking period and readout results in an image frame of a particular range of electromagnetic energy. For example, a first frame 4404 can be generated based on the spectrum of a first one or more pulses 4416, a second frame 4406 can be generated based on the spectrum of a second one or more pulses 4418, a third frame 4408 can be generated based on the spectrum of a third one or more pulses 4420, a fourth frame 4410 can be generated based on the spectrum of a fourth one or more pulses 4422, a fifth frame 4412 can be generated based on the spectrum of a fifth one or more pulses 4424, and an nth frame 4426 can be generated based on the spectrum of an nth one or more pulses 4426.

[0177] Pulses 4416-4426 can include energy from a single emitter or a combination of two or more emitters. For example, the spectra included within a single readout period or multiple frames 4404-4414 can be selected for the desired examination or detection of a particular tissue or condition. According to one embodiment, one or more pulses can include visible spectrum light to generate a color or black-and-white image, while one or more additional pulses are used to obtain a spectral response and classify the tissue type. For example, pulse 4416 can include red light, pulse 4418 can include blue light, and pulse 4420 can include green light, while the remaining pulses 4422-4426 can include wavelengths and spectra for detecting a particular tissue type. As a further example, the pulses of a single readout period can include spectra (e.g., different segments of the electromagnetic spectrum) generated from multiple different emitters that can be used to detect a particular tissue type. For example, if a combination of wavelengths results in a pixel having a value above or below a threshold, the pixel can be classified as corresponding to a particular type of tissue. Each frame can be used to further refine the type of tissue present at that pixel (e.g., each pixel in an image) to provide a very specific classification of tissue and / or tissue state (diseased / healthy) based on spectral response.

[0178] The multiple frames 4404-4414 are shown having readout periods of various lengths and pulses with different lengths or intensities, and the blanking periods, pulse lengths or intensities, etc. can be selected based on the sensitivity of the monochromatic sensor to a particular wavelength, the power output capabilities of the emitter(s), and / or the carrying capabilities of the waveguide.

[0179] Hyperspectral images or hyperspectral image data acquired in the manner illustrated in Figure 44 can provide multiple frames, each based on a different spectrum or combination of spectra. In some cases, tens or hundreds of different frames can be acquired. In other cases, such as for video streams, some frames may be limited to provide a viewable frame rate. Because different spectral combinations can be provided within a single readout period, useful and dynamic spectral information can still be acquired, even for video streams.

[0180] In one embodiment, the video or other image may include a black and white or color image overlaid with information derived from the spectral response of each pixel. For example, pixels corresponding to particular tissues or conditions may be shown in bright green or other colors to assist a physician or other medical professional during an examination.

[0181] In one embodiment, dual image sensors can be used to acquire three-dimensional images or video feeds. Three-dimensional examination can allow for improved understanding of the three-dimensional structure of the examined area as well as mapping of different tissue or material types within that area.

[0182] In one embodiment, multispectral or hyperspectral imaging can be used to see through materials or substances. For example, infrared wavelengths can pass through some tissues, such as muscle or fat, while reflecting off blood vessels. In one embodiment, infrared waves can penetrate 5, 8, or 10 mm or more into tissue. Acquiring a series of frames, including at least one infrared frame, can enable the examination to provide information regarding the location of blood vessels below the surface. This can be extremely useful in surgical procedures where it may be desirable to perform an incision that avoids blood vessels. In one embodiment, a color or grayscale image can be overlaid with a green color indicating the location of blood vessels below the surface. Similarly, the known spectral response of blood can be used to see through the blood during examination to reveal tissues or structures of interest.

[0183] Assembling the sub-frames into a single frame for display on a monitor or other display device can occur after capturing the series of frames 4404-4414. A color or grayscale image can be generated from one or more of the frames, and pixel overlay information can be determined based on all or the remaining frames. The color or grayscale image can be combined with the overlay information to generate a single frame. The single frame can be displayed as a single image or as an image in a video stream.

[0184] In one embodiment, the hyperspectral data acquired, as illustrated in FIG. 44 , can be provided for analysis by a third-party algorithm to classify the tissue or material captured in the image. In one embodiment, the third-party algorithm can be used to select the spectrum or wavelength bands to be used during imaging so that a desired spectral response analysis can be performed. In one embodiment, the spectral response analysis can be performed in real time during a medical imaging procedure or other medical procedure. The spectral data can be overlaid onto an RGB or black-and-white image, allowing a user to easily distinguish specific types of tissue, organs, chemical processes, diseases, etc. In one embodiment, the spectral data can be provided to a computer-operated system, such as a robotic system for automating medical imaging or medical procedures.

[0185] 45 is a schematic diagram of an imaging system 4500 with a single-cut filter. The system 4500 includes an endoscope 4506 or other suitable imaging device with a light source 4508 for use in low-light environments. The endoscope 4506 includes an image sensor 4504 and a filter 4502 for filtering unwanted wavelengths of light or other electromagnetic radiation before reaching the image sensor 4504. The light source 4508 transmits light that can illuminate a surface 4512 in a low-light environment, such as a body cavity. The light 4510 reflects off the surface 4512 and passes through the filter 4502 before hitting the image sensor 4504.

[0186] Filter 4502 can be used in an implementation where a fluorescent reagent or dye is administered. In one such embodiment, filter 4502 is configured to filter all light or other electromagnetic radiation except light of one or more desired wavelengths or spectral bands. In one embodiment, filter 4502 is configured to filter electromagnetic radiation at an excitation wavelength that causes the reagent or dye to fluoresce, such that only the expected relaxation wavelength of the fluorescing reagent or dye is allowed to pass through filter 4502 and reach image sensor 4504. In one embodiment, filter 4502 filters at least fluorescent reagent excitation wavelengths between 770 nm and 790 nm. In one embodiment, filter 4502 filters at least fluorescent reagent excitation wavelengths between 795 nm and 815 nm. In one embodiment, filter 4502 filters at least fluorescent reagent excitation wavelengths between 770 nm and 790 nm and between 795 nm and 815 nm. In these embodiments, filter 4502 filters out the excitation wavelength of the reagent, allowing only the relaxation wavelength of the fluorescent reagent to be read out by image sensor 4504. Image sensor 4504 can be a wavelength-independent image sensor, and filter 4502 can be configured to allow image sensor 4504 to receive only the relaxation wavelength of the fluorescent reagent and not the emission excitation wavelength of the reagent. The data determined by image sensor 4504 can then indicate the presence of important body structures, tissues, biological processes, or chemical processes determined by the location of the reagent or dye.

[0187] A filter 4502 can also be used in an implementation in which no fluorescent reagent or dye is administered. The filter 4502 can be selected to allow wavelengths corresponding to a desired spectral response to pass and be read out by the image sensor 4504. The image sensor 4504 can be a monochrome image sensor such that pixels of the captured image above or below a threshold can be characterized as corresponding to a particular spectral response or fluorescent emission. The spectral response or fluorescent emission determined by the pixels captured by the image sensor 4504 can indicate the presence of a particular body tissue or structure, a particular condition, a particular chemical process, etc.

[0188] In one embodiment, the light source 4508 transmits white light that contacts the surface 4512, where the white light reflects back where it is filtered by the filter 4502 before hitting the image sensor 4504. In one embodiment, the light source 4508 transmits white light that passes through the filter 4502, such that filtered light of only one or more desired wavelengths emerges from the filter 4502, is reflected off the surface 4512, and is read by the image sensor 4504. For example, in one embodiment, the filter 4502 allows only light having a wavelength of 795 nm to pass through the filter 4502 and contact the image sensor 4504. Further, in one embodiment, the filter 4502 allows only light of certain wavelengths to reflect back to the image sensor 4504 of the endoscope 4506 or other imaging device. The filter 4502 can be located anywhere within the system 4500 or can be an attachment to the lens or image sensor 4504. The filter 4502 can be located in front of and / or behind the image sensor 4504. In one embodiment, the light emitted by the light source 4508 is filtered before it reaches the surface 4512, and the reflected light is filtered by an additional filter before it becomes usable by the image sensor 4504.

[0189] Light source 4508 can be an emitter that can be configured to emit white light or electromagnetic radiation of one or more specific wavelengths. Light source 4508 can include multiple lasers configured to emit or pulse light of specified wavelengths. In one embodiment, light source 4508 emits white light, and filter 4502 is selected to filter out all unwanted light or other electromagnetic radiation except for light of one or more desired wavelengths. Filter 4502 can be selected for a particular examination or purpose, for example, to highlight certain types of body tissues or structures, or to highlight particular conditions or chemical processes.

[0190] FIG. 46 is a schematic diagram of an imaging system 4600 having multiple cut-off filters. The system 4600 includes an endoscope 4606 or other suitable imaging device having a light source 4608 for use in low-light environments. The endoscope 4606 includes an image sensor 4604 and two filters 4602a, 4602b. It should be understood that in alternative embodiments, the system 4600 can include any number of filters, and the number and type of filters can be selected for a particular purpose, such as to collect imaging information for a particular body tissue, body condition, chemical process, etc. The filters 4602a, 4602b are configured to filter unwanted wavelengths of light or other electromagnetic radiation. The filters 4602a, 4602b can be configured to filter unwanted wavelengths from white light or other electromagnetic radiation that may be emitted by the light source 4608. The filtered light can strike a surface 4612 (e.g., body tissue) and reflect back to the image sensor 4604.

[0191] Continuing with this disclosure with respect to FIG. 45 , filters 4602a, 4602b can be used in an implementation in which a fluorescent reagent or dye is administered. Filters 4602a, 4602b can be configured to block the emission excitation wavelength of the reagent or dye, allowing image sensor 4604 to read only the relaxation wavelength of the reagent or dye. Additionally, filters 4602a, 4602b may be used in an implementation in which a fluorescent reagent or dye is not administered. In such an implementation, filters 4602a, 4602b can be selected to allow wavelengths corresponding to a desired spectral response to pass and be read by image sensor 4604.

[0192] Each of the multiple filters 4602a, 4602b can be configured to filter a different wavelength range of the electromagnetic spectrum. For example, one filter can be configured to filter wavelengths above a desired wavelength range, and an additional filter can be configured to filter wavelengths below the desired wavelength range. The combination of two or more filters can result in only a particular wavelength or band of wavelengths being read out by the image sensor 4604.

[0193] In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 513 nm and 545 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 565 nm and 585 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 900 nm and 1000 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 425 nm and 475 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 520 nm and 545 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 625 nm and 645 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 760 nm and 795 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 795 nm and 815 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 370 nm and 420 nm to contact image sensor 4604. In one embodiment, filters 4602a, 4602b are customized to allow electromagnetic radiation between 600 nm and 670 nm to contact image sensor 4604. In one embodiment, the filters 4602 a , 4602 b are configured to allow only specific fluorescent relaxation emissions to pass through the filters 4602 a , 4602 b and contact the image sensor 4604 .

[0194] In one embodiment, system 4600 includes multiple image sensors 4604, and in particular, can include two image sensors for use in generating three-dimensional images. Image sensor(s) 4604 can be color / wavelength independent and can be configured to read any wavelength of electromagnetic radiation reflected from surface 4612. In one embodiment, image sensors 4604 are each color-dependent or wavelength-dependent and configured to read specific wavelengths of electromagnetic radiation reflected from surface 4612 back to image sensor 4604. Alternatively, image sensor 4604 can include a single image sensor having multiple different pixel sensors configured to read different wavelengths or colors of light, such as a Bayer filter color filter array. Alternatively, image sensor 4604 can include one or more color-independent image sensors that can be configured to read different wavelengths of electromagnetic radiation according to a pulsed schedule, such as those illustrated in FIGS. 5-7E and 15-16.

[0195] FIG. 47 is a schematic diagram illustrating a system 4700 for mapping a surface and / or tracking an object in a low-light environment. In one embodiment, an endoscope 4702 in a low-light environment pulses a grid array 4706 (which may also be referred to as a laser map pattern) of a surface 4704. The grid array 4706, in one embodiment illustrated in FIG. 47, may include vertical hashing 4708 and horizontal hashing 4710. It should be understood that the grid array 4706 may include any suitable array for mapping the surface 4704, including, for example, a raster grid of individual locations, an occupancy grid map, a dot array, etc. Additionally, the endoscope 4702 may pulse multiple grid arrays 4706, and may pulse one or more individual grid arrays for each of multiple objects or structures in the low-light environment, for example.

[0196] In one embodiment, system 4700 pulses grid array 4706, which can be used to determine three-dimensional surfaces and / or track the position of an object, such as a tool or another device, in a low-light environment. In one embodiment, system 4700 can provide data to a third-party system or computer algorithm to determine the dimensions and configuration of a surface through light detection and ranging (LIDAR) mapping. System 4700 can pulse any suitable wavelength of light or electromagnetic radiation in grid array 4706, including, for example, ultraviolet, visible, and / or infrared or near-infrared. Surfaces 4704 and / or objects in the environment can be mapped and tracked with very high resolution and with very high accuracy and precision.

[0197] In one embodiment, the system 4700 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. The system 4700 can include a light engine having an illumination source that generates one or more pulses of electromagnetic radiation and a lumen that transmits the one or more pulses of electromagnetic radiation to a distal tip of an endoscope within a light-deficient environment, such as a body cavity. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation include a laser map pattern that is emitted onto a surface within the light-deficient environment, such as a surface of body tissue and / or a surface of a tool or other device within the body cavity. The endoscope 4702 can include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations within the light-deficient environment.

[0198] In one embodiment, system 4700 includes a processor for determining the distance of the endoscope or tool from an object, such as surface 4704. The processor can further determine the angle between the endoscope or tool and the object. The processor can further determine surface area information about the object, including, for example, surgical tool size, structure size, anatomical structure size, location information, and other location data and metrics. System 4700 can include one or more image sensors that provide image data that is output to a control system for determining the distance from the endoscope or tool to an object, such as surface 4704. The image sensors can output information to the control system for determining the angle between the endoscope or tool and the object. Additionally, the image sensors can output information to the control system for determining the surface area information about the object, surgical tool size, structure size, anatomical structure size, location information, and other location data and metrics.

[0199] In one embodiment, the grid array 4706 is pulsed by an illumination source of the endoscope 4702 at a sufficient rate so that the grid array 4706 is not visible to a user. In various implementations, a user may be distracted by viewing the grid array 4706 during an endoscopic imaging procedure and / or an endoscopic surgical procedure. The grid array 4706 may be pulsed for a sufficiently short period so that the grid array 4706 is not detectable by the human eye. In an alternative embodiment, the endoscope 4702 pulses the grid array 4706 at a sufficient repetition rate so that the grid array 4706 is visible to a user. In one such embodiment, the grid array 4706 can be superimposed on an image of the surface 4704 on a display. The grid array 4706 can be superimposed on a black and white or RGB image of the surface 4704 so that the grid array 4706 is visible to a user during use of the system 4700. A user of the system 4700 can indicate whether the grid array 4706 should be overlaid on an image of the surface 4704 and / or whether the grid array 4706 should be visible to the user. The system 4700 can include a display that provides real-time measurements of the distance from the endoscope 4702 to the surface 4704 or another object in a low-light environment. The display can further provide real-time surface area information for the surface 4704 and / or any object, structure, or tool in the low-light environment. The accuracy of the measurements can be accurate to within one millimeter.

[0200] The endoscope 4702 can pulse the electromagnetic radiation according to a pulsing schedule such as those illustrated in Figures 5-7E and 15-16, which can further include pulsing the grid array 4706 along with pulsing red, green, and blue light to generate an RGB image, and further generating a grid array 4706 that can be overlaid on the RGB image and / or used to map and track surfaces 4704 and objects in low light environments.

[0201] In one embodiment, the endoscope 4702 includes one or more color-independent image sensors. In one embodiment, the endoscope 4702 includes two color-independent image sensors for generating a three-dimensional image or map of the low-light environment. The image sensors can generate RGB images of the low-light environment according to a pulsing schedule disclosed herein. In addition, the image sensors can determine data for mapping the low-light environment and tracking one or more objects within the low-light environment based on data determined when the grid array 4706 is pulsed. In addition, the image sensors can determine spectral or hyperspectral data along with the fluorescence imaging data according to a pulsing schedule that can be modified by a user to meet the specific needs of the imaging procedure. In one embodiment, the pulsing schedule includes red, green, and blue pulses along with pulsing of the grid array 4706 and / or pulsing to generate hyperspectral image data and / or fluorescence image data. In various implementations, the pulsing schedule can include any suitable combination of pulses of electromagnetic radiation depending on the needs of the user. The repetition rate of the different wavelengths of electromagnetic radiation can be determined based on, for example, the energy of a particular pulse, the needs of the user, whether particular data (e.g., hyperspectral data and / or fluorescence imaging data) needs to be updated continuously or less frequently, etc.

[0202] The pulsing schedule can be modified in any suitable manner, and a particular pulse of electromagnetic radiation can be repeated at any suitable frequency according to the needs of a user or a computer-implemented program for a particular imaging procedure. For example, in an embodiment in which surface tracking data generated based on grid array 4706 is provided to a computer-implemented program for use in, for example, a robotic surgical procedure, grid array 4706 can be pulsed more frequently than if the surface tracking data were provided to a user visualizing the scene during the imaging procedure. In such an embodiment in which the surface tracking data is used in a robotic surgical procedure, the surface tracking data may need to be updated more frequently or may need to be extremely accurate to enable the computer-implemented program to perform the robotic surgical procedure with precision and accuracy.

[0203] In one embodiment, the system 4700 is configured to generate an occupancy grid map comprising an array of cells divided into a grid, and the system 4700 is configured to store respective height values ​​for each grid cell to determine a surface mapping of a three-dimensional environment within the low-light environment.

[0204] 48 is a schematic flow chart diagram of a method 4800 for hyperspectral imaging in low light environments. The method 4800 can be performed by an imaging system such as the endoscopic imaging system illustrated in FIG.

[0205] The method 4800 includes, at 4802, emitting multiple narrowband pulses during a readout period of a monochrome image sensor. The pulses can be emitted using a light source including multiple emitters that emit electromagnetic energy within a narrow frequency band. For example, the light source can include at least one emitter for multiple frequency bands covering a desired spectrum. At 4804, the monochrome image sensor reads pixel data from the monochrome image sensor and generates multiple frames after the readout period. Each frame can include a different spectral content. These frames can include multiple repeating frames that can be used to generate a digital video stream. Each frame can be based on energy emitted by one or more emitters of the light source. In one embodiment, the frames can be based on a combination of light emitted by the light source to generate a combination of frequencies that matches the frequency response of a desired tissue or material. At 4806, a controller, CCU, or other system determines a tissue spectral response for one or more pixels based on the multiple frames. For example, the frequency response of a particular pixel can be determined based on the values ​​of the pixel in the multiple frames using the pixel value and knowledge of the frequency of light emitted for each frame. At 4808, the system can generate a composite image based on the multiple frames, the composite image including an overlay showing the spectral response of one or more pixels. For example, the composite image can be grayscale or a color image in which pixels corresponding to a particular tissue or classification are shown in bright green.

[0206] 49 is a schematic flow chart diagram of a method 4900 for fluorescence imaging in low light environments. The method 4900 can be performed by an imaging system such as the endoscopic imaging system illustrated in FIG.

[0207] The method 4900 includes, at 4902, emitting multiple narrowband pulses during a readout period of a monochrome image sensor. The pulses can be emitted using a light source including multiple emitters that emit electromagnetic energy within a narrow frequency band. For example, the light source can include at least one emitter for multiple frequency bands covering a desired spectrum. At 4904, the monochrome image sensor reads pixel data from the monochrome image sensor and generates multiple frames after the readout period. Each frame can include a different spectral component. These frames can include multiple repeating frames that can be used to generate a digital video stream. Each frame can be based on energy emitted by one or more emitters of the light source. In one embodiment, the frames can be based on a combination of light emitted by the light source to generate a combination of frequencies that matches the frequency response of a desired tissue or material. At 4906, a controller, CCU, or other system determines the fluorescent relaxation emission of a reagent for one or more pixels based on the multiple frames. For example, the frequency response of a particular pixel can be determined based on the pixel values ​​in the multiple frames using the pixel values ​​and knowledge of the frequency of light emitted for each frame. At 4908, the system can generate a composite image based on multiple frames, the composite image including an overlay showing the fluorescent relaxation emission of one or more pixels. For example, the composite image can be grayscale or a color image in which pixels corresponding to a particular tissue or classification are shown in bright green. [Example]

[0208] The following examples relate to further embodiments. Example 1 is an endoscope system for use in low-light environments. The system includes an imaging device. The imaging device includes a tube, one or more image sensors, and a lens assembly with at least one optical element corresponding to the image sensor. The system includes a display for a user to visualize the scene, and an image signal processing controller. The system includes a light engine. The light engine includes an illumination source that generates one or more pulses of electromagnetic radiation. The light engine further includes a lumen that transmits the one or more pulses of electromagnetic radiation to a distal tip of the endoscope, at least a portion of the one or more pulses of electromagnetic radiation including a laser map pattern that is emitted onto a surface of body tissue.

[0209] A second embodiment is the endoscope system according to the first embodiment, further comprising a two-dimensional camera.

[0210] A third embodiment is an endoscope system according to any one of the first and second embodiments, further comprising a three-dimensional camera.

[0211] A fourth embodiment is an endoscope system according to any one of the first to third embodiments, further including an n-dimensional camera.

[0212] A fifth embodiment is the endoscope system according to any one of the first to fourth embodiments, in which the laser map pattern is not visible to a user of the endoscope system.

[0213] A sixth embodiment is the endoscope system according to any one of the first to fifth embodiments, in which the laser map pattern is visible to a user of the endoscope system.

[0214] Example 7 is an endoscopic system described in any of Examples 1 to 6, wherein the system further includes a processor that determines, for a user of the endoscopic system, at least one of the distance of the endoscope or tool from the object, the angle between the endoscope or tool and the object, or surface area information about the object.

[0215] Example 8 is an endoscopic system described in any of Examples 1 to 7, wherein the image sensor provides image data to a user of the endoscopic system that is output to a control system that determines at least one of the distance of the endoscope from the object, the angle between the endoscope and the object, surface area information about the object, size of the surgical tool, size of the structure, size of the anatomical structure, position information, and other positional data and metrics.

[0216] A ninth embodiment is the endoscope system according to any one of the first to eighth embodiments, in which the display of the laser map pattern can be started or stopped by a user of the endoscope system.

[0217] Example 10 is an endoscopic system described in any of Examples 1 to 9, wherein the display provides a user of the endoscopic system with real-time measurements of one or more of the distance of the endoscope from the object, the angle between the endoscope and the object, or surface area information about the object.

[0218] Example 11 is the endoscopic system of any of Examples 1 to 10, wherein the real-time measurements provided by the display are accurate to within 10 centimeters.

[0219] Example 12 is the endoscopic system of any of Examples 1 to 11, wherein the real-time measurements provided by the display are accurate to within less than one millimeter.

[0220] A thirteenth embodiment is the endoscope system according to any one of the first to twelfth embodiments, in which the imaging device includes a first image sensor and a second image sensor for generating a three-dimensional image.

[0221] Example 14 is the endoscopic system according to any one of Examples 1 to 13, wherein the system includes a plurality of tools, each tool having one or more laser map patterns.

[0222] Example 15 is an endoscopic system described in any of Examples 1 to 14, wherein the optical engine has a first output and a second output that are independent of each other, the first output being for optical illumination and the second output being for tool tracking.

[0223] Example 16 is an endoscopic system described in any of Examples 1 to 15, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are displayed to a user as a single image on a display.

[0224] Example 17 is an endoscopic system described in any of Examples 1 to 16, in which a single image is assigned a visible color for use on the display, and the visible color is 8 bits, 16 bits, or n bits.

[0225] Example 18 is an endoscopic system described in any of Examples 1 to 17, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to the user as an overlay image on the display.

[0226] Example 19 is an endoscopic system described in any of Examples 1 to 18, in which the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits, 16 bits, or n bits.

[0227] Example 20 is the endoscope system according to any one of Examples 1 to 19, wherein the optical engine includes a polarizing filter.

[0228] Example 21 is the endoscope system according to any one of Examples 1 to 20, wherein a polarizing filter is disposed in the path of the electromagnetic radiation.

[0229] Example 22 is an endoscopic system as described in any of Examples 1-21, wherein the polarizing filter is disposed at the proximal end of the lumen.

[0230] Example 23 is the endoscope system according to any one of Examples 1 to 22, wherein the polarizing filter is disposed at the distal end of the lumen.

[0231] Example 24 is an endoscope system according to any one of Examples 1 to 23, wherein the lens assembly includes an electromagnetic radiation filter.

[0232] Example 25 is the endoscope system according to any one of Examples 1 to 24, wherein the lens assembly includes a polarizing filter.

[0233] Example 26 is an endoscopic system described in any of Examples 1 to 25, wherein each pulse of electromagnetic radiation results in an exposure frame created by an image sensor, and one or more exposure frames are supplied to a corresponding system that provides the location of important tissue structures.

[0234] Example 27 is an endoscopic system described in any of Examples 1 to 26, wherein the locations of important structures are received by the endoscopic system and overlaid on the display, and the important structures are coded in any color selected by either an algorithm or a user.

[0235] It will be appreciated that the various features described herein represent important advantages and advances in the art, some of which are exemplified in the following claims.

[0236] In the foregoing Detailed Description, various features of the disclosure are grouped together in a single embodiment for purposes of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. Rather, inventive aspects lie in less than all features of a single embodiment of the foregoing disclosure.

[0237] It is to be understood that the above-described arrangements are merely illustrative of the application of the principles of the present disclosure, and that numerous modifications and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present disclosure, and the appended claims are intended to cover all such modifications and arrangements.

[0238] Thus, while the present disclosure has been illustrated and described above with specificity and detail, it will be apparent to those skilled in the art that numerous modifications, including but not limited to variations in size, material, shape, form, function and method of operation, assembly and use, can be made without departing from the principles and concepts described herein.

[0239] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) may be programmed to execute one or more of the systems and procedures described herein. Certain terms are used throughout the following description and claims to refer to particular system components. As will be understood by those skilled in the art, components may be referred to by different names. This document does not intend to distinguish between components that differ in name but not function.

[0240] The foregoing description has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. Furthermore, it should be noted that any or all of the above-described alternative embodiments may be used in any combination desired to form further hybrid embodiments of the present disclosure.

[0241] Furthermore, although particular embodiments of the present disclosure have been described and illustrated, the disclosure should not be limited to the specific forms and arrangements of parts so described and illustrated. The scope of the present disclosure is to be defined by the claims appended hereto, any future claims filed herewith, and in any subsequent applications and their equivalents.

[0242] [Embodiment] (1) An endoscope system for use in a low-light environment, comprising: An imaging device, Tube and one or more image sensors; a lens assembly comprising at least one optical element corresponding to the image sensor; and a display for a user to visualize the scene; an image signal processing controller; 1. A light engine, comprising: an illumination source generating one or more pulses of electromagnetic radiation; An endoscope system comprising: an optical engine having a lumen that transmits one or more pulses of electromagnetic radiation to a distal tip of the endoscope, wherein at least a portion of the one or more pulses of electromagnetic radiation include a laser map pattern that is emitted at a surface of body tissue. (2) An endoscopic system as described in embodiment 1, further comprising a two-dimensional camera. (3) An endoscopic system as described in embodiment 1, further comprising a three-dimensional camera. (4) An endoscopic system as described in embodiment 1, further comprising an n-dimensional camera. (5) An endoscopic system as described in embodiment 1, wherein the laser map pattern is invisible to a user of the endoscopic system.

[0243] (6) An endoscopic system as described in embodiment 1, wherein the laser map pattern is visible to a user of the endoscopic system. (7) An endoscopic system as described in embodiment 1, wherein the system further comprises a processor that determines, for the user of the endoscopic system, at least one of the distance of an endoscope or tool from an object, the angle between the endoscope or tool and the object, or surface area information regarding the object. (8) An endoscopic system as described in embodiment 1, wherein the image sensor provides image data to the user of the endoscopic system that is output to a control system that determines at least one of the distance of the endoscope from the object, the angle between the endoscope and the object, surface area information about the object, size of the surgical tool, size of the structure, size of the anatomical structure, position information, and other positional data and metrics. (9) An endoscopic system as described in embodiment 1, wherein the display of the laser map pattern can be activated or deactivated by the user of the endoscopic system. (10) An endoscopic system as described in embodiment 1, wherein the display provides the user of the endoscopic system with real-time measurements of one or more of the distance of the endoscope from the object, the angle between the endoscope and the object, or surface area information about the object.

[0244] (11) An endoscopic system as described in embodiment 10, wherein the real-time measurements provided by the display are accurate to less than 10 centimeters. (12) An endoscopic system as described in embodiment 10, wherein the real-time measurements provided by the display are accurate to less than 1 millimeter. (13) An endoscopic system as described in embodiment 1, wherein the imaging device comprises a first image sensor and a second image sensor for generating a three-dimensional image. (14) An endoscopic system as described in embodiment 1, wherein the system includes a plurality of tools, each tool having one or more laser map patterns. (15) An endoscopic system as described in embodiment 1, wherein the optical engine has a first output and a second output that are independent of each other, the first output being for optical illumination and the second output being for tool tracking.

[0245] (16) An endoscopic system as described in embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to the user as a single image on the display. (17) An endoscopic system as described in embodiment 16, wherein the single image is assigned a visible color for use on the display, and the visible color is 8 bits, 16 bits, or n bits. (18) An endoscopic system as described in embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are displayed to the user as an overlay image on the display. (19) An endoscope system as described in embodiment 18, wherein the overlay image is assigned a visible color for use on the display, and the visible color is 8 bits, 16 bits, or n bits. (20) An endoscopic system as described in embodiment 1, wherein the optical engine comprises a polarizing filter.

[0246] (21) An endoscopic system as described in embodiment 20, wherein the polarizing filter is positioned within the path of the electromagnetic radiation. (22) An endoscopic system as described in embodiment 21, wherein the polarizing filter is positioned at the proximal end of the lumen. (23) An endoscopic system as described in embodiment 21, wherein the polarizing filter is positioned at the distal end of the lumen. (24) An endoscopic system as described in embodiment 1, wherein the lens assembly includes an electromagnetic radiation filter. (25) An endoscopic system as described in embodiment 1, wherein the lens assembly includes a polarizing filter.

[0247] (26) An endoscopic system as described in embodiment 1, wherein each pulse of electromagnetic radiation results in an exposure frame created by the image sensor, and one or more exposure frames are supplied to a corresponding system that provides the location of important tissue structures. (27) An endoscopic system as described in embodiment 26, wherein the location of the important structures is received by the endoscopic system and overlaid on a display, and the important structures are coded in any color selected by either an algorithm or a user.

Claims

1. 1. An endoscope system for use in a low light environment, comprising: An imaging device, one or more image sensors; a lens assembly comprising at least one optical element corresponding to the image sensor; a display for a user to visualize the scene; an image signal processing controller; a light engine comprising a plurality of illumination sources each generating one or more pulses of electromagnetic radiation; a controller in electrical communication with the light engine and the image sensor; the one or more pulses of electromagnetic radiation generated from at least one of the plurality of illumination sources are pulses including a laser map pattern emitted onto a surface within the low-light environment, and the image sensor senses the electromagnetic radiation reflected from the surface illuminated by the pulses to generate image data including surface area information for one or more anatomical structures and one or more tools; the controller synchronizes operation of the image sensor and the light engine such that the image sensor senses reflected electromagnetic radiation emitted by the illumination source to generate a plurality of exposure frames; An endoscope system, wherein a portion of the plurality of exposure frames constitute a plurality of mapping image frames, and the mapping image frames constitute the image data.

2. The endoscope system of claim 1 , further comprising a two-dimensional camera.

3. The endoscope system of claim 1 , further comprising a three-dimensional camera.

4. The endoscope system of claim 1 , further comprising an n-dimensional camera.

5. The endoscopic system of claim 1 , wherein the laser map pattern is invisible to the user of the endoscopic system.

6. The endoscopic system of claim 1 , wherein the laser map pattern is visible to the user of the endoscopic system.

7. The endoscopic system of claim 1, further comprising an endoscope and a processor, wherein the processor analyzes the image data to determine, for the user of the endoscopic system, at least one of distances between the endoscope or the multiple tools and the multiple anatomical structures, angles between the endoscope or the multiple tools and the multiple anatomical structures, or surface area information of the multiple anatomical structures or the multiple tools.

8. the endoscope system further comprises an endoscope; 2. The endoscopic system of claim 1, wherein the image data is output to a control system to determine at least one of the following for the user of the endoscopic system: a distance of the endoscope from the anatomical structure, an angle between the endoscope and the anatomical structure, surface area information about the anatomical structure, a size of a surgical tool, a size of the anatomical structure, position information of the anatomical structure, or position data and metrics of the surgical tool.

9. An endoscopic system as described in claim 1, wherein the display of the laser map pattern can be started or stopped by the user of the endoscopic system.

10. the endoscope system further comprises an endoscope; The endoscopic system of claim 1, wherein the display provides the user of the endoscopic system with real-time measurements of one or more of the distance of the endoscope from the anatomical structure or the tool, the angle between the endoscope and the anatomical structure or the tool, or surface area information regarding the anatomical structure or the tool according to the image data.

11. 11. The endoscopic system of claim 10, wherein the real-time measurement provided by the display is the distance of the endoscope from the anatomical structure or the tool to an accuracy of less than 10 centimeters.

12. The endoscopic system of claim 10 , wherein the real-time measurement provided by the display is the distance of the endoscope from the anatomical structure or the tool to an accuracy of less than one millimeter.

13. The endoscopic system of claim 1 , wherein the imaging device comprises a first image sensor and a second image sensor for generating a three-dimensional image.

14. 10. The endoscope system of claim 1, wherein the light engine comprises a first output and a second output that are independent of each other, the first output being for light illumination and the second output being for tool tracking.

15. 2. The endoscopic system of claim 1, wherein the one or more electromagnetic radiation pulses result in an exposure frame in the plurality of exposure frames created by the image sensor, and one or more of the plurality of exposure frames are displayed to the user as a single image on the display.

16. 16. The endoscope system of claim 15, wherein the single image is assigned a visible color for use on the display, the visible color being 8-bit, 16-bit, or n-bit.

17. 2. The endoscopic system of claim 1, wherein the one or more electromagnetic radiation pulses result in an exposure frame in the plurality of exposure frames created by the image sensor, and one or more in the plurality of exposure frames are displayed to the user as an overlay image on the display.

18. 18. The endoscope system of claim 17, wherein the overlay image is assigned a visible color for use on the display, the visible color being 8-bit, 16-bit, or n-bit.

19. 2. The endoscopic system of claim 1, wherein the one or more pulses of electromagnetic radiation result in an exposure frame in the plurality of exposure frames created by the image sensor, and one or more of the plurality of exposure frames are supplied to a corresponding system that provides the location of one or more tissue structures.

20. 20. The endoscopic system of claim 19, wherein the location of the tissue structure is received by the endoscopic system and overlaid on the display, and the tissue structure is coded in any color selected by either an algorithm or the user.

21. the endoscope system includes the plurality of tools, and the pulses including the laser map pattern are also emitted onto surfaces of the plurality of tools; the laser map pattern is pulsed according to a pulsing schedule; 2. The endoscopic system of claim 1, wherein the pulsing schedule includes pulsing the laser map pattern at a rate that is invisible to the user and pulsing the laser map pattern at a rate that is visible to the user, and is selected according to the needs of the user.

Images