Hyperspectral imaging for tool tracking in low-light environments

Abstract

The present invention discloses an endoscopic imaging system for use in a light-deficient environment, the endoscopic imaging system comprising an imaging device having a tube, one or more image sensors and a lens assembly, the lens assembly comprising at least one optical element corresponding to the one or more image sensors. The endoscopic system comprises a display and an image signal processing controller, the display being used to provide a visualized scene for a user. The endoscopic system comprises a light engine, the light engine having an illumination source and an inner cavity, the illumination source generating one or more electromagnetic radiation pulses, the inner cavity transmitting the one or more electromagnetic radiation pulses to the distal end of the endoscope.

Description

Background Art

[0001] Advances in technology have provided advances in medical imaging capabilities. Endoscopes can be used to view the inside of the body and examine the inside of an organ or cavity of the body. Endoscopes can be used to investigate a patient's symptoms, confirm a diagnosis, or provide medical treatment. Medical endoscopes can be used to view a variety of body systems and parts, including, for example, the gastrointestinal tract, respiratory tract, urinary tract, abdominal cavity, etc., through small incisions. Endoscopes can also be used for surgical procedures, such as orthopedic procedures, procedures performed on joints or bones, procedures performed on the nervous system, procedures performed within the abdominal cavity, etc.

[0002] Endoscopes have also been used in non-medical fields to observe and inspect spaces that may be out of reach or difficult to see. For example, a planner or architect may use an endoscope to visualize a scale model of a proposed building or city. Endoscopes can be used to visualize the internal spaces of complex systems such as computers. Law enforcement or military personnel may even use endoscopes to conduct surveillance or inspect explosive devices in confined spaces. In addition, endoscopic imaging may be used by computer-implemented programs to perform robotic medical procedures. In particular, in such implementations, it may be beneficial to track the position of an object during endoscopic imaging. The object may include the endoscope itself, a medical device, tissues or structures within the body, and the like.

[0003] In various uses of endoscopes, it may be advantageous to observe color space. For each pixel of the image, a digital color image may include at least three layers or "color channels". Each of the color channels measures the intensity and chromaticity of light of a spectral band. Typically, a digital color image includes color channels for red, green, and blue spectral bands (this may be referred to as an RGB image). Each of the red, green, and blue color channels includes brightness information for a red, green, or blue spectral band. The brightness information of the separate red, green, and blue layers may be combined to generate a digital color image. Because the color image is composed of separate layers, a digital camera image sensor typically includes a color filter array that allows red, green, and blue visible light wavelengths to hit the selected pixel sensor. Each individual pixel sensor element is sensitive to a red, green, or blue wavelength and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image.

[0004] In the case where endoscopic imaging is used for medical diagnosis or medical procedures, it may be advantageous or even necessary to observe the body cavity through color images. For example, if an endoscope is used to observe the abdominal cavity of the body, the color image can provide valuable information to help identify different organs or tissues within the abdomen, or to identify certain conditions or diseases within the space. As discussed above, a digital camera capable of capturing color images may have at least three different types of pixel sensors to capture the red, green, and blue layers of the color image separately. The at least three different types of pixel sensors may consume relatively significant physical space (when compared to a color-indeterminate pixel array), so that the complete pixel array cannot fit on the small distal end of the endoscope inserted into the body. Because the color digital camera may include the at least three different types of pixel sensors, the total pixel array (i.e., the image sensor) is typically located in the handpiece unit of the endoscope, which is held by the endoscope operator and is not placed in the body cavity. For such endoscopes, light is transmitted from the handpiece unit along the length of the endoscope to the distal end of the endoscope placed in the body cavity. This endoscope configuration has significant limitations. Endoscopes with this configuration are delicate and can easily become misaligned or damaged when they are bumped or impacted during regular use. This can significantly reduce the quality of the images generated by the endoscope, and the endoscope requires frequent repair or replacement.

[0005] Endoscopic imaging can be used to guide practitioners during medical procedures such as diagnostic imaging procedures or surgical procedures. In addition, in some implementations, it may be desirable to use endoscopic imaging and endoscopic medical devices to perform computer-implemented robotic surgery. In both cases, it may be desirable to determine precise measurements indicating, for example, distances and / or angles between structures within the body, devices in a body cavity, or tools and critical structures within the body. Such measurements can improve the results of endoscopic procedures and may be necessary in the case of robotic endoscopic procedures.

[0006] Various measurement systems and methods exist in the art for applications in archeology, geography, atmospheric physics, autonomous vehicles, and the like. One such system includes light detection and ranging (LIDAR), which is a three-dimensional laser scanning system. LIDAR has been applied in navigation systems such as aircraft or satellites to determine the position and orientation of the sensor in combination with other systems and sensors. LIDAR can use active sensors to illuminate an object and detect energy reflected from the object and returned to the sensor. Laser scanning technology has been applied to airborne and terrestrial environments. Airborne laser scanning has been used by aircraft during flight to generate a three-dimensional point cloud of the aircraft's surrounding landscape. Terrestrial laser scanning has been used to survey stationary or moving objects on the surface of the earth for topography, monitoring, and position determination.

[0007] However, the application of laser scanning technology known in the art generally requires highly specialized equipment, which may not be available for additional applications. In addition, laser scanning technology provides a limited view of the environment and must generally be used in conjunction with multiple separate systems. For example, an autonomous vehicle deploys a LIDAR system in conjunction with multiple imaging cameras, radar sensors, ultrasonic sensors, and the like. The many sensors and imaging systems required for an autonomous vehicle or other system may be very expensive and may take up significant physical space. In the context of medical imaging procedures such as medical endoscopic imaging, all sensors must fit within a small physical area within a body cavity. In some cases, such as in the imaging of joints or organs, the geographic area may be particularly small and may accommodate only a very small end of an endoscope. Therefore, medical endoscopes are necessarily small and cannot accommodate multiple different imaging and ranging systems.

[0008] In certain implementations, it may be advantageous to deploy an endoscopic imaging system for generating images of a body cavity and further for determining measurements within the body cavity. Such images and measurements may be provided to a user for viewing the body cavity and / or used by a computer-implemented program for executing a robotic system.

[0009] The present disclosure generally relates to electromagnetic sensing and sensors applicable to endoscopic imaging. The present disclosure also relates to low energy electromagnetic input conditions and low energy electromagnetic throughput conditions. More specifically, the present invention relates to (but not necessarily exclusively to) systems for producing images in light-deficient environments and associated structures, methods and features, which may include controlling a light source by duration, intensity, or both, pulsing a light source controlled by a component during a blanking period of an image sensor, maximizing the blanking period to allow for optimal light, and maintaining color balance.

[0010] The features and advantages of the present disclosure will be described in the following description, and will be partially obvious from the description, or the practice of the present disclosure can be learned without excessive experiments. The features and advantages of the present disclosure can be realized and obtained by the instruments and combinations particularly pointed out in the attached claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] A non-limiting and non-exhaustive embodiment of the present disclosure is described with reference to the following drawings, wherein like reference numerals refer to like parts in the various views unless otherwise specified. The advantages of the present disclosure will be better understood with reference to the following description and drawings, wherein:

[0012] Figure 1 is a schematic diagram of an embodiment of a paired sensor and electromagnetic transmitter system in operation for producing an image in a light-poor environment according to one embodiment;

[0013] Figure 2is a schematic diagram of the complementary system hardware;

[0014] FIG. 2A to FIG. 2D is a diagram of an operation cycle of a sensor for constructing an image frame according to an embodiment of the present disclosure;

[0015] Figure 3 is a graphical representation of the operation of an embodiment of an electromagnetic transmitter according to one embodiment;

[0016] Figure 4 A graphical representation of varying the duration and magnitude of emitted electromagnetic pulses to provide exposure control according to one embodiment;

[0017] Figure 5 According to one embodiment, Figures 2A to 4 A graphical representation of an embodiment of the present disclosure of an operating cycle of a sensor, an electromagnetic transmitter and an electromagnetic pulse transmitted in combination, which shows the imaging system during operation;

[0018] Figure 6 A schematic diagram showing two different methods for recording video frames of full spectrum light and partitioned spectrum light during a time period from t(0) to t(1) according to one embodiment;

[0019] 7A to 7E A schematic diagram showing a method for recording video frames of both full spectrum light and partitioned spectrum light within a time interval in accordance with the principles and teachings of the present disclosure;

[0020] Figures 8 to 12 shows adjustments of both an electromagnetic transmitter and a sensor in accordance with the principles and teachings of the present disclosure, wherein in some embodiments such adjustments may be made simultaneously;

[0021] Figures 13 to 21 shows a sensor calibration method and hardware schematic for use with a zoned light system according to an embodiment of the present disclosure;

[0022] Figure 22 to Figure 23 A method and hardware schematic diagram for increasing the dynamic range in a closed environment or a limited light environment according to an embodiment of the present disclosure is shown;

[0023] Fig.24 shows the effect of color correction on the signal-to-noise ratio of a typical Bayer-based sensor compared to no color correction;

[0024] Fig.25 The chromaticity of 3 monochromatic lasers compared to the sRGB color gamut is shown;

[0025] Figures 26 to 27BA method and hardware schematic diagram for increasing the dynamic range in a closed environment or a limited light environment according to an embodiment of the present disclosure is shown;

[0026] FIG. 28A to FIG. 28C shows the use of pulsed and / or synchronized white light emission with corresponding color sensors according to embodiments of the present disclosure;

[0027] Fig.29A and Fig.29B A specific implementation having a plurality of pixel arrays for generating a three-dimensional image according to an embodiment of the present disclosure is shown;

[0028] Fig. 30A and 30B A perspective view and a side view, respectively, of a specific implementation of an imaging sensor constructed on multiple substrates, wherein multiple columns of pixels forming a pixel array are located on a first substrate, and multiple columns of circuits are located on a second substrate, showing electrical connections and communications between a column of pixels and its associated or corresponding column of circuits;

[0029] Fig.31A and Fig.31B A perspective view and a side view, respectively, illustrate a specific implementation of an imaging sensor having multiple pixel arrays for producing a three-dimensional image, wherein the multiple pixel arrays and the image sensor are constructed on multiple substrates;

[0030] Figure 32 to Figure 36 Embodiments of emitters including various mechanical filter and shutter configurations according to embodiments of the present disclosure are shown;

[0031] Fig.37 is a schematic diagram showing a system for providing lighting to a light-deficient environment according to one embodiment;

[0032] Fig.38 is a schematic block diagram showing a light source having multiple emitters according to one embodiment;

[0033] Fig.39 is a schematic block diagram showing a light source having multiple emitters according to another embodiment;

[0034] Fig.40 is a schematic block diagram showing a light source having a plurality of emitters according to yet another embodiment;

[0035] Fig.41 is a schematic diagram showing a single optical fiber output at an output through a diffuser to illuminate a scene according to one embodiment;

[0036] Fig.42 is a block diagram illustrating the use of a filter to generate a filtered image according to one embodiment;

[0037] Fig.43 shows a portion of an electromagnetic spectrum divided into a plurality of different sub-spectra that may be emitted by an emitter of a light source according to one embodiment;

[0038] Fig.44 is a schematic diagram showing a timing diagram of emission and readout for generating a multispectral or hyperspectral image according to one embodiment;

[0039] Fig.45 is a block diagram illustrating the use of a filter to generate a filtered image according to one embodiment;

[0040] Fig.46 is a block diagram illustrating the use of multiple filters to generate a filtered image according to one embodiment;

[0041] Fig.47 is a schematic diagram showing a grid array for object and / or surface tracking according to one embodiment;

[0042] Fig.48 is a schematic flow chart illustrating a method for generating emission and readout of a multispectral or hyperspectral image according to one embodiment; and

[0043] Fig.49 is a schematic flow chart illustrating a method for generating emission and readout of a fluorescence image according to one embodiment. DETAILED DESCRIPTION

[0044] The present disclosure relates to methods, systems and computer-based products that are primarily applicable to digital imaging for medical applications such as medical endoscopic imaging. Such methods, systems and computer-based products as disclosed herein can provide imaging or diagnostic capabilities for medical robotic applications, such as the use of robots for performing imaging procedures, surgical procedures, etc. In the following description of the present disclosure, reference is made to the accompanying drawings, which form a part of the present disclosure and in which specific implementations in which the present disclosure may be implemented are shown by way of illustration. It should be understood that other implementations may be adopted and structural changes may be made without departing from the scope of the present disclosure.

[0045] Endoscopes have a variety of uses and can provide significant benefits in the medical field. Endoscopy is used in medicine to view the inside of the body and in some cases can provide imaging that would otherwise be impossible to see or would require invasive surgical procedures. Endoscopes can be used for medical diagnosis, investigation or research, and can also be used to perform medical procedures in a minimally invasive manner. Medical endoscopes can provide significant benefits to patients and practitioners by eliminating the need for painful invasive corrective or exploratory surgery.

[0046] As disclosed herein, an endoscopic system for use in a light-deficient environment (such as a cavity of the body) may include an imaging device and a light engine. The light engine may include an illumination source for generating electromagnetic radiation pulses, and may also include an inner cavity for transmitting the electromagnetic radiation pulses to the distal end of the endoscope. The inner cavity may transmit electromagnetic radiation pulses at a specific wavelength or wavelength band of the electromagnetic spectrum. The inner cavity may transmit such pulses in a timed sequence, and imaging data may be captured by a sensor during each pulse. Imaging data associated with different wavelengths of the pulses can be used to generate red, green, and blue (RGB) images and / or fluorescent images. In one embodiment, the fluorescent imaging can be superimposed on the black and white or RGB image.

[0047] As disclosed herein, systems, methods, and devices for endoscopic image systems can provide dedicated image data for light-deficient environments. The dedicated image data can be used to generate fluorescence imaging and / or identify certain substances, tissues, components, or processes in light-deficient environments. In certain embodiments, the fluorescence imaging can be provided to a medical practitioner or a computer-implemented program to enable identification of certain structures or tissues in the body. Such fluorescence imaging data can be layered on a black and white or RGB image to provide additional information and context.

[0048] In addition, such systems, methods and devices for endoscopic image systems can be used in conjunction with certain reagents or dyes. In a medical imaging specific implementation, certain reagents or dyes can be applied to the patient, and these reagents or dyes can fluoresce or react to electromagnetic radiation of certain wavelengths. As disclosed herein, the endoscopic image system can transmit electromagnetic radiation of a specific wavelength to make the reagent or dye fluoresce. The fluorescence of the reagent or dye can be captured by an image sensor to generate imaging, thereby facilitating the identification of tissues or structures and / or facilitating diagnosis or research. In a specific implementation, a plurality of reagents or dyes can be applied to the patient, each of which is configured to emit fluorescence of different wavelengths and / or provide indications of different structures, tissues, chemical reactions, biological processes, etc. In such a specific implementation, the endoscopic system as disclosed herein can emit each of the applicable wavelengths to make each of the applicable reagents or dyes fluoresce. This can eliminate the historical need to perform individual imaging procedures for each of the multiple reagents or dyes.

[0049] A medical endoscope can provide a continuous stream of digital images of the interior space of the body into which the distal end of the endoscope is inserted. In various embodiments, it may be advantageous or even necessary for the digital image stream to provide full-color imaging so that a medical practitioner can better distinguish tissues and structures in the body. In other embodiments, it may be advantageous to provide hyperspectral imaging data to distinguish structures, tissues, processes, and conditions with enhanced accuracy. In addition, hyperspectral imaging can enable a medical practitioner or a computer program to receive information about human conditions that are not visible to the human eye or that are not distinguishable in RGB color images.

[0050] Disclosed herein are systems, methods, and devices for generating color image data and / or fluorescent image data by an endoscope. The disclosed system includes an imaging device having a tube, one or more image sensors, and a lens assembly. The lens assembly may include at least one optical element corresponding to at least one of the one or more image sensors. The system also includes a display and an image signal processing controller, the display being used to visualize the scene. The system may also include a light engine. The light engine includes an illumination source and an inner cavity, the illumination source being configured to generate one or more electromagnetic radiation pulses, the inner cavity transmitting the one or more electromagnetic radiation pulses to the distal end of the endoscope. In one embodiment, at least a portion of the one or more electromagnetic radiation pulses includes an excitation wavelength of electromagnetic radiation between 770nm and 790nm, the excitation wavelength causing one or more agents to fluoresce at a wavelength different from the excitation wavelength of the portion of the one or more electromagnetic radiation pulses.

[0051] In one embodiment of the present disclosure, the endoscope system illuminates the source and pulses electromagnetic radiation at certain wavelengths for exciting electrons in the reagent or dye. In one embodiment, the reagent or dye is configured to fluoresce in response to electromagnetic radiation of certain wavelengths emitted by the endoscope system. An image sensor in the endoscope system can read the fluorescence relaxation emission of the reagent or dye, which may have a lower energy than the pulsed electromagnetic radiation used to excite the reagent or dye. The reagent or dye may be specifically used to mark certain tissues, structures, biological processes, and / or chemical processes.

[0052] Imaging agents (including fluorescent agents) can enhance the imaging capabilities in the pharmaceutical, medical, biotechnology, diagnostic and medical procedure industries. Many imaging techniques such as X-ray, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI) and nuclear medicine mainly analyze anatomical structures and morphology, and cannot detect changes at the molecular level. Fluorescent agents, dyes and probes (including quantum dot nanoparticles and fluorescent proteins) can assist medical imaging techniques by providing additional information about certain tissues, structures, chemical processes and / or biological processes present in the imaging area. Imaging using fluorescent agents can achieve cell tracking and / or tracking of certain molecular biomarkers. Fluorescent agents can be used to image cancer, infection, inflammation, stem cell biology, etc. Many fluorescent agents and dyes are being developed and applied to visualize and track biological processes in a non-destructive manner. Such fluorescent agents can be excited by electromagnetic radiation of certain wavelengths or wavelength bands. Similarly, these fluorescent agents can emit relaxation energy of certain wavelengths or wavelength bands when fluorescing, and the relaxation energy emitted can be read by a sensor to determine the position and / or boundary of the agent or dye.

[0053] In one embodiment of the present disclosure, the endoscope system pulses electromagnetic radiation for exciting electrons in a fluorescent agent or dye. The wavelength or wavelength band of the electromagnetic radiation can be specifically selected for causing certain agents or dyes to fluoresce. In one embodiment, the endoscope system can pulse electromagnetic radiation of multiple different wavelengths during a single imaging process for causing multiple different agents or dyes to fluoresce. The sensors of the endoscope system can determine the position and / or boundaries of the agent or dye based on the relaxation emission of the agent or dye. The endoscope system can also pulse electromagnetic radiation in the red, green, and blue bands of visible light. The endoscope system can determine data for RGB images and fluorescent images based on the pulse scheduling of the electromagnetic radiation pulses.

[0054] In one embodiment of the present disclosure, the endoscope system illuminates the source and pulses electromagnetic radiation for spectral or hyperspectral imaging. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from conventional cameras that only capture light across three wavelengths (including red, green, and blue wavelengths) based on the visible spectrum that can be distinguished by the human eye to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can stack imaging generated based on invisible bands (e.g., infrared) on top of imaging based on visible bands (e.g., standard RGB images) to provide additional information that can be easily distinguished by humans or computer algorithms.

[0055] Hyperspectral imaging is a subcategory of spectral imaging. Hyperspectral imaging includes spectroscopy and digital photography. In one embodiment of hyperspectral imaging, a complete spectrum or some spectral information is collected at each pixel in the image plane. A hyperspectral camera can use special hardware to capture any suitable number of wavelength bands for each pixel that can be interpreted as a complete spectrum. The goals of hyperspectral imaging can vary for different applications. In one application, the goal of hyperspectral imaging is to obtain the entire electromagnetic spectrum for each pixel in the image scene. This can make it possible to find certain objects that might not be recognizable in the visible light wavelength band. This can make it possible to accurately identify certain substances or tissues, although these substances or tissues might not be recognizable in the visible light wavelength band. In addition, this can make it possible to detect certain processes by capturing images across all wavelengths of the electromagnetic spectrum.

[0056] In medical applications, hyperspectral imaging can provide specific advantages over conventional imaging. The information obtained by hyperspectral imaging can enable practitioners and / or computer-implemented programs to accurately identify certain tissues or conditions that may not be diagnosed or may be diagnosed less accurately when using conventional imaging such as RGB imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, which can enable practitioners to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells that contrast with typical healthy cells, identify certain tissues or conditions, identify key structures, etc. Hyperspectral imaging can provide specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0057] In various applications and implementations of the present disclosure, endoscopic hyperspectral imaging can present advantages over conventional imaging. In medical implementations, endoscopic hyperspectral imaging can allow a practitioner or a computer-implemented program to distinguish, for example, neural tissue, muscle tissue, various blood vessels, blood flow direction, etc. Hyperspectral imaging can enable atypical cancer tissue to be accurately distinguished from typical healthy tissue, and thus can enable a practitioner or a computer-implemented program to distinguish the boundaries of cancerous tumors during surgery or research imaging. In addition, hyperspectral imaging in a light-deficient environment as disclosed herein can be combined with the use of reagents or dyes to allow further distinction of certain tissues or substances. In such embodiments, reagents or dyes can fluoresce at specific wavelength bands in the electromagnetic spectrum, thereby providing information specific to the purpose of the reagent or dye. Systems, methods, and devices as disclosed herein can enable pulsing of any number of wavelength bands, so that one or more reagents or dyes can fluoresce at different times. In certain implementations, this can enable identification or study of multiple medical conditions during a single imaging procedure.

[0058] Medical endoscopes can pulse electromagnetic radiation at wavelengths outside the visible light spectrum to enable the generation of hyperspectral images. Endoscopic hyperspectral imaging is a non-contact and non-invasive means for medical imaging that does not require the patient to experience harmful radiation exposure common in other imaging methods.

[0059] Conventional endoscopes used in, for example, robotic endoscopic procedures such as arthroscopy and laparoscopy are designed so that the image sensor is typically placed in a handpiece unit that is held by the endoscope operator and is not inserted into the cavity. In such a configuration, the endoscope unit transmits incident light to the sensor along the length of the endoscope tube with minimal loss and distortion via a complex set of precisely coupled optical components. Since the optical components are expensive and labor-intensive to manufacture, the cost of the endoscope unit depends primarily on the optical components. In addition, endoscopes of this type are mechanically delicate, and relatively minor impacts can easily damage the components or disturb the relative alignment of those components. Even a slight misalignment with the endoscope components (such as precisely coupled optical components) can result in a significant reduction in image quality or render the endoscope unusable. When the components are misaligned, the incident light traveling along the length of the endoscope can be attenuated, so that there is little or no light at the distal end of the endoscope, and the endoscope becomes unusable. Because conventional endoscopes require such precise and complex optical components, and because those components can easily become misaligned, such conventional endoscopes require frequent and expensive repair cycles to maintain image quality.

[0060] One solution to this problem is to place the image sensor at the distal end within the endoscope itself. Such a solution can eliminate the need for a complex and precise collection of coupled optical components that can be easily misaligned and / or damaged. This solution is potentially close to the optical simplicity, robustness, and economy commonly achieved in, for example, cell phone cameras. However, it should be understood that a large number of benefits provided by the endoscope arise from the compact size of the distal end of the endoscope. If the distal end of the endoscope is enlarged to accommodate a plurality of different wavelength sensitive pixel sensors conventionally used for color imaging or hyperspectral imaging, the pixel array may be too large, and the endoscope may no longer fit into a narrow space or may be obstructive or invasive when used in medical implementations. Because the distal end of the endoscope must remain very small, it is challenging to place one or more image sensors at the distal end. An acceptable solution to this approach is very important because it introduces its own set of engineering challenges, not just the fact that the sensor for color imaging or hyperspectral imaging must fit into a very limited area. This is particularly challenging when the pixel array in a conventional camera includes separate pixel sensors for each of the red, green, and blue visible light bands, as well as additional pixel sensors for other wavelength bands for hyperspectral imaging. The area of ​​the distal tip of an endoscope may be particularly limited side-to-side in the X and Y dimensions, while there is more space along the length of the endoscope tube in the Z dimension.

[0061] Because many of the benefits of the endoscope arise from the small size of the distal end of the endoscope, when the image sensor is located at the distal end, strong constraints must be imposed on the image sensor area. These strong constraints imposed on the sensor area naturally lead to fewer and / or smaller pixels in the pixel array. Reducing the number of pixels can directly affect the spatial resolution, while reducing the pixel area can reduce the available signal capacity and thus reduce the sensitivity of the pixel, and optimizing the number of pixels maximizes the image quality, the minimum pixel resolution and the use of the maximum pixel quality and spacing of the original number of pixels so that the resolution is not a problem and reduces the signal-to-noise ratio (SNR) of each pixel. Reducing the signal capacity will reduce the dynamic range, that is, the ability of the imaging device or camera to capture all useful information from a wide range of brightness scenes at the same time. There are a variety of methods that can extend not only the dynamic range of the pixel itself, but also the dynamic range of the imaging system. However, all of these methods may have some kinds of defects (for example, in terms of resolution or frame rate), and they will introduce undesirable artifacts, which become problematic in extreme cases. The result of reduced sensitivity is that more optical power is required to adjust the darker areas in the scene to an acceptable signal level. Reducing the aperture number (enlarging the aperture) can compensate for the loss of sensitivity, but it will cause spatial distortion and reduce the depth of focus.

[0062] In the sensor industry, complementary metal oxide semiconductor ("CMOS") image sensors have a large shift from conventional charge coupled device ("CCD") image sensors in modern camera applications. Compared to CCD image sensors, CMOS image sensors have greater ease of integration and operation, better or comparable image quality, greater flexibility, and lower cost. Generally, a CMOS image sensor may include the necessary circuitry to convert image information into digital data, and then incorporate various levels of digital processing. These processes may include basic algorithms for correcting non-ideal factors (e.g., non-ideal factors caused by variations in power amplifier behavior) to a full image signal processing (ISP) chain to provide video data in a standard red-green-blue ("RGB") color space, for example (a camera-on-a-chip).

[0063] The control unit of the endoscope or image sensor may be located remotely from the image sensor and may be at a significant physical distance from the image sensor. When the control unit is remote from the sensor, it may be desirable to transmit data in the digital domain because it is largely immune to interfering noise and signal attenuation compared to transmitting analog data streams. It should be understood that various electronic digital signaling standards may be used, such as LVDS (low voltage differential signaling), sub-LVDS, SLVS (scalable low voltage signaling), or other electronic digital signaling standards.

[0064] It may be highly desirable to minimize the number of electrical conductors to reduce the number of pads taking up space on the sensor and to reduce the complexity and cost of sensor manufacturing. While adding analog-to-digital conversion to the sensor may be advantageous, the additional area occupied by the conversion circuitry is offset by the significant reduction in analog buffering power required for the earlier conversion to a digital signal.

[0065] In terms of area consumption, given the typical feature sizes available in CMOS image sensor (CIS) technology, in some implementations it may be preferable to generate all internal logic signals on the same chip as the pixel array via a set of control registers and a simple command interface.

[0066] Some specific implementations of the present disclosure include various aspects of sensor and system combination designs that can generate high-definition images with reduced pixel counts in highly constrained lighting environments. This can be achieved by pulsing a monochromatic wavelength frame by frame, and switching or alternating between a single different color wavelength each frame using a controlled light source in combination with a high frame capture rate and a specially designed corresponding monochromatic sensor. In addition, electromagnetic radiation outside the visible spectrum can be pulsed to enable the generation of hyperspectral images. Pixels can be color amorphous so that each pixel can generate data for each electromagnetic radiation pulse, including pulses of red, green, and blue visible light wavelengths, as well as other wavelengths that can be used for hyperspectral imaging.

[0067] As used herein, a monochrome sensor refers to an imaging sensor without filtering. Because the pixels are color amorphous, their effective spatial resolution is significantly higher than their pixel color (usually filtered with a Bayer pattern) counterparts in traditional single sensor cameras. They can also have a higher quantum efficiency because fewer incident photons are wasted between individual pixels. In addition, in order to blur the color artifacts associated with the Bayer pattern, spatial color modulation based on the Bayer theory requires a lower modulation transfer function (MTF) of the corresponding optics compared to monochrome modulation. This has a detrimental effect on the actual spatial resolution achievable with a color sensor.

[0068] The present disclosure is also concerned with system solutions for endoscopic inspection applications where the image sensor is located at the distal end of the endoscope. In pursuit of a system based on a minimum area sensor, there are other design aspects that can be developed in addition to reducing the number of pixels. The area of ​​the digital portion of the chip can be minimized. In addition, the number of connections to the chip (pads) can also be minimized. The present disclosure describes a novel method for achieving these goals for such a system. This involves the design of a fully customized CMOS image sensor with several novel features.

[0069] For the purpose of promoting understanding of the principles according to the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe these embodiments. However, it should be understood that it is not intended to limit the scope of the present disclosure. Any changes and further modifications to the inventive features shown herein, and any additional applications of the principles of the present disclosure as shown herein (which will generally occur for those skilled in the relevant art and those who understand the present disclosure) will be considered to be within the scope of the disclosure protected by the claims.

[0070] Before disclosing and describing structures, systems and methods for generating images in a light-deficient environment, it should be understood that the present disclosure is not limited to the specific structures, configurations, process steps and materials disclosed herein, because such structures, configurations, process steps and materials may vary to a certain extent. It should also be understood that the terminology used herein is only for the purpose of describing specific embodiments and is not intended to be limiting, because the scope of the present invention will be limited only by the appended claims and their equivalents.

[0071] In describing and claiming the presently disclosed subject matter, the following terminology will be used in accordance with the following definitions.

[0072] It must be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0073] As used herein, the terms "comprising," "including," "characterized by," and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unrecited elements or method steps.

[0074] As used herein, the phrase "consisting of and its grammatical equivalents exclude any elements or steps not listed in a claim.

[0075] As used herein, the phrase "consisting essentially of and its grammatical equivalents limit the scope of a claim to the specified materials or steps and those materials or steps that do not materially affect one or more of the basic and novel characteristics of the claimed disclosure.

[0076] As used herein, the term "proximal side" broadly refers to the concept of a portion close to a starting point.

[0077] As used herein, the term "distal" generally refers to the opposite of the proximal side, and thus refers to the concept of a portion farther from a starting point or a portion farthest from a starting point according to the context.

[0078] As used herein, color sensors or multispectral sensors are those sensors known to have a color filter array (CFA) thereon to filter incident electromagnetic radiation into its individual components. In the visible range of the electromagnetic spectrum, such CFAs may be based on a Bayer template or a modified form thereof to separate the green, red and blue spectral components of light.

[0079] See now Figures 1 to 5 , systems and methods for producing images in a low-light environment will now be described. Figure 1 A schematic diagram of a paired sensor and electromagnetic emitter in operation for producing an image in a low-light environment is shown. This configuration allows for increased functionality in light-controlled or ambient low-light environments.

[0080] It should be noted that, as used herein, the term "light" is both a particle and a wavelength, and is intended to represent electromagnetic radiation that can be detected by a pixel array, and may include wavelengths from the visible and invisible spectrum of electromagnetic radiation. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is less than the entire spectrum, or in other words, the wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that is controllable with respect to a portion of the electromagnetic spectrum emitted, or a light source that can manipulate the physical properties of its components, the emission intensity, or the emission duration, or all of the above. An emitter can emit light in any jittered, diffuse, or collimated emission, and can be controlled digitally or by an analog method or system. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and it includes light sources such as lasers, LEDs, incandescent lights, or any light source that can be digitally controlled.

[0081] The pixel array of the image sensor can be electronically paired with the transmitter so that during operation they are synchronized for receiving transmissions and adjustments made within the system. Figure 1 As can be seen in FIG. 1 , the emitter 100 can be tuned to emit electromagnetic radiation in the form of a laser, which can be pulsed to illuminate the object 110. The emitter 100 can be pulsed at intervals corresponding to the operation and function of the pixel array 122. The emitter 100 can pulse light in multiple electromagnetic sectors 105 so that the pixel array receives electromagnetic energy and generates a data set corresponding (in time) to each specific electromagnetic sector 105. For example, Figure 1A system is shown having a monochrome sensor 120 having a pixel array (black and white) 122 and supporting circuitry, the pixel array 122 being sensitive to electromagnetic radiation of any wavelength. The light emitter 100 shown in the figure may be a laser emitter capable of emitting a red electromagnetic partition 105a, a blue electromagnetic partition 105b, and a green electromagnetic partition 105c in any desired sequence. In an embodiment in which a hyperspectral image may be generated, the light emitter 100 may pulse electromagnetic radiation at any wavelength in the electromagnetic spectrum so that a hyperspectral image may be generated. It should be understood that other light emitters 100 may be used without departing from the scope of the present disclosure. Figure 1 such as digital or analog based transmitters.

[0082] During operation, the data generated by the monochrome sensor 120 for any individual pulse can be assigned a specific color or wavelength partition, where the assignment is based on the timing of the pulse color or wavelength partition from the emitter 100. Even if the pixel 122 is not color dedicated, any given data set can be assigned a color based on a priori information about the emitter.

[0083] In an exemplary embodiment of the present disclosure, the emitter 100 pulses electromagnetic radiation at a specific wavelength. Such pulses can enable the generation of specialized fluorescence images that are particularly suitable for certain medical or diagnostic applications. In an exemplary embodiment, at least a portion of the electromagnetic radiation emitted by the emitter 100 includes an excitation wavelength of electromagnetic radiation between 770 nm and 790 nm and between 795 nm and 815 nm that causes one or more agents to fluoresce at a wavelength different from the excitation wavelength of the portion of the electromagnetic radiation pulse.

[0084] In one embodiment, three data sets representing RED, GREEN and BLUE electromagnetic pulses can be combined to form a single image frame. One or more additional data sets representing other wavelength partitions can be stacked on a single image frame based on red, green and blue pulses. The one or more additional data sets can represent, for example, fluorescence imaging in response to excitation wavelengths between 770nm and 790nm and between 795nm and 815nm. The one or more additional data sets can represent fluorescence imaging and / or hyperspectroscopy that can be stacked on a single image frame based on red, green and blue pulses.

[0085] It should be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition, and any color combination or any electromagnetic partition may be used in place of RED, GREEN, and BLUE, such as cyan, magenta, and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible and invisible wavelengths, without departing from the scope of the present disclosure. In the figure, the object 110 to be imaged includes a red portion 110a, a green portion 110b, and a blue portion 110c. As shown, the reflected light from the electromagnetic pulse contains data only for the portion of the object having a particular color corresponding to the pulse color partition. These individual color (or color interval) data sets can then be used to reconstruct an image by combining the data sets at 130.

[0086] In one embodiment, multiple data sets representing red, green, and blue electromagnetic pulses may be combined along with additional wavelength partitions along the electromagnetic spectrum to form a single image frame having an RGB image with the hyperspectral image data layered on the RGB image. Depending on the application or instance, different combinations of wavelength data sets may be required. For example, in some implementations, data sets representing specific wavelength partitions may be used to generate specialized hyperspectral images for diagnosing specific medical conditions, studying certain body tissues, etc.

[0087] like Figure 2 As shown in, the specific implementation of the present disclosure may include or utilize a special-purpose computer or a general-purpose computer, including computer hardware, such as, for example, one or more processors and system memory, as discussed in detail below. Implementations within the scope of the present disclosure may also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media that can be accessed by a general-purpose or special-purpose computer system. The computer-readable medium that stores computer-executable instructions is a computer storage medium (device). The computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, the specific implementation of the present disclosure may include at least two obviously different types of computer-readable media: a computer storage medium (device) and a transmission medium.

[0088] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store the desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer.

[0089] "Network" refers to one or more data links that enable electronic data to be transmitted between computer systems and / or modules and / or other electronic devices. In one specific implementation, the sensor and camera control unit can be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer reasonably regards the connection as a transmission medium. The transmission medium may include a network and / or data link that can be used to carry the desired program code tools in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer. The above combinations should also be included in the scope of computer-readable media.

[0090] In addition, upon reaching various computer system components, program code tools in the form of computer executable instructions or data structures can be automatically transmitted to computer storage media (devices) by transmission media (or vice versa). For example, computer executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., "NIC") and then ultimately transmitted to the computer system RAM and / or the non-volatile computer storage media (devices) of the computer system. RAM can also include solid state drives (SSDs or PCIx-based real-time memory hierarchical storage devices, such as FusionIO). Therefore, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0091] Computer executable instructions include instructions and data that, when run in a processor, cause a general purpose computer, a special purpose computer, or a special purpose processing device to perform certain functions or groups of functions. Computer executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter of the present invention is described in language with respect to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or steps described above. Rather, the features and steps described above are disclosed as example forms of implementing the claims.

[0092] It should be understood by those skilled in the art that the present disclosure can be implemented in a network computing environment having various types of computer system configurations, including personal computers, desktop computers, laptop computers, information processors, control units, camera control units, handheld devices, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the above computing devices can be provided by a physical institution or located within a physical institution. The present disclosure can also be implemented in a distributed system environment, in which local and remote computer systems are connected via a network (via a hard-wired data link, a wireless data link, or a combination of a hard-wired data link and a wireless data link), and both can perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0093] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0094] Figure 2 is a block diagram illustrating an exemplary computing device 150. Computing device 150 may be used to execute various programs, such as those discussed herein. Computing device 150 may be used as a server, a client, or any other computing entity. Computing device 150 may perform the various monitoring functions discussed herein, and may execute one or more applications, such as the applications described herein. Computing device 150 may be any of a variety of computing devices, such as a desktop computer, a laptop computer, a server computer, a handheld computer, a camera control unit, a tablet computer, and the like.

[0095] The computing device 150 includes one or more processors 152, one or more memory devices 154, one or more interfaces 156, one or more mass storage devices 158, one or more input / output (I / O) devices 160, and a display device 180, all coupled to a bus 162. The processor 152 includes one or more processors or controllers that execute instructions stored in the memory device 154 and / or the mass storage device 158. The processor 152 may also include various types of computer-readable media, such as cache memory.

[0096] Memory device 154 includes various computer-readable media such as volatile memory (e.g., random access memory (RAM) 164) and / or nonvolatile memory (e.g., read-only memory (ROM) 166). Memory device 154 may also include a rewritable ROM such as flash memory.

[0097] The mass storage device 158 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., flash memory), etc. Figure 2 As shown in FIG. 1 , the particular mass storage device is a hard disk drive 174. Various drives may also be included in the mass storage device 158 to enable reading from and / or writing to various computer readable media. The mass storage device 158 includes removable media 176 and / or non-removable media.

[0098] I / O devices 160 include various devices capable of inputting or retrieving data and / or other information to or from computing device 150. Exemplary I / O devices 160 include digital imaging devices, electromagnetic sensors and transmitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCD or other image capture devices, and the like.

[0099] Display device 180 includes any type of device capable of displaying information to one or more users of computing device 150. Examples of display device 180 include a monitor, a display terminal, a video projection device, and the like.

[0100] The interface 106 includes various interfaces that enable the computing device 150 to interact with other systems, devices, or computing environments. Exemplary interface 156 may include any number of different network interfaces 170, such as interfaces to connect to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet. Other interfaces include a user interface 168 and a peripheral device interface 172. The interface 156 may also include one or more user interface elements 168. The interface 156 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touch pad, etc.), keyboards, etc.

[0101] The bus 162 enables the processor 152, the memory device 154, the interface 156, the mass storage device 158, and the I / O device 160 to communicate with each other and with other devices or components coupled to the bus 162. The bus 162 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, and the like.

[0102] For exemplary purposes, programs and other executable program components are shown herein as discrete blocks, but it should be understood that such programs and components may reside at various times in different storage components of the computing device 150 and be executed by one or more processors 152. Alternatively, the systems and programs described herein may be implemented in hardware, or in a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more of the systems and programs described herein.

[0103] Figure 2A An operating cycle of a sensor used in a rolling readout mode or during a sensor readout 200 is shown. A frame readout may begin at and may be represented by a vertical line 210. A readout cycle is represented by a diagonal or slanted line 202. The sensor may be read out row by row, with the top of the downward sloping edge being the sensor top row 212 and the bottom of the downward sloping edge being the sensor bottom row 214. The time between the last row readout and the next readout cycle may be referred to as a blanking time 216. It should be noted that some of the sensor pixel rows may be covered with a light shield (e.g., a metal coating or any other substantially black layer of another material type). These covered pixel rows may be referred to as optical black rows 218 and 220. Optically black rows 218 and 220 may be used as input to a correction algorithm. As shown Figure 2A As shown in , these optical black rows 218 and 220 can be located on the top of the pixel array or on the bottom of the pixel array, or on both the top and bottom of the pixel array. Figure 2B A method for controlling the amount of electromagnetic radiation (e.g., light) exposed to a pixel so that it is integrated or accumulated by the pixel is shown. It should be understood that photons are the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into an electric charge or current. An electronic shutter or rolling shutter (shown by dashed line 222) can be used to start the integration time by resetting the pixel. The light will then be integrated until the next readout phase. The position of the electronic shutter 222 can be moved between two readout cycles 202 to control the pixel saturation for a given amount of light. It should be noted that this technique allows a constant integration time between two different rows, but introduces a delay when moving from the top row to the bottom row. Figure 2C The case is shown where the electronic shutter 222 has been removed. In this configuration, integration of incident light may begin during readout 202 and may end at the next readout cycle 202, which also defines the start of the next integration. Figure 2DA configuration is shown without the electronic shutter 222 but with controlled and pulsed light 230 during the blanking time 216. This ensures that all rows see the same light as that emitted from the same light pulse 230. In other words, each row will begin its integration in a dark environment, which may be located at the optically black post row 220 of the readout frame (m) to obtain the maximum light pulse width, and will then receive light through and will end its integration in a dark environment, which may be located at the optically black pre row 218 of the next subsequent readout frame (m+1) to obtain the maximum light pulse width. In e.g. Figure 2D In the example, the image generated by the light pulse will be available only during the readout of frame (m+1) without interfering with frames (m) and (m+2). It should be noted that the condition for the light pulse to be read out only in one frame and not interfere with adjacent frames is that a given light pulse is fired during the blanking time 216. Because the optical black rows 218, 220 are not sensitive to light, the optical black post-row 220 time of frame (m) and the optical black pre-row 218 time of frame (m+1) can be added to the blanking time 216 to determine the maximum range of the firing time of the light pulse 230. Figure 2A As shown, the sensor may cycle multiple times to receive data for each pulse color or wavelength (e.g., red, green, blue or other wavelength on the electromagnetic spectrum). Each cycle may be timed. In one embodiment, the cycle may be timed to operate within an interval of 16.67 ms. In another embodiment, the cycle may be timed to operate within an interval of 8.3 ms. It should be understood that the present disclosure contemplates other timing intervals and is intended to fall within the scope of the present disclosure.

[0104] Figure 3 The operation of an embodiment of an electromagnetic transmitter is graphically illustrated.The transmitter may be timed to correspond to a cycle of the sensor such that electromagnetic radiation is emitted within and / or during a portion of a sensor operating cycle. Figure 3 Pulse 1 at 302, pulse 2 at 304, and pulse 3 at 306 are shown. In one embodiment, the emitter may pulse during the readout portion 202 of the sensor operating cycle. In one embodiment, the emitter may pulse during the blanking portion 216 of the sensor operating cycle. In one embodiment, the emitter may pulse for a duration that is during a portion of two or more sensor operating cycles. In one embodiment, the emitter may begin pulsing during the blanking portion 216 or during the optically black portion 220 of the readout portion 202, and end pulsing during the readout portion 202 or during the optically black portion 218 of the readout portion 202 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of the present disclosure as long as the pulsing of the emitter and the cycle of the sensor correspond.

[0105] Figure 4A graphical representation of varying the duration and magnitude of emitted electromagnetic pulses (e.g., pulse 1 at 402, pulse 2 at 404, pulse 3 at 406) to control exposure. An emitter with a fixed output magnitude may be combined with the above Figure 2D and Figure 3 The pulses are pulsed at intervals during any of the cycles described to provide the desired electromagnetic energy to the pixel array. An emitter with a fixed output magnitude may be pulsed at longer intervals to provide more electromagnetic energy to the pixel, or the emitter may be pulsed at shorter intervals to provide less electromagnetic energy. Whether longer or shorter intervals are required depends on the operating conditions.

[0106] Compared to adjusting the time interval of the fixed output magnitude of the emitter pulse, the emission magnitude itself can be increased to provide more electromagnetic energy to the pixel. Similarly, reducing the magnitude of the pulse can provide less electromagnetic energy to the pixel. It should be noted that embodiments of the system can have the ability to adjust both the magnitude and duration if desired. In addition, the sensor can be adjusted to increase its sensitivity and duration as required for optimal image quality. Figure 4 The magnitude and duration of the pulses are shown to be varied. In the illustration, pulse 1 at 402 has a higher magnitude or intensity than pulse 2 at 404 or pulse 3 at 406. In addition, pulse 1 at 402 has a shorter duration than pulse 2 at 404 or pulse 3 at 406, so that the electromagnetic energy provided by the pulse is shown by the area under the pulse shown in the illustration. In the illustration, pulse 2 at 404 has a relatively low magnitude or intensity and a longer duration when compared to pulse 1 at 402 or pulse 3 at 406. Finally, in the illustration, pulse 3 at 406 has an intermediate magnitude or intensity and duration when compared to pulse 1 at 402 and pulse 2 at 404.

[0107] Figure 5 According to the principles and teachings of this disclosure, Figures 2 to 4 200 , an electromagnetic transmitter and an electromagnetic pulse emitted are shown in a graphical representation of an embodiment of the present disclosure showing an imaging system during operation. As can be seen in the figure, the electromagnetic transmitter pulses primarily during the blanking period 216 of the sensor so that the pixels will be charged and ready to read during the readout portion 202 of the sensor cycle. The dashed line portion of the pulse (from Figure 3 ) illustrates the possibility or ability to emit electromagnetic energy during the optically black portions 220 and 218 of a read cycle (sensor cycle) 200 in the event that additional time is needed or desired to pulse the electromagnetic energy.

[0108] See now Figures 6 to 9 A, Figure 6Schematic diagrams of two different methods for recording video frames of full spectrum light and partitioned spectrum light over a time period from t(0) to t(1) are shown. It should be noted that the color sensor has a color filter array (CFA) for filtering out certain wavelengths of light per pixel that are typically used for full spectrum light reception. An example of a CFA is a Bayer pattern. Since the color sensor may include pixels within the array that are sensitive to a single color within the entire spectrum, an image with reduced resolution results from the pixel array having pixel space dedicated only to a single color of light within the entire spectrum. Typically, such an arrangement is formed in a checkerboard type pattern across the entire array.

[0109] In contrast, when a partitioned spectrum is used, the sensor can be made sensitive or responsive to the magnitude of all light energy because the pixel array will be instructed to sense electromagnetic energy from a predetermined partition of the full spectrum of electromagnetic energy in each cycle. Therefore, to form an image, the sensor only needs to cycle from multiple different partitions within the full spectrum and then reassemble the image to display a predetermined mixture of color values ​​for each pixel in the array. Therefore, a higher resolution image is also provided because the distance is reduced compared to a Bayer sensor between the centers of pixels with the same color sensitivity in each color pulse. Therefore, the formed color image has a higher modulation transfer function (MTF). Because the image from each color partition frame cycle has a higher resolution, the resulting image produced when the partitioned light frames are merged into a full color frame also has a higher resolution. In other words, because each pixel within the array (rather than, at most, every other pixel in a sensor with a color filter) senses the magnitude of energy for a given pulse and a given scene, only a period of time apart, a higher resolution image is produced for each scene that requires the introduction of less derived (less accurate) data.

[0110] For example, white or full spectrum visible light is a combination of red, green, and blue light. Figure 6 In the embodiment shown in , it can be seen that in the partitioned spectrum method 620 and the full spectrum method 610, the time of capturing the image is t(0) to t(1). In the full spectrum method 610, white light or full spectrum electromagnetic energy is emitted at 612. At 614, the white or full spectrum electromagnetic energy is sensed. At 616, the image is processed and displayed. Therefore, between time t(0) and t(1), the image has been processed and displayed. In contrast, in the partitioned spectrum method 620, the first partition is emitted at 622 and sensed at 624. At 626, the second partition is emitted and then sensed at 628. At 630, the third partition is emitted and sensed at 632. At 634, the image is processed and displayed. It should be understood that any system using an image sensor cycle that is at least twice faster than the white light cycle is intended to fall within the scope of the present disclosure.

[0111] As in Figure 6 In the embodiment shown in FIG. 6 , it can be seen graphically between time t(0) and t(1) that the sensors used in the partitioned spectrum system 620 cycle three times for each of the full spectrum systems. In the partitioned spectrum system 620, the first of three sensor cycles is for the green spectrum 622 and 624, the second of three sensor cycles is for the red spectrum 626 and 628, and the third is for the blue spectrum 630 and 632. Therefore, in an embodiment where the display device (LCD panel) operates at 50-60 frames per second, the partitioned light system should operate at 150-180 frames per second to maintain continuity and smoothness of the displayed video.

[0112] In other embodiments, there may be different capture and display frame rates. In addition, the average capture rate may be any multiple of the display rate.

[0113] In one embodiment, it may be desirable that not all partitions are represented equally within the system frame rate. In other words, not all light sources must be pulsed with the same regularity in order to emphasize and de-emphasize various aspects of the recorded scene as desired by the user. It should also be understood that invisible and visible partitions of the electromagnetic spectrum can be pulsed together within the system, with their corresponding data values ​​stitched into the video output for display to the user.

[0114] Embodiments may include the following pulse cycle patterns:

[0115] i. Green pulse;

[0116] ii. Red pulse;

[0117] iii. Blue pulse;

[0118] iv. Green pulse;

[0119] v.Red pulse;

[0120] vi.Blue pulse;

[0121] vii. Infrared (IR) pulses;

[0122] viii.(repeat)

[0123] As can be seen in the examples, infrared partitions or specific wavelength partitions (e.g., 513nm-545nm, 565nm-585nm, and / or 900nm-100nm) can be pulsed at a different rate than the other partitions. Doing so can emphasize a certain aspect of a scene, where the IR data is simply overlapped with other data in the video output to make the desired emphasis. It should be noted that adding electromagnetic partitions on top of the red, green, and blue partitions does not necessarily require a serialized system to operate at four times the rate of a full-spectrum non-serial system, because each partition does not have to be equally represented in the pulse pattern. As seen in this embodiment, adding a partition pulse that represents fewer partitions in the pulse pattern (infrared in the above example) will result in an increase in the sensor's cycle speed by less than 20% in order to accommodate irregular partition sampling.

[0124] In one embodiment, electromagnetic partitions sensitive to dyes or materials used to highlight aspects of a scene may be emitted. In this embodiment, the location of the dye or material may be highlighted adequately without requiring high resolution. In this embodiment, the dye sensitive electromagnetic partitions may be cycled much less frequently than other partitions in the system in order to include the emphasized data.

[0125] In various embodiments, the pulse cycle pattern may include any of the following wavelengths in any suitable order. Such wavelengths may be particularly useful for determining multispectral or hyperspectral image data or for determining image data based on relaxation emission of fluorescent agents:

[0126] i.465±5nm;

[0127] ii.533±4nm;

[0128] iii.638±5nm;

[0129] iv.780±5nm;

[0130] v.805±5nm;

[0131] vi.975±5nm;

[0132] vii. 577 ± 2 nm; or

[0133] viii.523±4nm.

[0134] The partition loop can be divided into various imaging and video standards to accommodate or approximate. In one embodiment, the partition loop may include the following: FIG. 7A to FIG. 7D Pulses of electromagnetic energy in the red, green, and blue spectrum are best shown in Fig. 7A In , different light intensities have been achieved by modulating the light pulse width or duration within the operating range shown by the vertical grey dashed line. Figure 7BIn , different light intensities have been achieved by modulating the power of an optical or electromagnetic emitter (which may be a laser or LED emitter), but keeping the pulse width or duration constant. Figure 7C The case where both the light power and light pulse width are modulated is shown to allow for greater flexibility. The zone cycle can use CMY, IR and UV using invisible pulse sources mixed with visible pulse sources and any other color space required to produce an image or approximate a desired video standard currently known or yet to be developed. It should also be understood that the system can switch between color spaces on the fly to provide the desired image output quality.

[0135] When using the color space green-blue-green-red (such as Fig.7D In an embodiment (as shown in ), it may be desirable to pulse the luma component more frequently than the chroma component because users are generally more sensitive to differences in light magnitude than to differences in light color. Fig.7D The monochrome sensor shown in Figure utilises this principle. Fig.7D In the scheme, green, which contains the most brightness information, can be pulsed more frequently or with greater intensity to obtain brightness data in the (GBGRGBGR...) scheme. Such a configuration will create a video stream with more perceptible details without creating and transmitting imperceptible data.

[0136] In one embodiment, duplicate pulses of weaker partitions may be used to produce an output that has been adjusted for the weaker pulses. For example, given the weak sensitivity of blue lasers relative to silicon-based pixels, and the difficulty of producing blue lasers compared to red or green light, it may be pulsed more frequently during a frame cycle to compensate for the weakness of the light. These additional pulses may be done continuously over time, or by using multiple lasers to pulse simultaneously to produce the desired compensating effect. It should be noted that by pulsing during blanking periods (time during which the sensor is not reading out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same kind, and simply accumulates the light for the desired output. In another embodiment, the maximum light pulse range may be different from frame to frame. This may be done in a controlled manner. Fig. 7E where the light pulses are different from frame to frame. The sensor can be constructed to be able to program different blanking times in a repeating pattern of 2 frames or 3 frames or 4 frames or n frames. Fig. 7E In , 4 different light pulses are shown, and pulse 1 may be repeated, for example, after pulse 4, and may have a pattern of 4 frames with different blanking times. This technique may be used to place the most powerful partition at the minimum blanking time, and thus allow the weakest partition to have a wider pulse on one of the subsequent frames without increasing the readout speed. The reconstructed frame may still have a regular pattern from frame to frame, since it is made up of many pulse frames.

[0137] As in Figure 8 As can be seen in , because each partitioned spectrum of light can have a different energy value, the sensor and / or light emitter can be adjusted to compensate for the difference in energy values. At 810, data obtained from a histogram from a previous frame can be analyzed. At 820, the sensor can be adjusted as described below. Additionally, at 830, the emitter can be adjusted. At 840, an image can be acquired from the adjusted sample time from the sensor, or an image can be acquired using adjusted (increased or decreased) emitted light, or a combination of the above. For example, because the red light spectrum is more easily detected by a sensor within the system than the blue light spectrum, the sensor can be adjusted to be less sensitive during the red partition cycle and more sensitive during the blue partition cycle because the blue partition has a low quantum efficiency relative to silicon (in Fig. 9 ). Similarly, the emitters can be adjusted to provide adjusted zones (e.g., higher or lower intensity and duration). In addition, adjustments can be made at both the sensor and emitter levels. The emitters can also be designed to emit at one specific frequency, or can be changed to emit multiple frequencies for a specific zone to broaden the spectrum of light emitted, if required for a particular application.

[0138] Fig.10 A schematic diagram of an unshared 4T pixel is shown. The TX signal is used to transfer the accumulated charge from the photodiode (PPD) to the floating diffusion (FD). The reset signal is used to reset the FD to the reset bus. If the reset signal and the TX signal are "on" at the same time, the PPD is constantly reset (each photocharge generated in the PPD is directly collected on the reset bus) and the PPD is always empty. A common pixel array implementation includes a horizontal reset row to which the reset signal of all pixels in a row is attached and a horizontal TX row to which the TX signal of all pixels in a row is attached.

[0139] In one embodiment, the timing of sensor sensitivity adjustment is shown and can be implemented using a global reset mechanism (i.e., a device that activates all pixel array reset signals at once) and a global TX mechanism (i.e., a device that activates all pixel array TX signals at once). This is shown in Fig.11 In this case, the duration and amplitude of the light pulse are constant, but the light integrated in all pixels starts with the "on" to "off" transition of the global TX and ends with the light pulse. Therefore, modulation is achieved by shifting the falling edge of the global TX pulse.

[0140] Conversely, the emitter can emit red light at a lower intensity than blue light to produce a correctly exposed image (in Fig.12). At 1210, data obtained from the histogram from the previous frame can be analyzed. At 1220, the emitter can be adjusted. At 1230, an image can be obtained from the adjusted emitted light. Additionally, in one embodiment, both the emitter and the sensor can be adjusted simultaneously.

[0141] In some embodiments, reconstructing the partitioned spectral frame into a full spectral frame for later output can be as simple as blending the sensed values ​​of each pixel in the array. Additionally, the blending and mixing of the values ​​can be a simple average, or can be tuned to the values ​​of a predetermined lookup table (LUT) for the desired output. In embodiments of systems using partitioned spectroscopy, the sensed values ​​can be post-processed or further refined remotely from the sensor by an image or auxiliary processor, and just before outputting them to a display.

[0142] Fig.13 A basic example of a monochrome ISP is shown at 1300, and how an ISP chain may be assembled for the purpose of generating an sRGB image sequence from raw sensor data produced in the presence of a GRGB light pulsing scheme.

[0143] The first stage involves making corrections (see Fig.13 1302, 1304 and 1306 in ) to account for any non-ideal factors in the sensor technology that are best suited to work in the raw data domain (see Fig.21 ).

[0144] In the next stage, two frames are buffered (see Fig.13 1308 and 1310 in FIG. 1306 ), because each final frame derives data from three original frames. Frame reconstruction at 1314 will continue by sampling data from the current frame and the two buffered frames (1308 and / or 1310). The reconstruction process results in a full-color frame in a linear RGB color space.

[0145] In this example, white balance coefficients at 1318 and a color correction matrix at 1320 are applied before conversion to YCbCr space at 1322 for subsequent edge enhancement at 1324. After edge enhancement at 1324, the image is converted back to linear RGB at 1326 for scaling at 1328 (if applicable).

[0146] Finally, a gamma transfer function is applied at 1330 to convert the data to the sRGB domain at 1332 .

[0147] Fig.141404 is an example of an implementation scheme of color fusion hardware. The color fusion hardware takes an RGBGRGBGRGBG video data stream at 1402 and converts it into a parallel RGB video data stream at 1405. The bit width on the input side may be, for example, 12 bits per color. The output width of this example will be 36 bits per pixel. Other implementation schemes may have different initial bit widths and a number 3 times the output width. At 1402, the memory writer block takes the RGBG video stream as its input and writes each frame to its correct frame memory buffer at 1404 (the memory writer triggers the same pulse generator 1410 as the pulse generator that runs the laser source). As shown at 1404, writing to memory follows the pattern red, green 1, blue, green 2, and then starts back again with red. At 1406, the memory reader reads three frames at a time to construct RGB pixels. Each pixel is three times the bit width of a single color component. At 1410, the reader also triggers the laser pulse generator. The reader waits until the Red, Green 1, and Blue frames have been written, then continues to read them out in parallel while the writer continues to write Green 2 and begins returning to Red. When Red is finished, the reader begins reading from Blue, Green 2, and Red. This pattern continues indefinitely.

[0148] See now Fig.15 and Fig.16 As shown, in one embodiment, Fig.16 The RG1BG2RG1BG2 pattern reconstruction shown in allows 60fps output and 120fps input. Each successive frame contains either the red component or the blue component from the previous frame. Fig.16 In , each color component is available within 8.3 ms, and the resulting reconstructed frame has a period of 16.67 ms. In general, for this pulse scheme, the reconstructed frame has a period of Fig.15 In other embodiments, different pulse schemes may be used. For example, an embodiment may be based on the timing of each color component or frame (T1), and the reconstructed frame has a period that is twice the period of the input color frame (2×T1). Different frames within a sequence may have different frame periods, and the average capture rate may be any multiple of the final frame rate.

[0149] Figures 17 to 20A color correction method and hardware schematic for use with a zoned light system are shown. It is common in digital imaging to regulate values ​​within the image data to correct the output to meet the user's expectations or highlight certain aspects of the imaged object. Most commonly, this is done in satellite imagery that is tuned and adjusted to emphasize one type of data over another. Most commonly, there is a full spectrum of electromagnetic energy available in satellite acquired data because the light source is not controlled, i.e., the sun is the light source. In contrast, there are imaging conditions where the light is controlled and even provided by the user. In this case, calibration of the image data is still desirable because, without calibration, incorrect emphasis may be given to certain data over other data. In systems where the light is controlled by the user, it is advantageous to provide light emissions that are known to the user and may be only a portion of the electromagnetic spectrum or multiple portions of the full electromagnetic spectrum. Calibration is still important to meet the user's expectations and check for faults within the system. One calibration method may be a table of expected values ​​for a given imaging condition that can be compared to the data from the sensor. One embodiment may include a neutral color scene with known values ​​that should be output by the imaging device, and when the device samples the neutral color scene, the device may be adjusted to meet these known values.

[0150] In use, and at startup, the system may sample a neutral color scene at 1710 by running a full cycle of multiple electromagnetic spectrum partitions at 1702 (e.g., Fig.17 ). A value table 1708 may be formed to generate a histogram of the frame at 1704. At 1706, the values ​​of the frame may be compared to known values ​​or expected values ​​from a neutral color scene. The imaging device may then be adjusted at 1712 to meet the desired output. Fig.17 In the embodiment shown in , the system may include an image signal processor (ISP) that can be adjusted to perform color correction on the imaging device.

[0151] It should be noted that because each partitioned spectrum of light can have different energy values, the sensor and / or light emitter can be adjusted to compensate for the difference in energy values. For example, in one embodiment, because the blue light spectrum has a lower quantum efficiency than the red light spectrum with respect to silicon-based imagers, the responsiveness of the sensor can then be adjusted to respond less during the red cycle and more during the blue cycle. Conversely, because the quantum efficiency of blue light is lower than that of red light, the emitter can emit blue light at a higher intensity than red light to produce a correctly exposed image.

[0152] exist Fig.18In the embodiment shown in , where the system provides and controls the light source emissions, these light emissions can be adjusted to color correct the image at 1800. Any aspect of the emitted light can be adjusted, such as the amount, duration (i.e., on time), or range within a spectral partition. In addition, in some embodiments, both the emitter and the sensor can be adjusted simultaneously, such as Fig.19 as shown in .

[0153] To reduce the amount of noise and artifacts in the output image stream or video, the sensors or transmitters in the system can be adjusted in stages, such as in Fig. 20 Visible in. Fig. 20 2000 is shown in which both the emitter 2006 and the sensor 2008 are adjustable, but imaging devices in which one of the emitter or sensor is adjusted during use or for a portion of use are also contemplated and are within the scope of the present disclosure. It may be advantageous to adjust only the emitter during a portion of use and only the sensor during another portion of use, while further adjusting both simultaneously during a portion of use. In any of the above embodiments, improved image quality may be obtained by limiting the overall adjustments that the system may make between frame cycles. In other words, an embodiment may be limited so that only a portion of the operating range of the emitter can be adjusted at any time between frames. Similarly, the sensor may be limited so that only a portion of its operating range can be adjusted at any time between frames. Furthermore, in one embodiment both the emitter and the sensor may be limited so that only a portion of their respective operating ranges can be adjusted together at any time between frames.

[0154] In an exemplary embodiment, a graded adjustment of a component within the system may be performed, for example, at about 0.1 dB of the operating range of the component to correct the exposure of the previous frame. 0.1 dB is merely an example, and it should be noted that in other embodiments, the allowable adjustment of a component may be any portion of its corresponding operating range. The components of the system may be changed by intensity or duration adjustments that are typically controlled by the number of bits (resolution) output by the component. The component resolution may typically range from about 10 bits to 24 bits, but should not be limited to this range because it is intended to include the resolution of devices yet to be developed in addition to those currently available. For example, after the first frame, it is determined that the scene is too blue when being observed, then the emitter may be adjusted by a graded adjustment as described above (such as about 0.1 dB) to reduce the pulse magnitude or duration of blue light during the blue cycle of the system.

[0155] In this exemplary embodiment, more than 10% may be needed, but the system has limited itself to an operating range of 0.1 dB adjustment per system cycle. Therefore, the blue light can be adjusted again during the next system cycle if necessary. The stepped adjustment between cycles can have a damping effect on the output image, and will reduce noise and artifacts when operating the emitters and sensors at their operating limits. Any stepped amount that can determine the operating adjustment range of a component can be used as a limiting factor, or it can be determined that certain embodiments of the system may include components that are adjustable throughout their operating range.

[0156] Additionally, the optical black area of ​​any image sensor can be used to aid in image correction and noise reduction. In one embodiment, the values ​​read from the optical black area can be compared with those of the active pixel area of ​​the sensor to establish a reference point to be used for image data processing. Fig.21 The types of sensor correction methods that can be adopted in color pulse systems are shown. CMOS image sensors usually have multiple non-ideal factors that have an adverse effect on image quality, especially in low light. The main ones are fixed pattern noise and row noise. Fixed pattern noise is the dispersion in the offset of the sensing element. Usually, most of the FPN is inter-pixel dispersion, which comes from the random variation of dark current from photodiode to photodiode and other sources. This looks very unnatural to the observer. Even more noteworthy is column FPN, which is caused by the offset in the readout chain associated with a specific column of pixels. This causes perceived vertical stripes within the image.

[0157] Fully controlling the illumination has the following benefits: An entire dark data frame can be periodically acquired and used to correct for pixel and column offsets. In the illustrated example, a single frame buffer can be used to perform a sliding average over the entire frame in the absence of light using, for example, a simple exponential smoothing filter. During normal operation, this dark average frame will be subtracted from each illuminated frame.

[0158] Row noise is the random temporal variation of pixel offsets within each row. Since it is temporal, the correction must be recalculated for each row and each frame. To do this, there are typically a number of light-blind (OB) pixels within each row in the array that must first be sampled to estimate the row offset before sampling the light-sensitive pixels. The row offset is then simply subtracted during the row noise correction process.

[0159] exist Fig.21 In the example in , there are other corrections involving obtaining data in the correct order, monitoring and controlling voltage offsets in the analog domain (black clamping), and identifying / correcting individual defective pixels.

[0160] Fig. 22 and Fig.23A method and hardware schematic for increasing the dynamic range in a closed environment or limited light environment is shown. In one embodiment, the exposure input can be input at different levels over time and combined to produce a larger dynamic range. Fig. 22 As can be seen in , the imaging system may cycle a first cycle at a first intensity at 2202, then cycle a second cycle at a second intensity at 2204, and then by combining these first and second cycles into a single frame at 2206, a greater dynamic range may be achieved. A greater dynamic range may be particularly desirable due to the limited spatial environment in which the imaging device is used. In a limited spatial environment that lacks light or is dark except for the light provided by the light source, and where the light source is close to the light emitter, exposure has an exponential relationship with distance. For example, objects close to the light source and the optical opening of the imaging device tend to be overexposed, while objects farther away tend to be extremely underexposed because there is very little (any) ambient light.

[0161] As in Fig.23 As can be seen in FIG. 2 , the loop of a system having electromagnetic energy emission in multiple partitions may be continuously looped according to the partitioning of the electromagnetic spectrum at 2300. For example, in an embodiment where the emitter emits lasers in different red partitions, different blue partitions, and different green partitions, the two loop data sets to be combined may be of the following form:

[0162] i. Red with intensity of one at 2302,

[0163] ii. Red with an intensity of two at 2304,

[0164] iii. Blue with intensity of one at 2302,

[0165] iv. Blue with intensity two at 2304,

[0166] v. Green with intensity of one at 2302,

[0167] vi. Green with intensity two at 2304.

[0168] Alternatively, the system can loop as follows:

[0169] i. Red with intensity of one at 2302,

[0170] ii. Blue with intensity of one at 2302,

[0171] iii. Green with intensity of one at 2302,

[0172] iv. Red with an intensity of two at 2304,

[0173] v. Blue with intensity 2 at 2304,

[0174] vi. Green with intensity two at 2304.

[0175] In such embodiments, the first image may be derived from intensity one values ​​and the second image may be derived from intensity two values ​​and then combined or processed into a complete image dataset at 2310 rather than at its component parts.

[0176] It is contemplated within the scope of the present invention that any number of transmit partitions may be used in any order. Fig.23 As shown in , "n" is used as a variable to represent any number of electromagnetic partitions, and "m" is used to represent any level of strength of the "n" partitions. Such a system can be looped as follows:

[0177] i. n with intensity m at 2306,

[0178] ii. n+1 with strength m+1,

[0179] iii. n+2 with strength m+2,

[0180] iv. n+i with intensity m+j at 2308.

[0181] Thus, any sequenced cyclic pattern may be used to produce the desired image correction, where "i" and "j" are additive values ​​within the operating range of the imaging system.

[0182] To maximize the fidelity of color reproduction, digital color cameras contain an image processing stage. This is achieved through a 3×3 matrix called the color correction matrix (CCM):

[0183]

[0184] The terms in the CCM are tuned using a set of reference colors (e.g., from a Macbeth chart) to provide the best overall match to the sRGB standard color space. The diagonal terms a, e, and i are effectively white balance gains. However, typically, white balance is applied separately, and the sum of the rows is constrained to unity gain so that the CCM itself applies no net gain. The off-diagonal terms effectively handle color crosstalk in the input channels. Therefore, Bayer sensors have higher off-diagonals than 3-chip cameras because the color filter array has a large stack of responses between channels.

[0185] There is a signal-to-noise ratio penalty for color correction, which depends on the magnitude of the off-diagonal terms. A hypothetical sensor with channels that exactly match the sRGB components would have the identification matrix CCM:

[0186]

[0187] In this case, for a pure white photon signal of 10,000e / pixel (ignoring read noise), the signal-to-noise ratio evaluated in the green channel would be:

[0188]

[0189] Any deviation will reduce the SNR. Take for example the following CCM, which has values ​​that are not uncommon for Bayer CMOS sensors:

[0190]

[0191] In this case, the green SNR is:

[0192]

[0193] Fig.24 Results of full SNR simulation using D65 illumination for a typical Bayer sensor CCM using the identification matrix vs. tuned CCM are shown. The SNR evaluated for the luma component is about 6dB different as a result of color correction.

[0194] The system described in the present invention uses monochromatic illumination at multiple discrete wavelengths and therefore inherently has no color crosstalk. Fig.25 The crosses in represent the positions of the three wavelengths (465 nm, 532 nm and 639 nm) that can be obtained via a laser diode source, compared with the sRGB color gamut represented by the triangles.

[0195] Compared to a Bayer sensor, the off-diagonal terms of the CCM are significantly reduced in this case, which provides a significant SNR advantage.

[0196] Fig.26 An imaging system with increased dynamic range provided by the pixel configuration of the pixel array of the image sensor is shown. As can be seen from the figure, adjacent pixels 2602 and 2604 can be set at different sensitivities so that each cycle includes data generated by more sensitive and less sensitive pixels relative to each other. Since multiple sensitivities can be recorded in a single cycle of the array, the dynamic range can be increased with parallel recording, which is contrary to the time-dependent serial nature of other embodiments.

[0197] In one embodiment, the array may include rows of pixels that may be placed in rows based on their sensitivity. In one embodiment, based on those sensitivities, pixels of different sensitivities may alternate within rows or columns relative to their nearest neighbors to form a checkerboard pattern across the array. The above may be accomplished with any pixel circuit sharing arrangement or with any independent pixel circuit arrangement.

[0198] Wide dynamic range can be achieved by having multiple global TXs, each firing only on different pixel groups. For example, in global mode, the global TX1 signal is firing pixel group 1, the global TX2 signal is firing pixel group 2, and the global TXn signal is firing pixel group n.

[0199] based on Fig.11 , Fig.27A An example of the timing of 2 different pixel sensitivities (dual pixel sensitivity) in a pixel array is shown. In this case, the global TX1 signal fires half of the pixels of the array and the global TX2 fires the other half of the pixels. Because global TX1 and global TX2 have different "on" to "off" edge locations, the integrated light is different between TX1 pixels and TX2 pixels. Fig.27B Different implementations of timing for dual pixel sensitivity are shown. In this case, the light pulse is modulated twice (pulse duration and / or amplitude). The TX1 pixel integrates the P1 pulse, and the TX2 pixel integrates the P1+P2 pulse. Separating the global TX signal can be done in a variety of ways. The following are examples:

[0200] i. Distinguish the TX row from each row; and

[0201] ii. Multiple TX lines are sent per row, each TX line addresses a different group of pixels.

[0202] In one specific implementation, a device is described that provides wide dynamic range video that utilizes the color pulse system described in the present disclosure. The basis is pixels with multiple characteristics or pixels that can be tuned in different ways within the same monochrome array, which are able to integrate incident light of different durations within the same frame. An example of the arrangement of pixels in an array of such a sensor will always be a uniform checkerboard pattern with two independently variable integration times. For this case, both red and blue information can be provided within the same frame. In fact, this can be done while extending the dynamic range of the green frame, which is most needed at this time, because the two integration times can be adjusted on a frame-by-frame basis. The beneficial effect is that color motion artifacts are not a problem if all data comes from two frames instead of three frames. Of course, there is a subsequent loss of spatial resolution of the red and blue data, but that has less impact on image quality than green because the brightness component is dominated by the green data.

[0203] An inherent property of a monochrome wide dynamic range (WDR) array is that pixels with long integration times must integrate a superset of the light seen by short integration time pixels. For conventional wide dynamic range operation in the green frame, this is desirable. For red and blue frames, this means that the pulses must be controlled in conjunction with the exposure cycle to, for example, provide blue light from the start of the long exposure and switch to red light at the point where the short exposure pixel is turned on (charge for both pixel types is transferred simultaneously).

[0204] In the color fusion stage, the pixels of both characteristics are divided into two buffers. Empty pixels are then filled using, for example, linear interpolation. At this point, one buffer contains the full image of blue data, while the other contains red+blue. The blue buffer can be subtracted from the second buffer to give pure red data.

[0205] FIG. 28A to FIG. 28C The use of pulsed and / or synchronized or constant white light emission and corresponding color sensors is shown. Fig.28A As can be seen in FIG. 1 , the white light emitter can be configured to emit a light beam during the blanking period of the corresponding sensor to provide a controlled light source in a controlled light environment. The light source can emit a light beam of constant magnitude and vary the duration of the pulse, such as Fig.28A Alternatively, the pulse may be kept constant while varying the magnitude to achieve correctly exposed data, such as Fig.28B shown. Fig.28C Shown in is a graphical representation of a constant light source that can be modulated with a varying current controlled by and synchronized with a sensor.

[0206] In one embodiment, white light or multi-spectral light can be emitted as pulses, if desired, to provide data for use within the system (in FIG. 28A to FIG. 28C ). White light emission combined with partitioning of the electromagnetic spectrum can be used to emphasize and de-emphasize certain aspects within a scene. Such an embodiment may use the following pulse pattern:

[0207] i. Green pulse;

[0208] ii. Red pulse;

[0209] iii. Blue pulse;

[0210] iv. Green pulse;

[0211] v.Red pulse;

[0212] vi.Blue pulse;

[0213] vii. White light (multi-spectrum) pulses;

[0214] viii.(repeat)

[0215] Any system using an image sensor cycle at least twice faster than white light cycle is intended to fall within the scope of the present disclosure.It should be understood that any combination of partitions of the electromagnetic spectrum are contemplated herein, whether from the visible or invisible spectrum of the full electromagnetic spectrum.

[0216] Fig.29A and Fig.29B A perspective view and a side view, respectively, of an implementation of a monolithic sensor 2900 having multiple pixel arrays for producing a three-dimensional image in accordance with the teachings and principles of the present disclosure are shown. Such an implementation may be desirable for three-dimensional image capture, where the two pixel arrays 2902 and 2904 may be offset during use. In another implementation, the first pixel array 2902 and the second pixel array 2904 may be dedicated to receiving electromagnetic radiation of a predetermined wavelength range, where the first pixel array is dedicated to electromagnetic radiation of a different wavelength range than the second pixel array.

[0217] Fig. 30A and Fig. 30BA perspective view and a side view of a specific implementation of an imaging sensor 3000 built on multiple substrates are shown respectively. As shown, multiple pixel columns 3004 forming the pixel array are located on a first substrate 3002 and multiple circuit columns 3008 are located on a second substrate 3006. The figure also shows the electrical connection and communication between a pixel column and its associated or corresponding circuit column. In a specific implementation, the image sensor may have a pixel array separated from all or most of the supporting circuits, while it may be otherwise manufactured so that its pixel array and supporting circuits are on a single, monolithic substrate / chip. The present disclosure may use at least two substrates / chips, which will be stacked together using three-dimensional stacking technology. The first 3002 of the two substrates / chips may be processed using an image CMOS process. The first substrate / chip 3002 may consist of only a pixel array, or may consist of a pixel array surrounded by a limited circuit. The second or subsequent substrate / chip 3006 may be processed using any process, rather than necessarily from an image CMOS process. The second substrate / chip 3006 may be, but is not limited to, a high density digital process for integrating various and multiple functions into a very limited space or area on the substrate / chip, or a mixed mode or analog process for integrating, for example, precise analog functions, or an RF process for implementing wireless capabilities, or a MEMS (micro-electro-mechanical system) for integrating MEMS devices. The image CMOS substrate / chip 3002 may be stacked with the second or subsequent substrate / chip 3006 using any three-dimensional technology. The second substrate / chip 3006 may support a large portion or a large portion of the circuitry that would otherwise be implemented in the first image CMOS chip 3002 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increase the overall system area while keeping the pixel array size constant and optimized as much as possible. The electrical connection between the two substrates / chips may be accomplished through interconnects 3003 and 3005, which may be bonding wires, lugs, and / or TSVs (through silicon vias).

[0218] Fig.31A and Fig.31B A perspective view and a side view of a specific implementation of an imaging sensor 3100 having multiple pixel arrays for producing a three-dimensional image are shown, respectively. The three-dimensional image sensor can be constructed on multiple substrates and can include multiple pixel arrays and other associated circuits, wherein multiple pixel columns 3104a forming a first pixel array and multiple pixel columns 3104b forming a second pixel array are located on respective substrates 3102a and 3102b, respectively, and multiple circuit columns 3108a and 3108b are located on a separate substrate 3106. Electrical connections and communications between pixel columns and associated or corresponding circuit columns are also shown.

[0219] It should be understood that the teachings and principles of the present disclosure can be used for reusable device platforms, limited-use device platforms, reconfigurable device platforms, or single-use / disposable device platforms without departing from the scope of the present disclosure. It should be understood that in a reusable device platform, the end user is responsible for the cleaning and disinfection of the device. In a limited-use device platform, the device can be used a specified number of times before becoming inoperable. A typical new device has been sterilized before delivery and should be cleaned and disinfected by the end user before other uses if it is to be used for other purposes. In a reconfigurable device platform, a third party can reprocess the device (e.g., clean, package, and disinfect) a single-use device for additional use at a lower cost than a new unit. In a single-use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0220] One embodiment of the emitter may use a mechanical shutter and filters to produce pulsed color light. Fig.32 As shown in, an alternative method of generating pulsed color light using a white light source and a mechanical color filter and shutter system 3200 is provided. The wheel may contain a pattern of semi-transparent color filter windows and opaque portions for shielding. The opaque portions will not allow light to pass through and will produce dark periods where sensor readout may occur. The white light source may be based on any technology: laser, LED, xenon, halogen, metal halide or other technology. White light may be projected through a series of color filters 3207, 3209 and 3211 of the desired color light pulse pattern. The pattern of one embodiment may be a red filter 3207, a green filter 3209, a blue filter 3211, and a green filter 3209. The filter and shutter system 3200 may be arranged on a wheel that rotates at a desired frequency to synchronize with the sensor, so that the knowledge of the arch length and rotation rate of the mechanical color filters 3207, 3209 and 3211 and the shutter 3205 system will provide timing information for the operation of the corresponding monochrome image sensor.

[0221] like Fig.33As shown in , one embodiment may include a pattern of only semi-transparent filters 3307, 3309 and 3311 on the filter wheel 3300. In this configuration, different shutters may be used. The shutter may be mechanical, and the "pulse" duration may be dynamically adjusted by changing its size. Alternatively, the shutter may be electronic and incorporated into the sensor design. The motor that rotates the filter wheel 3300 will need to communicate with or be controlled in conjunction with the sensor so that knowledge of the arch length and rotation rate of the mechanical filter 3307, 3309 and 3311 system will provide timing information for the operation of the corresponding monochrome image sensor. The control system will need to know the correct color filter for each frame captured by the sensor so that a full-color image can be correctly reconstructed in the ISP. An RGBG color pattern is shown, but other colors and / or patterns may be used if advantageous. The relative sizes of the color segments are shown as equal, but may also be adjusted if advantageous. The mechanical structure of the filter is shown as a circle with rotational movement, but may be a rectangle with linear movement, or a different shape with different movement patterns.

[0222] like Fig.34 As shown, the embodiment for pulsed color light can be composed of a mechanical wheel or cylinder that holds the electronics and heat sink of a red, green, blue or white LED. The LED will be spaced apart at a distance related to the rotation or torsion rate of the cylinder or wheel to allow timing of light pulses according to other embodiments in this patent. The wheel or cylinder will rotate using an electric motor and a mechanical bracket that attaches the wheel or cylinder to the electric motor. The motor will be controlled using a microcontroller, FPGA, DSP or other programmable device that will contain a control algorithm for correct timing, as described in this patent. There will be a mechanical opening on one side that will optically couple to an optical fiber to transmit the fiber to the end of the range using the method described in this patent. The coupling piece may also have a mechanical aperture that can be opened and closed to control the amount of light allowed to flow down the fiber cable. This will be a mechanical shutter device, alternatively an electronic shutter designed to a CMOS or CCD type sensor may be used. This device will be difficult to control and calibrate in production, but this is another way that people can get pulsed light into our system.

[0223] Fig.35 An embodiment of an emitter 3502 is shown in FIG. 3 , which includes a linear filter 3504 and a shutter mechanism for providing pulsed electromagnetic radiation. The linear filter 3504 and the shutter mechanism move horizontally at a desired frequency to filter the appropriate wavelength of light.

[0224] Fig.36, an embodiment of an emitter 3602 is shown that includes a prismatic filter 3604 and a shutter mechanism for providing pulsed electromagnetic radiation. The prismatic filter 3604 filters the light and delivers an output tank that may include a shutter. The prismatic filter 3604 moves at the required frequency to provide the correct color output pattern.

[0225] Additionally, the teachings and principles of the present disclosure may include any and all wavelengths of electromagnetic energy, including the visible and invisible spectrum, such as infrared (IR), ultraviolet (UV), and X-rays.

[0226] Fig.37 3700 is a schematic diagram showing a system for providing illumination to a light-deficient environment (such as for endoscopic imaging). System 3700 can be used in combination with any of the systems, methods or devices disclosed herein. System 3700 includes a light source 3702, a controller 3704, a jumper waveguide 3706, a waveguide connector 3708, an inner cavity waveguide 3710, an inner cavity 3712, and an image sensor 3714 with an accompanying optical component (such as a lens). Light source 3702 generates light that passes through jumper waveguide 3706 and inner cavity waveguide 3710 to illuminate the scene at the distal end of inner cavity 3712. Light source 3700 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared wavelengths, ultraviolet wavelengths or other wavelengths. Inner cavity 3712 can be inserted into a patient for imaging, such as during a procedure or examination. Light is output as shown by dashed line 3716. An image sensor 3714 can be used to capture a scene illuminated by light and display the scene to a doctor or some other medical personnel. Controller 3704 can provide control signals to light source 3702 to control when illumination is provided to a scene. In one embodiment, light source 3702 and controller 3704 are located in a camera control unit (CCU) or an external console to which the endoscope is connected. If image sensor 3714 comprises a CMOS sensor, light can be periodically provided to the scene in a series of illumination pulses between readout cycles of image sensor 3714 during so-called blanking periods. Thus, light can be pulsed in a controlled manner to avoid stacking into the readout cycles of image pixels in the pixel array of image sensor 3714.

[0227] In one embodiment, the intracavity waveguide 3710 includes one or more optical fibers. These optical fibers can be made of low-cost materials (such as plastic) to allow processing of the intracavity waveguide 3710 and / or other parts of the endoscope. In one embodiment, a single glass fiber with a diameter of 500 microns can be used. The jumper waveguide 3706 can be permanently attached to the light source 3702. For example, the jumper waveguide 3706 can receive light from a transmitter in the light source 3702 and provide light to the intracavity waveguide 3710 at the location of the connector 3708. In one embodiment, the jumper waveguide 106 may include one or more glass optical fibers. The jumper waveguide may include any other type of waveguide for guiding light to the intracavity waveguide 3710. The connector 3708 can selectively couple the jumper waveguide 3706 to the intracavity waveguide 3710 and allow the light in the jumper waveguide 3706 to pass through the intracavity waveguide 3710. In one embodiment, the intracavity waveguide 3710 can be coupled directly to the light source without any intervening jumper waveguide 3706.

[0228] Figures 38 to 40 is a schematic block diagram showing a light source 3800 having multiple emitters. Fig.38 , the emitters include a first emitter 3802, a second reflector 3804, and a third emitter 3806. Additional emitters may be included, as discussed further below. The emitters 3802, 3804, and 3806 may include one or more laser generators that emit light having different wavelengths. For example, the first emitter 3802 may emit a wavelength consistent with a blue laser, the third emitter 3804 may emit a wavelength consistent with a green laser, and the third emitter 3806 may emit a wavelength consistent with a red laser. For example, the first emitter 3802 may include one or more blue lasers, the second emitter 3804 may include one or more green lasers, and the third emitter 3806 may include one or more red lasers. The lasers 3802, 3804, 3806 emit laser beams toward a collection area 3808, which may be a waveguide, a lens, or a device for collecting light and / or transmitting light to a waveguide (such as Fig.37 The jumper waveguide 3706 or the intracavity waveguide 3710) provides a location for other optical components of the light.

[0229] In a specific implementation where a reagent or dye has been administered to a patient that aids in identifying certain tissues, structures, chemical reactions, biological processes, etc., the emitters 3802, 3804, and 3806 may emit wavelengths for causing the reagent or dye to fluoresce. Such wavelengths may be determined based on the reagent or dye administered to the patient. In such embodiments, the emitters may need to be highly accurate in order to emit the desired wavelengths to cause certain reagents or dyes to fluoresce or activate.

[0230] exist Fig.38In an embodiment, the emitters 3802, 3804, 3806 each deliver laser light to the collection area 3808 at a different angle. Changes in angle can result in changes in the location of the electromagnetic energy in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at the collection area 3808, the changing angles can result in different amounts of light entering different fibers. For example, the angles can result in intensity variations across the collection area 3808. In addition, the light from different emitters may not be mixed evenly, so some fibers may receive different amounts of different colors of light. Variations in the color or intensity of the light in different fibers can result in non-optimal illumination of the scene. For example, variations in the delivered light or light intensity can result in both the scene and the captured image.

[0231] In one embodiment, an intervening optical element may be placed between the fiber bundle and the transmitters 3802, 3804, 3806 to mix light of different colors (wavelengths) before entering the fiber or other waveguide. Exemplary intervening optical elements include diffusers, mixing rods, one or more lenses, or other optical components for mixing light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. This mixing may result in the same color in each fiber, but, in some embodiments, may still result in different total brightness delivered to different fibers. In one embodiment, the intervening optical element may also spread or evenly spread the light over the collection area so that each fiber carries the same total amount of light (e.g., the light may be spread out in a top hat profile). Diffusers or mixing rods may cause light loss.

[0232] Although collection area 3808 is Fig.38 3808 is represented as a physical component, but collection region 3808 may simply be an area that delivers light from emitters 3802, 3804, and 3806. In some cases, collection region 3808 may include optical components such as diffusers, mixing rods, lenses, or any other intervening optical components between emitters 3802, 3804, 3806 and the output waveguides.

[0233] Fig.39An embodiment of a light source 3800 is shown having emitters 3802, 3804, 3806 that provide light to a collection area 3808 at the same or substantially the same angle. The light is provided at an angle substantially perpendicular to the collection area 3808. The light source 3800 includes a plurality of dichroic mirrors, including a first dichroic mirror 3902, a second dichroic mirror 3904, and a third dichroic mirror 3906. The dichroic mirrors 3902, 3904, 3906 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 3906 may reflect blue laser light provided by a third emitter, while being transparent to red and green light provided by each of the first and second emitters 3802, 3804. The second dichroic mirror 3904 may be transparent to light from the first emitter 3802, but reflective to light from the second emitter 3804. If other colors or wavelengths are included, the dichroic mirrors may be selected to reflect light corresponding to at least one emitter and be transparent to other emitters. For example, the third dichroic mirror 3906 reflects light from the third emitter 3806, but is transparent to emitters "behind" it, such as the first emitter 3802 and the second emitter 3804. In embodiments where there are dozens or hundreds of emitters, each dichroic mirror may reflect the corresponding emitter and the emitter in front of it, while being transparent to the emitter behind it. This may allow dozens or hundreds of emitters to emit electromagnetic energy to the collection area 3808 at substantially the same angle.

[0234] Because these dichroic mirrors allow other wavelengths to be transmitted or passed through, each of these wavelengths can arrive at the collection area 3808 from the same angle and / or with the same center point or focus. Providing light from the same angle and / or the same focus / center point can significantly improve reception and color mixing at the collection area 3808. For example, a particular fiber can receive different colors in the same proportions as they are transmitted / reflected by the emitters 3802, 3804, 3806 and mirrors 3902, 3904, 3906. Fig.38 Compared to the embodiment of the present invention, the light mixing can be significantly improved at the collection area. In one embodiment, any optical component discussed herein can be used at the collection area 3808 to collect light before providing it to the fiber or fiber bundle.

[0235] Fig.40An embodiment of a light source 3800 is shown having emitters 3802, 3804, 3806 that also provide light to a collection area 3808 at the same or approximately the same angle. For example, the light incident on the collection area 3808 is offset from vertical. Angle 4002 indicates the angle of offset from vertical. In one embodiment, the laser emitters 3802, 3804, 3806 may have a Gaussian cross-sectional intensity profile. As previously described, improved distribution of light energy between fibers can be achieved by forming a flatter or top-hat shaped intensity profile. In one embodiment, as angle 4002 increases, the intensity across the collection area 3808 approaches a top-hat profile. For example, by increasing angle 4002 until the profile is sufficiently flat, a top-hat profile can even be approximated as a non-flat output beam.

[0236] A top hat profile may also be achieved using one or more lenses, diffusers, mixing rods, or any other intervening optical components between the emitters 3802, 3804, 3806 and the output waveguides, fibers, or fiber bundles.

[0237] Fig.41 4104. FIG. 4105 is a schematic diagram showing a single optical fiber 4102 outputted via a diffuser 4104 at the output. In one embodiment, the optical fiber 4102 may have a diameter of 500 microns and have a numerical aperture of 0.65 and emit a light cone 4106 of about 70 or 80 degrees without the diffuser 4104. With the diffuser 4104, the light cone 4106 may have an angle of about 110 or 120 degrees. The light cone 4106 may be the bulk of where all the light arrives and is evenly distributed. The diffuser 4104 may allow for a more even distribution of the electromagnetic energy of the scene observed by the image sensor.

[0238] In one embodiment, the intracavity waveguide 4102 may include a single plastic or glass optical fiber of approximately 500 microns. Plastic fibers are less expensive, but their width may allow the fiber to carry a sufficient amount of light to the scene due to coupling, diffusers, or other losses. For example, a smaller fiber may not be able to carry as much light or power as a larger fiber. The intracavity waveguide 3710 may include a single or multiple optical fibers. The intracavity waveguide 3702 may be directly from the light source or via a patch cord waveguide (e.g., see Fig.37 The light may be received by a jumper waveguide 3706. A diffuser may be used to broaden the light output 3706 to obtain a desired field of view for an image sensor 3714 or other optical component.

[0239] Although in Figures 38 to 40Three emitters are shown, but in some embodiments, a number ranging from one to hundreds or more emitters may be used. The emitters may have different wavelengths or spectra of light that they emit, and these lights may be used to continuously cover desired portions of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra).

[0240] In one embodiment, a light source with multiple emitters can be used for multi-spectral or hyper-spectral imaging in a light-deficient environment. For example, different chemicals, materials, or tissues may have different responses to electromagnetic energy of different colors or wavelengths. Some tissues have their own spectral characteristics (how they respond to or change the reflected wavelength of electromagnetic radiation). In one embodiment, a specific type of tissue can be detected based on how the tissue responds to a specific wavelength or a specific wavelength combination. For example, vascular tissue can absorb and reflect electromagnetic energy of different wavelengths or spectra in a unique way to distinguish it from muscle, fat, bone, nerve, ureter, or other tissues or substances in the body. In addition, specific types of muscle or other types of tissue can be distinguished based on their spectral response. The disease state of the tissue can also be determined based on spectral information. See U.S. Patent 8,289,503. See also U.S. Patent 8,158,957.

[0241] In one embodiment, fluorescence image data and / or multispectral or hyperspectral image data may be obtained using one or more filters to filter out all light or electromagnetic energy except light or electromagnetic energy in a desired wavelength or spectrum. Fig.42 4204 or other imaging medium (e.g., film). In one embodiment, white light 4208 passes through filter 4202, and filtered light 4210 passes through lens 4206 to be focused onto imaging sensor 4204 for image capture and readout. The filter may be located anywhere in the system, or may be an attribute of lens 4206 or image sensor 4204.

[0242] In a light-deficient environment, light 4208 may include white light emitted by an emitter in the light-deficient environment. Filter 4202 may be selected for the desired inspection. For example, if a specific tissue needs to be detected or highlighted, filter 4202 may be selected to allow wavelengths corresponding to the spectral response of a specific tissue or the fluorescent emission of a specific agent to pass through. Image sensor 4204 (which may include a monochrome image sensor) may generate an image. Pixels of the captured image that exceed a threshold or are below a threshold may then be characterized as corresponding to a specific tissue. This data may then be used to generate an image indicating the location of the specific tissue.

[0243] In another embodiment, fluorescent dyes or agents can be used to image specific tissue types, pathways, etc. in the body. For example, a fluorescent dye can be administered to a patient and an image of the dye can be captured. In one embodiment, electromagnetic energy of a specific wavelength can be used to trigger the dye to fluoresce. For example, the dye will only fluoresce when electromagnetic energy is present.

[0244] However, both filters and fluorescent dyes significantly constrain the examination. For example, if filters are used, the desired spectral response that can be detected and therefore the substances or tissues that can be detected are limited by the available filters. In addition, filters may need to be changed or replaced. In the case of dyes, the dyes must be applied before imaging, and there may be conflicts between different dyes applied for different purposes during the same examination. Therefore, examinations using filters and dyes may take a long time and may require many different examinations to obtain the desired information.

[0245] In one embodiment, a monochromatic image sensor and emitter that emits electromagnetic energy of multiple different wavelengths or spectra can be used to achieve multispectral or hyperspectral imaging in light-deficient environments. Figures 38 to 40 Any of the light sources 3800 may include multiple emitters to cover the desired spectrum.

[0246] Fig.43 A portion of the electromagnetic spectrum 4300 is shown divided into twenty different sub-spectra. The number of sub-spectra is exemplary only. In at least one embodiment, the spectrum 4300 can be divided into hundreds of sub-spectra, each with a small band. The spectrum can extend from the infrared spectrum 4302, through the visible spectrum 4304, and into the ultraviolet spectrum 4306. The sub-spectra each have a band 4308 covering a portion of the spectrum 4300. Each band can be defined by an upper wavelength and a lower wavelength.

[0247] In one embodiment, at least one emitter (such as a laser emitter) may be included in a light source (such as Figures 37 to 403702, 3800) in order to provide complete and continuous coverage of the entire spectrum 4300. For example, the light source for providing coverage of the sub-spectrum shown may include at least 20 different emitters, with at least one emitter for each sub-spectrum. In one embodiment, each emitter may cover a spectrum covering 40 nanometers. For example, one emitter may emit light in a band from 500nm to 540nm, while another emitter may emit light in a band from 540nm to 580nm. In another embodiment, the emitters may cover bands of other sizes, depending on the type of available emitters or imaging needs. For example, a plurality of emitters may include a first emitter covering a band from 500nm to 540nm, a second emitter covering a band from 540nm to 640nm, and a third emitter covering a band from 640nm to 650nm. Each emitter may cover a different segment of the electromagnetic spectrum ranging from far infrared, mid-infrared, near infrared, visible light, near ultraviolet and / or far ultraviolet. In some cases, multiple emitters of the same type or wavelength may be included to provide sufficient output power for imaging. The number of emitters required for a particular wavelength band may depend on the sensitivity of the monochromatic sensor to the wavelength band and / or the power output capability of the emitters in that wavelength band.

[0248] The bandwidth and coverage provided by the emitters can be selected to provide any desired combination of spectra. For example, continuous coverage of spectra using very small bandwidths (e.g., 10 nm or less) can allow highly selective hyperspectral imaging. Because the wavelengths come from emitters that can be selectively activated, extreme flexibility in determining the spectral response of a substance during an inspection can be achieved. Therefore, more information about the spectral response can be achieved in less time and within a single inspection, which would otherwise require multiple inspections, delays due to the application of dyes or stains, etc. In one embodiment, the system can capture hyperspectral image data and process the data to identify what type of tissue is present at each pixel.

[0249] Fig.44 4400 is a schematic diagram showing a timing diagram 4400 of emission and readout for generating a multispectral or hyperspectral image, according to one embodiment. The solid lines represent the readout (peak 4402) and blanking period (valley) for capturing a series of frames 4404-4414. The series of frames 4404-4414 may include a repeating series of frames that may be used to generate hyperspectral data for a video feed. The series of frames includes a first frame 404, a second frame 4406, a third frame 4408, a fourth frame 4410, a fifth frame 4412, and an Nth frame 4426.

[0250] In one embodiment, each frame is generated based on at least one pulse of electromagnetic energy. The electromagnetic energy pulses are reflected and detected by the image sensor and then read out in a subsequent readout (4402). Thus, each blanking period and readout results in an image frame for a particular electromagnetic energy spectrum. For example, a first frame 404 may be generated based on the spectrum of a first one or more pulses 4416, a second frame 4406 may be generated based on the spectrum of a second one or more pulses 4418, a third frame 4408 may be generated based on the spectrum of a third one or more pulses 4420, a fourth frame 4410 may be generated based on the spectrum of a fourth one or more pulses 4422, a fifth frame 4412 may be generated based on the spectrum of a fifth one or more pulses 4424, and an Nth frame 4426 may be generated based on the spectrum of an Nth one or more pulses 4426.

[0251] Pulses 4416-4426 may include energy from a single emitter or from a combination of two or more emitters. For example, a spectrum included in a single readout cycle or within multiple frames 4404-4414 may be selected for the desired inspection or detection of a particular tissue or condition. According to one embodiment, one or more pulses may include visible spectrum light for generating a color or black and white image, and one or more additional pulses are used to obtain a spectral response to classify the type of tissue. For example, pulse 4416 may include red light, pulse 4418 may include blue light, and pulse 4420 may include green light, while the remaining pulses 4422-4426 may include wavelengths and spectra for detecting a particular tissue type. As another example, pulses of a single readout cycle may include spectra (e.g., different fragments of the electromagnetic spectrum) generated by multiple different emitters that can be used to detect a particular tissue type. For example, if a combination of wavelengths causes a pixel to have a value that exceeds or is below a threshold, the pixel may be classified as corresponding to a particular type of tissue. Each frame can be used to further narrow down the type of tissue present at that pixel (eg, and each pixel in the image) to provide a very specific classification of the tissue and / or state of the tissue (disease / healthy) based on the spectral response.

[0252] Multiple frames 4404-4414 are shown with readout periods of varying lengths and pulses of varying lengths or intensities. Blanking periods, pulse lengths or intensities, etc. may be selected based on the sensitivity of the monochromatic sensor to a particular wavelength, the power output capability of the transmitter, and / or the carrying capacity of the waveguide.

[0253] by Fig.44The hyperspectral image or hyperspectral image data obtained in the manner shown in can result in multiple frames, each of which is based on a different spectrum or combination of spectra. In some cases, dozens or hundreds of different frames can be obtained. In other cases, such as for video streams, the number of frames can be limited to provide an observable frame rate. Because a combination of different spectra can be provided in a single readout cycle, useful dynamic spectral information can still be obtained even in a video stream.

[0254] In one embodiment, a video or other image may include a black and white or color image superimposed with information derived from the spectral response of each pixel. For example, pixels corresponding to a particular tissue or condition may be displayed in bright green or other colors to aid a physician or other medical professional during an examination.

[0255] In one embodiment, dual image sensors may be used to obtain a three-dimensional image or video feed.Three-dimensional examination may allow for an improved understanding of the three-dimensional structure of the examination region and mapping of different tissue or material types within the region.

[0256] In one embodiment, multispectral or hyperspectral imaging can be used to take a closer look at a material or substance. For example, infrared wavelengths can pass through some tissues such as muscle or fat and reflect from blood vessels. In one embodiment, infrared waves can penetrate 5mm, 8mm, or 10mm or deeper into the tissue. Obtaining a series of frames including at least one infrared frame allows inspection to provide information about the location of blood vessels below the surface. This can be extremely helpful for surgical procedures that may wish to perform incisions that avoid blood vessels. In one embodiment, a color or grayscale image can be superimposed with a green color that indicates the location of blood vessels below the surface. Similarly, during inspection, the known spectral response of blood can be used to observe through the blood and observe tissues or structures of interest.

[0257] After capturing a series of frames 4404-4414, the sub-frames may be assembled into a single frame for display on a monitor or other display device. A color or grayscale image may be generated from one or more of the frames, and stacking information for the pixels may be determined based on all or the remaining frames. The color or grayscale image may be combined with the stacking information to generate a single frame. The single frame may be displayed as a single image or an image in a video stream.

[0258] In one embodiment, the Fig.44The hyperspectral data obtained as shown in the figure is provided for analysis by a third-party algorithm to classify the tissue or substance captured in the image. In one embodiment, the third-party algorithm can be used to select the spectrum or waveband to be used during imaging so that the required spectral response analysis can be performed. In one embodiment, the spectral response analysis can be performed in real time during a medical imaging procedure or other medical procedure. The spectral data can be superimposed on an RGB or black and white image so that a user can easily distinguish certain types of tissues, organs, chemical processes, diseases, etc. In one embodiment, the spectral data can be provided to a computer operating system such as a robotic system for automation of medical imaging or medical procedures.

[0259] Fig.45 4500. System 4500 includes an endoscope 4506 or other suitable imaging device with a light source 4508 for use in a light-deficient environment. Endoscope 4506 includes an image sensor 4504 and a filter 4502 for filtering out unwanted wavelengths of light or other electromagnetic radiation before reaching image sensor 4504. Light source 4508 transmits light that can illuminate surface 4512 in a light-deficient environment such as a body cavity. Light 4510 reflects from surface 4512 and passes through filter 4502 before striking image sensor 4504.

[0260] Optical filter 4502 can be used in the specific implementation of applying fluorescent reagent or dye. In such embodiments, optical filter 4502 is configured to filter out all light except light or other electromagnetic radiation of one or more required wavelengths or spectral bands. In one embodiment, optical filter 4502 is configured to filter out the excitation wavelength of electromagnetic radiation causing reagent or dye to fluoresce, so that only the expected relaxation wavelength of the reagent or dye that fluoresces is allowed to pass through optical filter 4502 and reach image sensor 4504. In one embodiment, optical filter 4502 filters out the fluorescent reagent excitation wavelength at least between 770nm and 790nm. In one embodiment, optical filter 4502 filters out the fluorescent reagent excitation wavelength at least between 795nm and 815nm. In one embodiment, optical filter 4502 filters out the fluorescent reagent excitation wavelength at least between 770nm and 790nm and between 795nm and 815nm. In these embodiments, optical filter 4502 filters out the excitation wavelength of reagent, and allows the only relaxation wavelength of fluorescent reagent to be read by image sensor 4504. Image sensor 4504 may be a wavelength agnostic image sensor, and filter 4502 may be configured to allow image sensor 4504 to receive only the relaxation wavelength of the fluorescent agent and not the emission excitation wavelength of the agent. The data determined by image sensor 4504 may then indicate the presence of key body structures, tissues, biological processes, or chemical processes as determined by the location of the agent or dye.

[0261] Filter 4502 may also be used in implementations where fluorescent agents or dyes have not been administered. Filter 4502 may be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read by image sensor 4504. Image sensor 4504 may be a monochrome image sensor such that pixels of a captured image that are above a threshold or below a threshold may be characterized as corresponding to certain spectral responses or fluorescent emissions. The spectral response or fluorescent emissions determined by the pixels captured by image sensor 4504 may indicate the presence of certain body tissues or structures, certain conditions, certain chemical processes, and the like.

[0262] In one embodiment, light source 4508 transmits white light that contacts surface 4512 and is reflected back by filter 4502 where it is filtered before it strikes image sensor 4504. In one embodiment, light source 4508 transmits white light that passes through filter 4502 so that only filtered light of one or more desired desired wavelengths emerges from filter 4502 to be reflected from surface 4512 and read by image sensor 4504. For example, in one embodiment, filter 4502 allows only light having a wavelength of 795 nm to pass through filter 4502 and contact image sensor 4504. Additionally, in one embodiment, filter 4502 allows only certain wavelengths of light to be reflected back to image sensor 4504 of endoscope 4506 or other imaging device. Filter 4502 may be located anywhere in system 4500 or may be a property of a lens or image sensor 4504. Filter 4502 may be located in front of and / or behind image sensor 4504. In one embodiment, light emitted by light source 4508 is filtered before it reaches surface 4512 and reflected light is filtered by additional filters before being prepared by image sensor 4504.

[0263] Light source 4508 may be an emitter that may be configured to emit white light or electromagnetic radiation of one or more specific wavelengths. Light source 4508 may include multiple lasers configured to emit or pulse light of specified wavelengths. In one embodiment, light source 4508 emits white light, and filter 4502 is selected to filter all unwanted light except one or more desired wavelengths of light or other electromagnetic radiation. Filter 4502 may be selected for a specific examination or purpose, such as highlighting a type of body tissue or structure, or highlighting certain conditions or chemical processes.

[0264] Fig.464600. The system 4600 includes an endoscope 4606 or other suitable imaging device having a light source 4608 for a light-deficient environment. The endoscope 4606 includes an image sensor 4604 and two filters 4602a, 4602b. It should be understood that in alternative embodiments, the system 4600 may include any number of filters, and the number of filters and the type of filters may be selected for certain purposes, such as for collecting imaging information of specific body tissues, body conditions, chemical processes, etc. The filters 4602a, 4602b are configured to filter out unwanted wavelengths of light or other electromagnetic radiation. The filters 4602a, 4602b may be configured to filter out unwanted wavelengths from white light or other electromagnetic radiation that may be emitted by the light source 4608. The filtered light may hit a surface 4612 (e.g., body tissue) and reflect back onto the image sensor 4604.

[0265] Relative to Fig.45 To further describe the invention, filters 4602a, 4602b may be used in implementations where fluorescent reagents or dyes have been applied. Filters 4602a, 4602b may be configured to block the emission excitation wavelengths of the reagents or dyes and allow image sensor 4604 to read only the relaxation wavelengths of the reagents or dyes. In addition, filters 4602a, 4602b may be used in implementations where fluorescent reagents or dyes have not been applied. In such implementations, filters 4602a, 4602b may be selected to allow wavelengths corresponding to the desired spectral response to pass through image sensor 4604 and be read by the image sensor.

[0266] Multiple filters 4602a, 4602b can each be configured to filter out different ranges of wavelengths of the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than a desired wavelength range, and an additional filter can be configured to filter out wavelengths shorter than a desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 4604.

[0267] In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 513nm and 545nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 565nm and 585nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 900nm and 1000nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 425nm and 475nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 520nm and 545nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 625nm and 645nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 760nm and 795nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 795nm and 815nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 370nm and 420nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are tailored to allow electromagnetic radiation between 600nm and 670nm to contact the image sensor 4604. In one embodiment, the filters 4602a, 4602b are configured to allow only certain fluorescent relaxation emissions to pass through the filters 4602a, 4602b and contact the image sensor 4604.

[0268] In one embodiment, system 4600 includes multiple image sensors 4604, and may specifically include two image sensors for generating three-dimensional images. Image sensors 4604 may be color / wavelength agnostic and configured to read electromagnetic radiation of any wavelength reflected from surface 4612. In one embodiment, image sensors 4604 are each color-dependent or wavelength-dependent and configured to read electromagnetic radiation of a specific wavelength reflected from surface 4612 and returned to image sensor 4604. Alternatively, image sensor 4604 may include a single image sensor having multiple different pixel sensors configured to read light of different wavelengths or colors, such as a Bayer filter color filter array. Alternatively, image sensor 4604 may include one or more color-agnostic image sensors that may be configured to read electromagnetic radiation of any wavelength reflected from surface 4612 according to a pulse schedule such as Figures 5 to 7E and Figure 15 to Figure 16 Those shown in ) read electromagnetic radiation of different wavelengths.

[0269] Fig.47 4700 is a schematic diagram showing a system 4700 for mapping a surface and / or tracking an object in an environment lacking light. In one embodiment, an endoscope 4702 pulses a grid array 4706 (which may be referred to as a laser mapping pattern) onto a surface 4704 in an environment lacking light. Fig.47 In one embodiment shown in FIG. 4 , the grid array 4706 may include vertical hashes 4708 and horizontal hashes 4710. It should be understood that the grid array 4706 may include any suitable array for mapping the surface 4704, including, for example, a raster grid of discrete points, an occupancy grid map, a dot array, etc. In addition, the endoscope 4702 may pulse multiple grid arrays 4706, and may pulse one or more separate grid arrays on each of multiple objects or structures, for example, in an abluminal environment.

[0270] In one embodiment, the system 4700 pulses a grid array 4706 that can be used to determine a three-dimensional surface and / or track the position of an object such as a tool or another device in a light-deficient environment. In one embodiment, the system 4700 can provide data to a third-party system or computer algorithm for determining surface size and configuration by way of light detection and ranging (LIDAR) mapping. The system 4700 can pulse any suitable wavelength of light or electromagnetic radiation in the grid array 4706, including, for example, ultraviolet light, visible light, light, and / or infrared or near infrared light. Surfaces 4704 and / or objects within an environment can be mapped and tracked at very high resolution and with very high accuracy and precision.

[0271] In one embodiment, system 4700 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. System 4700 may include a light engine having an illumination source and an inner cavity, the illumination source generating one or more electromagnetic radiation pulses, the inner cavity transmitting the one or more electromagnetic radiation pulses to a distal end of an endoscope in a light-deficient environment such as a body cavity. In one embodiment, at least a portion of the one or more electromagnetic radiation pulses include a laser mapping pattern emitted onto a surface in the light-deficient environment (such as a surface of body tissue in the body cavity and / or a surface of a tool or other device). Endoscope 4702 may include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations in the light-deficient environment.

[0272] In one embodiment, system 4700 includes a processor for determining the distance between an endoscope or tool and an object, such as surface 4704. The processor may also determine the angle between the endoscope or tool and the object. The processor may also determine surface area information about the object, including, for example, the size of the surgical tool, the size of the structure, the size of the anatomical structure, position information, and other position data and measurements. System 4700 may include one or more image sensors that provide image data output to a control system for determining the distance between an endoscope or tool and an object, such as surface 4704. The image sensor may output information to the control system for determining the angle between the endoscope or tool and the object. In addition, the image sensor may output information to the control system for determining surface area information about the object, the size of the surgical tool, the size of the structure, the size of the anatomical structure, position information, and other position data and measurements.

[0273] In one embodiment, the grid array 4706 is pulsed by the illumination source of the endoscope 4702 at a sufficient speed so that the grid array 4706 is not visible to the user. In various specific implementations, the grid array 4706 may be distracting to the user during the endoscopic imaging procedure and / or the endoscopic surgical procedure. The grid array 4706 may be pulsed at a sufficiently short period so that the grid array 4706 cannot be detected by the human eye. In an alternative embodiment, the endoscope 4702 pulses the grid array 4706 at a sufficient repetition frequency so that the grid array 4706 can be viewed by the user. In such embodiments, the grid array 4706 may be superimposed on the image of the surface 4704 on the display. The grid array 4706 may be superimposed on the black and white or RGB image of the surface 4704 so that the grid array 4706 may be visible to the user during the use of the system 4700. The user of the system 4700 may indicate whether the grid array 4706 should be superimposed on the image of the surface 4704 and / or whether the grid array 4706 should be visible to the user. The system 4700 may include a display that provides real-time measurements of the distance from the endoscope 4702 to a surface 4704 or another object within the light-deficient environment. The display may also provide real-time surface area information about the surface 4704 and / or any object, structure, or tool within the light-deficient environment. The accuracy of the measurement may be accurate to less than one millimeter.

[0274] Endoscope 4702 can be configured according to a pulse schedule (such as Figures 5 to 7E and Figure 15 to Figure 16 ) pulsed electromagnetic radiation, for example, which may also include a pulsed grid array 4706 and the pulses are used to generate an RGB image and further generate red, green and blue light of a grid array 4706 that can be superimposed on the RGB image and / or used to map and track surfaces 4704 and objects in a light-starved environment.

[0275] In one embodiment, the endoscope 4702 includes one or more color-agnostic image sensors. In one embodiment, the endoscope 4702 includes two color-agnostic image sensors for generating a three-dimensional image or map of an aphotic environment. The image sensor can generate an RGB image of an aphotic environment according to a pulse schedule as disclosed herein. In addition, the image sensor can determine data for mapping an aphotic environment and tracking one or more objects within the aphotic environment based on data determined when pulsing the grid array 4706. In addition, the image sensor can determine spectral or hyperspectral data and fluorescence imaging data according to a pulse schedule, which can be modified by the user to accommodate the specific needs of the imaging procedure. In one embodiment, the pulse schedule includes red, green, and blue pulses and pulses of the grid array 4706 and / or pulses for generating hyperspectral image data and / or fluorescence image data. In various specific implementations, the pulse schedule may include any suitable combination of electromagnetic radiation pulses according to user needs. The recurrence frequency of electromagnetic radiation of different wavelengths can be determined based on, for example, the energy of certain pulses, the needs of the user, whether certain data (e.g., hyperspectral data and / or fluorescence imaging data) need to be continuously updated or can be updated less frequently, etc.

[0276] The pulse schedule may be modified in any suitable manner, and certain electromagnetic radiation pulses may be repeated at any suitable frequency, as desired by a user or a computer-implemented program for certain imaging procedures. For example, in embodiments where surface tracking data generated based on grid array 4706 is provided to a computer-implemented program for use in, for example, a robotic surgical procedure, grid array 4706 may be pulsed more frequently than where surface tracking data is provided to a user visualizing a scene during an imaging procedure. In such embodiments where surface tracking data is used for a robotic surgical procedure, the surface tracking data may need to be updated more frequently or may need to be extremely accurate so that the computer-implemented program can perform the robotic surgical procedure with precision and accuracy.

[0277] In one embodiment, the system 4700 is configured to generate an occupancy grid map including an array of cells divided into a grid. The system 4700 is configured to store height values ​​of each of the corresponding grid cells to determine a surface map of the three-dimensional environment in the light-poor environment.

[0278] Fig.48 4800 is a schematic flow chart showing an example method 4800 for hyperspectral imaging in a light-deficient environment. The method 4800 may be performed by an imaging system such as Fig.37 The endoscopic imaging system shown in ) is performed.

[0279] Method 4800 includes, at 4802, emitting a plurality of narrowband pulses during a readout cycle of a monochrome image sensor. At 4802, pulses may be emitted using a light source including a plurality of emitters that emit electromagnetic energy within a narrow frequency band. For example, the light source may include at least one emitter for covering a plurality of frequency bands of a desired spectrum. At 4804, the monochrome image sensor reads out pixel data from the monochrome image sensor after the readout cycle to generate a plurality of frames. Each frame may include different spectral content. The frames may include a plurality of repeated frames that may be used to generate a digital video stream. Each frame may be based on energy emitted by one or more emitters of the light source. In one embodiment, the frame may be based on a combination of light emitted by the light source to generate a frequency combination to match a frequency response of a desired tissue or substance. At 4806, a controller, CCU, or other system determines a spectral response of the tissue to one or more pixels based on the plurality of frames. For example, based on the values ​​of the pixels in the plurality of frames, the pixel values ​​and knowledge of the frequency of the light emitted for each frame may be used to determine the frequency response of a particular pixel. At 4808, the system may generate a combined image based on the plurality of frames, the combined image including a stack indicating the spectral response to the one or more pixels. For example, the combined image may be a grayscale or color image in which pixels corresponding to a particular tissue or classification are shown in bright green.

[0280] Fig.49 4900 is a schematic flow chart showing an example method 4900 for fluorescence imaging in a light-deficient environment. The method 4900 may be performed by an imaging system such as Fig.37 The endoscopic imaging system shown in ) is performed.

[0281] Method 4900 includes, at 4902, emitting a plurality of narrowband pulses during a readout cycle of a monochromatic image sensor. At 4902, a light source including a plurality of emitters may be used to emit pulses, and the plurality of emitters emit electromagnetic energy in a narrow frequency band. For example, the light source may include at least one emitter for covering a plurality of frequency bands of a desired spectrum. At 4904, the monochromatic image sensor reads out pixel data from the monochromatic image sensor after the readout cycle to generate a plurality of frames. Each frame may include different spectral contents. The frames may include a plurality of repeated frames that may be used to generate a digital video stream. Each frame may be based on energy emitted by one or more emitters of the light source. In one embodiment, the frame may be based on a combination of light emitted by the light source to generate a frequency combination to match the frequency response of the desired tissue or substance. At 4906, a controller, CCU or other system determines the fluorescence relaxation emission of the reagent to one or more pixels based on the plurality of frames. For example, based on the values ​​of the pixels in the plurality of frames, the pixel values ​​and knowledge of the frequency of the light emitted for each frame may be used to determine the frequency response of a particular pixel. At 4908, the system can generate a combined image based on the plurality of frames, the combined image comprising a stack indicating fluorescence relaxation emissions for the one or more pixels. For example, the combined image can be a grayscale or color image in which pixels corresponding to a particular tissue or classification are shown in bright green.

[0282] Example

[0283] The following examples relate to further embodiments:

[0284] Embodiment 1 is an endoscope system for use in a light-deficient environment. The system includes an imaging device. The imaging device includes a tube, one or more image sensors and a lens assembly, the lens assembly including at least one optical element corresponding to the image sensor. The system includes a display and an image signal processing controller, the display being used to provide a visualized scene to a user. The system includes a light engine. The light engine includes an illumination source that generates one or more electromagnetic radiation pulses. The light engine also includes an inner cavity that transmits one or more electromagnetic radiation pulses to the distal end of the endoscope, wherein at least a portion of the one or more electromagnetic radiation pulses includes a laser mapping pattern emitted onto the surface of body tissue.

[0285] Embodiment 2 is an endoscope system according to embodiment 1, further comprising a two-dimensional camera.

[0286] Embodiment 3 is an endoscope system according to any one of embodiments 1 to 2, further comprising a three-dimensional camera.

[0287] Embodiment 4 is an endoscope system according to any one of embodiments 1 to 3, further comprising an n-dimensional camera.

[0288] Embodiment 5 is an endoscopic system according to any one of embodiments 1 to 4, wherein the laser mapping pattern is not visible to a user of the endoscopic system.

[0289] Embodiment 6 is an endoscopic system according to any one of embodiments 1 to 5, wherein the laser mapping pattern is visible to a user of the endoscopic system.

[0290] Embodiment 7 is an endoscope system according to any one of embodiments 1-6, wherein the system also includes a processor that determines at least one of the distance of the endoscope or tool from the object, the angle between the endoscope or tool and the object, or surface area information about the object to a user of the endoscope system.

[0291] Embodiment 8 is an endoscopic system according to any one of embodiments 1 to 7, wherein the image sensor provides image data, and the image data is output to a control system, and the control system determines to a user of the endoscopic system at least one of the distance of the endoscope from the object, the angle between the endoscope and the object, surface area information about the object, the size of the surgical tool, the size of the structure, the size of the anatomical structure, position information, and other position data and measurements.

[0292] Embodiment 9 is an endoscopic system according to any one of embodiments 1 to 8, wherein the display of the laser mapping pattern can be activated or deactivated by a user of the endoscopic system.

[0293] Embodiment 10 is an endoscope system according to any one of embodiments 1 to 9, wherein the display provides a user of the endoscope system with real-time measurements of one or more of the distance of an object from the endoscope, the angle between the endoscope and the object, or surface area information about the object.

[0294] Embodiment 11 is an endoscopic system according to any one of embodiments 1 to 10, wherein the real-time measurements provided by the display are accurate to less than 10 centimeters.

[0295] Embodiment 12 is an endoscopic system according to any one of embodiments 1 to 11, wherein the real-time measurements provided by the display are accurate to less than 1 mm.

[0296] Embodiment 13 is an endoscope system according to any one of embodiments 1 to 12, wherein the imaging device includes a first image sensor and a second image sensor to generate a three-dimensional image.

[0297] Embodiment 14 is an endoscopic system according to any one of embodiments 1 to 13, wherein the system includes a plurality of tools, wherein each tool has one or more laser mapping patterns.

[0298] Embodiment 15 is an endoscope system according to any one of embodiments 1 to 14, wherein the light engine includes a first output and a second output that are independent of each other, wherein the first output is used for light illumination, and the second output is used for tool tracking.

[0299] Embodiment 16 is an endoscopic system according to any one of embodiments 1 to 15, wherein each electromagnetic radiation pulse results in an exposure frame generated by the image sensor; wherein one or more exposure frames are displayed to the user as a single image on the display.

[0300] Embodiment 17 is an endoscopic system according to any one of embodiments 1 to 16, wherein the visible color for use on the display is specified for the single image; wherein the visible color is 8 bits or 16 bits or n bits.

[0301] Embodiment 18 is an endoscopic system according to any one of embodiments 1 to 17, wherein each electromagnetic radiation pulse causes an exposure frame to be generated by the image sensor; wherein one or more exposure frames are displayed to the user as a stacked image on the display.

[0302] Embodiment 19 is an endoscopic system according to any one of embodiments 1 to 18, wherein a visible color for use on the display is specified for the stacked image; wherein the visible color is 8 bits or 16 bits or n bits.

[0303] Embodiment 20 is an endoscopic system according to any one of embodiments 1 to 19, wherein the light engine includes a polarization filter.

[0304] Embodiment 21 is an endoscopic system according to any one of embodiments 1 to 20, wherein the polarization filter is located in the path of the electromagnetic radiation.

[0305] Embodiment 22 is an endoscopic system according to any one of embodiments 1 to 21, wherein the polarization filter is located at the proximal end of the lumen.

[0306] Embodiment 23 is an endoscopic system according to any one of embodiments 1 to 22, wherein the polarization filter is located at the distal end of the lumen.

[0307] Embodiment 24 is an endoscopic system according to any one of embodiments 1 to 23, wherein the lens assembly includes an electromagnetic radiation filter.

[0308] Embodiment 25 is an endoscopic system according to any one of embodiments 1 to 24, wherein the lens assembly includes a polarizing filter.

[0309] Embodiment 26 is an endoscopic system according to any one of embodiments 1 to 25, wherein each electromagnetic radiation pulse results in an exposure frame generated by the image sensor; wherein one or more exposure frames are fed to a corresponding system that will provide the location of critical tissue structures.

[0310] Embodiment 27 is an endoscopic system according to any one of embodiments 1 to 26, wherein the position of the critical structure is received by the endoscopic system and superimposed on a display, wherein the critical structure is encoded into any color selected by an algorithm or a user.

[0311] It should be appreciated that the various features disclosed herein provide significant advantages and advances in the art. The following claims are examples of some of those features.

[0312] In the above-mentioned detailed description of the present disclosure, for the purpose of simplifying the present disclosure, the various features of the present disclosure are concentrated in a single embodiment. The method of the present disclosure should not be understood as embodying such an intention: the disclosure protected by the claims requires more features than those explicitly listed in each claim. On the contrary, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0313] It should be understood that the above configuration is only an exemplary application of the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, those skilled in the art may design many modifications and alternative configurations, and the appended claims are intended to cover these modifications and configurations.

[0314] Thus, while the present disclosure is shown in the drawings and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications may be made without departing from the principles and concepts described herein, including but not limited to changes in size, material, shape, form, function and mode of operation, assembly and use.

[0315] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0316] For the purpose of illustration and description, the above-mentioned specific embodiments have been provided. These specific embodiments are not intended to be exhaustive or to limit the present disclosure to the disclosed specific forms. According to the above-mentioned teachings, many modifications and changes can be made to the present disclosure. In addition, it should be noted that any or all of the aforementioned alternative specific implementations can be used in any desired combination to form other mixed specific implementations of the present disclosure.

[0317] In addition, although specific implementations of the present disclosure have been described and illustrated, the present disclosure is not limited to the specific forms or arrangements of parts as described and illustrated. The scope of the present disclosure will be defined by the claims appended hereto, any future claims filed here and in different applications, and their equivalents.

Claims

1. An endoscope system for use in a light-deficient environment, comprising: An imaging device, comprising: Tube; one or more image sensors; and a lens assembly including at least one optical element corresponding to the image sensor; A display, wherein the display is used to provide a visual scene for a user; Image signal processing controller; A light engine, wherein the light engine comprises: an illumination source that generates a plurality of pulses of electromagnetic radiation according to a pulse scheme; and lumen that transmits a plurality of electromagnetic radiation pulses to a distal tip of an endoscope, wherein the pulse scheme includes pulsing different wavelengths to produce an RGB image, and further includes a grid array of pulses overlaid on the RGB image, wherein the grid array is used to map and track the surface of body tissue and objects in a light-deficient environment, wherein each pulse of electromagnetic radiation results in an exposure frame produced by the image sensor, and Wherein, each electromagnetic radiation pulse starts during the blanking portion of the sensor operation cycle of the image sensor, or starts during the optically black rear portion of the readout portion of the sensor operation cycle of the image sensor, and ends during the optically black front portion of the readout portion of the next subsequent sensor operation cycle of the image sensor.

2. The endoscope system according to claim 1 further comprising a two-dimensional camera.

3. The endoscopic system according to claim 1 further comprising a three-dimensional camera.

4. The endoscopic system of claim 1, wherein the grid array is not visible to a user of the endoscopic system.

5. The endoscopic system of claim 1, wherein the grid array is visible to a user of the endoscopic system.

6. The endoscopic system of claim 1 , wherein the system further comprises a processor that determines, to a user of the endoscopic system, at least one of a distance of the endoscope or surgical tool from an object, an angle between the endoscope or surgical tool and the object, or surface area information about the object.

7. The endoscopic system of claim 1 , wherein the image sensor provides image data that is output to a control system that determines at least one of a distance of the endoscope from an object, an angle between the endoscope and the object, surface area information about the object, a size of a surgical tool, a size of an anatomical structure, and position information to a user of the endoscopic system.

8. The endoscopic system of claim 1, wherein display of the grid array is configured to be activated or deactivated by a user of the endoscopic system.

9. The endoscopic system of claim 1 , wherein the display provides a user of the endoscopic system with real-time measurements of one or more of a distance from an object to an endoscope, an angle between the endoscope and the object, or surface area information about the object.

10. The endoscopic system of claim 9, wherein the real-time measurements provided by the display are accurate to less than 10 centimeters.

11. The endoscopic system of claim 9, wherein the real-time measurements provided by the display are accurate to less than 1 millimeter.

12. The endoscope system of claim 1, wherein the imaging device comprises a first image sensor and a second image sensor to generate a three-dimensional image.

13. The endoscopic system of claim 1, wherein the system comprises a plurality of surgical tools, wherein each surgical tool has one or more grid arrays.

14. The endoscope system of claim 1, wherein the light engine comprises a first output and a second output that are independent of each other, wherein the first output is used for light illumination, and the second output is used for tracking a surgical tool.

15. The endoscopic system of claim 1, wherein one or more exposure frames are displayed to a user as a single image on the display.

16. The endoscopic system of claim 15, wherein the single image is assigned a visible color for use on the display; wherein the visible color is 8-bit or 16-bit.

17. The endoscopic system of claim 1, wherein one or more exposure frames are displayed to a user as a stacked image on the display.

18. The endoscopic system of claim 17, wherein the stacked image is assigned a visible color for use on the display; wherein the visible color is 8-bit or 16-bit.

19. The endoscopic system of claim 1, wherein the light engine comprises a polarizing filter.

20. The endoscopic system of claim 19, wherein the polarization filter is located in the path of the electromagnetic radiation pulses.

21. The endoscopic system of claim 20, wherein the polarizing filter is located at a proximal end of the lumen.

22. The endoscopic system of claim 20, wherein the polarizing filter is located at a distal end of the lumen.

23. The endoscopic system of claim 1, wherein the lens assembly comprises an electromagnetic radiation filter.

24. The endoscopic system of claim 1, wherein the lens assembly comprises a polarizing filter.

25. The endoscopic system of claim 1, wherein one or more exposure frames are fed to a corresponding system that will provide the location of critical tissue structures.

26. The endoscopic system of claim 25, wherein the locations of critical structures are received by the endoscopic system and overlaid on a display, wherein the critical structures are encoded in any color selected by an algorithm or a user.

Images