Hyperspectral and fluorescence imaging using topological laser scanning in light-deficient environments

Abstract

The present invention discloses systems, methods and devices for performing fluorescence, hyperspectral and / or laser scanning in a light-deficient environment. The system includes an emitter and an image sensor, the emitter is used to emit electromagnetic radiation pulses, and the image sensor includes a pixel array for sensing reflected electromagnetic radiation. The system includes a controller, which includes a processor in electrical communication with the image sensor and the emitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during the blanking period of the image sensor. The system causes at least a portion of the electromagnetic radiation pulses emitted by the emitter to include electromagnetic radiation red wavelengths, electromagnetic radiation green wavelengths and electromagnetic radiation blue wavelengths, and also includes one or more of the following: electromagnetic radiation having a wavelength from about 513nm to about 545nm; electromagnetic radiation having a wavelength from about 565nm to about 585nm; electromagnetic radiation having a wavelength from about 900nm to about 1000nm; electromagnetic radiation excitation wavelengths that cause reagents to fluoresce; or laser scanning patterns.

Description

Technical Field

[0001] This patent application relates to digital imaging, and in particular to hyperspectral, fluorescence and laser scanning imaging in light-deficient environments. Background Art

[0002] Advances in technology have provided advances in medical imaging capabilities. Endoscopes can be used to view the inside of the body and examine the inside of an organ or cavity of the body. Endoscopes are used to investigate a patient's symptoms, confirm a diagnosis, or provide medical treatment. Medical endoscopes can be used to view a variety of body systems and parts, such as the gastrointestinal tract, respiratory tract, urinary tract, abdominal cavity, etc., through small incisions. Endoscopes can also be used for surgical procedures, such as plastic surgery, surgery on a joint or bone, surgery on the nervous system, surgery within the abdominal cavity, etc.

[0003] In some cases of endoscopic imaging, it may be beneficial or necessary to observe the color space. A digital color image includes at least three layers or "color channels" that accumulate to form an image with a range of tones. Each of the color channels measures the intensity and chromaticity of light in a spectral band. Typically, a digital color image includes color channels for a red spectral band, a green spectral band, and a blue spectral band (this may be referred to as a red, green, and blue or RGB image). Each of the red, green, and blue channels includes brightness information for a red spectral band, a green spectral band, or a blue spectral band. The brightness information of the individual red, green, and blue layers is combined to generate a color image. Because the color image is composed of individual layers, a conventional digital camera image sensor includes a color filter array (CFA) that allows red visible light wavelengths, green visible light wavelengths, and blue visible light wavelengths to hit the selected pixel sensor. Each individual pixel sensor element is sensitive to a red wavelength, a green wavelength, or a blue wavelength, and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image.

[0004] In the case of endoscopic imaging for medical diagnosis or medical procedures, it may be necessary to observe the body cavity with color images. However, as discussed above, conventional image sensors capable of capturing color images have at least three different types of pixel sensors to separately capture the red layer, green layer, and blue layer of the color image. The at least three different types of pixel sensors consume a large amount of physical space, so that the complete pixel array cannot be fitted in the small distal end of the endoscope.

[0005] Because conventional image sensors cannot fit in the distal end of an endoscope, the image sensor is traditionally located in the handpiece unit of the endoscope, which is held by the endoscope operator and is not placed in the body cavity. In such endoscopes, light is transmitted from the handpiece unit along the length of the endoscope to the distal end of the endoscope. This configuration has significant limitations. Endoscopes with this configuration are delicate and can be easily misaligned or damaged when they are bumped or impacted during regular use. This can significantly reduce the quality of the image, and the endoscope requires frequent repair or replacement.

[0006] Conventional endoscopes with image sensors placed in the handpiece unit are also limited to capturing only color images. However, in some implementations, it may be desirable to capture images using fluorescence data, hyperspectral data, and / or laser scanning data in addition to color image data. Fluorescence imaging captures light emitted by a substance that has absorbed electromagnetic radiation and "glows" when it emits a relaxation wavelength. Hyperspectral imaging can be used to identify different materials, biological processes, and chemical processes. Laser scanning imaging can capture the surface shape of objects and panoramas and measure distances between objects within a scene. In some implementations, it may be advantageous to use one or more of fluorescence imaging, hyperspectral imaging, and / or laser scanning imaging in conjunction with observing a scene.

[0007] However, the application of fluorescence, hyperspectral, and laser scanning techniques known in the art generally requires highly specialized equipment that may not be available for many applications. In addition, such techniques provide a limited view of the environment and must generally be used in conjunction with multiple individual systems and multiple individual image sensors that are made sensitive to specific bands of electromagnetic radiation. Therefore, it is desirable to develop an imaging system that can be used in a spatially constrained environment to generate fluorescence imaging data, hyperspectral imaging data, and / or laser scanning imaging data.

[0008] In view of the above, this article describes systems, methods and devices for fluorescence imaging, hyperspectral imaging and laser scanning imaging in a light-deficient environment. Such systems, methods and devices can provide multiple data sets for identifying key structures in the body and providing accurate and valuable information about body cavities. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A non-limiting and non-exhaustive embodiment of the present disclosure is described with reference to the following drawings, wherein like reference numerals refer to like parts in the various views unless otherwise specified. The advantages of the present disclosure will be better understood with reference to the following description and drawings, wherein:

[0010] Figure 1 is a schematic diagram of a system for digital imaging in a light-deficient environment having paired emitters and a pixel array;

[0011] Figure 2 A system for providing illumination to a light-deficient environment for endoscopic imaging;

[0012] Figure 2A is a schematic diagram of the complementary system hardware;

[0013] FIG. 3A to FIG. 3D is a diagram of the operating cycle of the sensor for constructing one image frame;

[0014] Figure 4A is a graphical representation of the operation of an embodiment of an electromagnetic transmitter;

[0015] Figure 4B is a graphical representation of varying the duration and magnitude of emitted electromagnetic pulses to provide exposure control;

[0016] Figure 5 Yes FIG. 3A to FIG. 4B An illustration of an embodiment of the present disclosure of an operating cycle of a sensor, an electromagnetic transmitter, and a combination of transmitted electromagnetic pulses showing the imaging system during operation;

[0017] Fig. 6A is a schematic diagram of a method for recording a video using full spectrum light during a time period from t(0) to t(1);

[0018] Figure 6B is a schematic diagram of the process of recording a video by pulsing a portion of the spectrum of light during the time period t(0) to t(1);

[0019] 7A to 7E A schematic diagram showing a method for recording video frames of both full spectrum light and partitioned spectrum light within a time interval;

[0020] Figure 8 is a schematic diagram of a process flow for simultaneously adjusting an electromagnetic emitter and an image sensor;

[0021] Fig. 9 is a schematic diagram of a process flow for adjusting the sensitivity of an image sensor;

[0022] Fig.10 is a schematic diagram of a process flow for simultaneously adjusting an image signal processor and a transmitter based on a frame cycle histogram;

[0023] Fig.11 is a schematic diagram of a process flow for limiting adjustments to an image signal processor and / or a transmitter based on a desired output;

[0024] Fig.12is a schematic diagram of a process flow for increasing the dynamic range of an image by cycling a sensor through a first emission intensity and a second emission intensity and combining data from the first emission intensity and the second emission intensity;

[0025] Fig.13 is a schematic diagram of a process flow for increasing the dynamic range of an image by cycling a sensor through a plurality of luminous intensities and combining data from each of the plurality of luminous intensities;

[0026] Fig.14 is a schematic diagram of a process flow for performing corrections and adjustments on digital image data;

[0027] Fig.15 is a schematic diagram of system hardware for writing, storing, and reading data from a digital video data stream;

[0028] Fig.16 is a schematic diagram illustrating methods and hardware for use with a zoned light system;

[0029] Fig.17 is a schematic diagram of a pattern reconstruction process for producing an RGB image with laser scan data superimposed thereon by pulse partitioning the spectrum;

[0030] FIG. 18A to FIG. 18B is a schematic diagram of a timing example for deploying two different pixel sensitivities in a dual-sensitivity pixel array;

[0031] FIG. 19A to FIG. 19C The use of pulsed and / or synchronized white light emission and corresponding color sensors is shown;

[0032] FIG. 20A to FIG. 20C A light source having a plurality of emitters is shown;

[0033] Fig.21 A single optical fiber is shown outputting at the output via a diffuser to illuminate a scene in a light-deficient environment;

[0034] Fig. 22 shows a portion of an electromagnetic spectrum divided into a plurality of different sub-spectra that may be emitted by an emitter of a light source in accordance with the principles and teachings of the present disclosure;

[0035] Fig.23 is a schematic diagram showing the timing of emission and readout for generating an image frame including a plurality of exposure frames produced by different partitions of pulsed light;

[0036] Fig.24 is an imaging system including a single cutoff filter for generating a fluorescence exposure frame;

[0037] Fig.25is an imaging system including a plurality of cutoff filters for generating fluorescence exposure frames;

[0038] Fig.26 An imaging system for pulsed laser scanning of a grid pattern for mapping the topology of a scene and / or for calculating distances and dimensions within the scene;

[0039] Fig.27A and Fig.27B Specific implementations with multiple pixel arrays for generating three-dimensional images according to the principles and teachings of the present disclosure are shown;

[0040] Fig.28A and 28B A perspective view and a side view, respectively, of a specific implementation of an imaging sensor constructed on multiple substrates, wherein multiple columns of pixels forming a pixel array are located on a first substrate, and multiple columns of circuits are located on a second substrate, showing electrical connections and communications between a column of pixels and its associated or corresponding column of circuits; and

[0041] Fig.29A and Fig.29B A perspective view and a side view, respectively, of a specific implementation of an imaging sensor having multiple pixel arrays for producing a three-dimensional image are shown, wherein the multiple pixel arrays and the image sensor are constructed on multiple substrates. DETAILED DESCRIPTION

[0042] Disclosed herein are systems, methods and apparatus for digital imaging that may be primarily applicable to medical applications such as medical endoscopic imaging.An embodiment of the present disclosure is an endoscopic system for fluorescence imaging, hyperspectral imaging and / or laser scanning imaging in a light-absent environment.

[0043] In one embodiment, a system includes an emitter and an image sensor, the emitter being used to emit electromagnetic radiation pulses, and the image sensor comprising a pixel array for sensing reflected electromagnetic radiation. The system includes a controller comprising a processor in electrical communication with the image sensor and the emitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor. The system causes at least a portion of the electromagnetic radiation pulses emitted by the emitter to include electromagnetic radiation red wavelengths, electromagnetic radiation green wavelengths, and electromagnetic radiation blue wavelengths, and also includes one or more of the following: electromagnetic radiation having a wavelength from about 513nm to about 545nm; electromagnetic radiation having a wavelength from about 565nm to about 585nm; electromagnetic radiation having a wavelength from about 900nm to about 1000nm; electromagnetic radiation excitation wavelengths that cause the reagent to fluoresce; or a laser scanning pattern. The systems, methods, and devices disclosed herein can generate RGB imaging, and also generate fluorescence imaging data, hyperspectral imaging data, and / or laser scanning imaging data. Fluorescence imaging and / or hyperspectral imaging in combination with laser scanning imaging can be used to identify key structures in the body cavity and also to generate surface topology and calculate the dimensions of key structures.

[0044] The fluorescence imaging techniques discussed herein can be used in combination with one or more fluorescent agents or dyes. The location of the agent can be identified by emitting an excitation wavelength of electromagnetic radiation that causes the agent to fluoresce. The relaxation wavelength emitted by the agent can be read by an image sensor to identify the location of the agent within the scene. Depending on the type of agent used, the location of the agent can also indicate the location of key structures (such as certain types of tissues, cancer cells and non-cancerous cells, etc.).

[0045] The hyperspectral imaging techniques discussed herein can be used to "see through" tissue layers in the foreground of a scene to identify specific types of tissue and / or specific biological or chemical processes. Hyperspectral imaging can be used in a medical setting to quantitatively track the course of a disease and determine tissue pathology. Additionally, hyperspectral imaging can be used to identify key structures such as neural tissue, muscle tissue, cancer cells, etc.

[0046] The laser scanning imaging techniques discussed herein can be evaluated to generate a three-dimensional panorama of a scene and calculate the distances between objects within the scene. The laser scanning data can be used in conjunction with fluorescence imaging and / or hyperspectral imaging to calculate the precise location and size of key structures. For example, fluorescence and / or hyperspectral imaging can be used to identify the location and boundaries of key structures. The location of the key structure, the size of the key structure, and precise measurements of the distances from the key structure to other objects can then be calculated based on the laser scanning data.

[0047] In one embodiment, a medical endoscopic imaging system generates laser scanning data for mapping topology and / or calculating the size of an object in a body cavity. The laser scanning data can be superimposed on an RGB video stream and used in real time by a medical practitioner or a computer program to calculate the distance between objects in a body cavity. Real-time laser scanning data can be used as a non-destructive means of mapping and measuring body cavities. The laser scanning data can assist medical practitioners during exploratory or surgical procedures. In addition, the laser scanning data can be provided to a robotic surgical system to enable the robotic system to accurately perform surgical or other medical procedures.

[0048] In some cases, it is desirable to generate endoscopic imaging with multiple data types or multiple images layered on top of each other. For example, it may be desirable to generate a color ("RGB") image that also includes laser scan data layered on the RGB image. Layered images of this nature may enable a practitioner or computer program to identify dimensions within a body cavity based on the laser scan data. Historically, this would require the use of multiple sensor systems, including an image sensor for color imaging and one or more additional image sensors and / or transmitters for generating laser scan data. These multiple image sensors consume excessive physical space and cannot be located at the distal end of the endoscope.

[0049] Conventional endoscopes are designed so that the image sensor is placed at the proximal end of the device within the handpiece unit. This configuration requires that the incident light travel the length of the endoscope through precisely coupled optical elements. The precise optical elements may not be easy to align during normal use, and this can result in image distortion or image loss. The embodiments of the present disclosure place the image sensor within the distal end of the endoscope itself. This provides greater optical simplicity when compared to specific implementations known in the art. However, an acceptable solution to this approach is non-trivial and introduces its own set of engineering challenges, especially that the image sensor must fit within a highly constrained area. Disclosed herein are systems, methods, and apparatus for digital imaging in light-deficient environments that employ image sensors of minimal area and can be configured to enable laser scanning and color imaging.

[0050] Hyperspectral imaging

[0051] In one embodiment, the systems, methods, and devices disclosed herein provide a device for generating hyperspectral imaging data in a light-deficient environment. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from conventional cameras that only capture light across three wavelengths (including red, green, and blue wavelengths) based on the visible spectrum that can be distinguished by the human eye to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands.

[0052] Hyperspectral imaging was originally developed for applications in mining and geology. Unlike normal camera images, which provide limited information to the human eye, hyperspectral imaging can identify specific minerals based on the spectral signatures of different minerals. Hyperspectral imaging is useful even when captured in aerial images and can provide information about, for example, oil or gas leaks from pipelines or natural wells and their impact on nearby vegetation. This information is gathered based on the spectral signatures of certain materials, objects, or processes that can be identified through hyperspectral imaging.

[0053] Hyperspectral imaging is a subcategory of spectral imaging. Hyperspectral imaging includes spectroscopy and digital photography. In one embodiment of hyperspectral imaging, a complete spectrum or some spectral information is collected at each pixel in the image plane. A hyperspectral camera can use special hardware to capture any suitable number of wavelength bands for each pixel that can be interpreted as a complete spectrum. The goals of hyperspectral imaging can vary for different applications. In one application, the goal of hyperspectral imaging is to obtain the entire electromagnetic spectrum for each pixel in the image scene. This can make it possible to find certain objects that might not be recognizable in the visible light wavelength band. This can make it possible to accurately identify certain substances or tissues, although these substances or tissues might not be recognizable in the visible light wavelength band. In addition, this can make it possible to detect certain processes by capturing images across all wavelengths of the electromagnetic spectrum.

[0054] In one embodiment of the present disclosure, the endoscope system illuminates the source and pulses electromagnetic radiation for spectral or hyperspectral imaging. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from conventional cameras that only capture light across three wavelengths (including red, green, and blue wavelengths) based on the visible spectrum that can be distinguished by the human eye to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can stack imaging generated based on invisible bands (e.g., infrared) on top of imaging based on visible bands (e.g., standard RGB images) to provide additional information that can be easily distinguished by humans or computer algorithms.

[0055] Hyperspectral imaging has many advantages over conventional imaging. The information obtained through hyperspectral imaging enables practitioners and / or computer-implemented programs to accurately identify certain tissues or conditions that may not be identified using RGB imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, allowing practitioners to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells that contrast with typical healthy cells, identify certain tissues or conditions, identify critical structures, etc. Hyperspectral imaging provides specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0056] In medical applications, hyperspectral imaging can provide specific advantages over conventional imaging. The information obtained by hyperspectral imaging can enable practitioners and / or computer-implemented programs to accurately identify certain tissues or conditions that may not be diagnosed or may be diagnosed less accurately when using conventional imaging such as RGB imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, which can enable practitioners to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells that contrast with typical healthy cells, identify certain tissues or conditions, identify key structures, etc. Hyperspectral imaging can provide specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0057] In various applications and implementations of the present disclosure, endoscopic hyperspectral imaging can present advantages over conventional imaging. In medical implementations, endoscopic hyperspectral imaging can allow a practitioner or a computer-implemented program to distinguish, for example, neural tissue, muscle tissue, various blood vessels, blood flow direction, etc. Hyperspectral imaging can enable atypical cancer tissue to be accurately distinguished from typical healthy tissue, and thus can enable a practitioner or a computer-implemented program to distinguish the boundaries of cancerous tumors during surgery or research imaging. In addition, hyperspectral imaging in a light-deficient environment as disclosed herein can be combined with the use of reagents or dyes to allow further distinction of certain tissues or substances. In such embodiments, reagents or dyes can fluoresce at specific wavelength bands in the electromagnetic spectrum, thereby providing information specific to the purpose of the reagent or dye. Systems, methods, and devices as disclosed herein can enable pulsing of any number of wavelength bands, so that one or more reagents or dyes can fluoresce at different times. In certain implementations, this enables identification or study of multiple medical conditions during a single imaging procedure.

[0058] Fluorescence imaging

[0059] The systems, methods, and devices disclosed herein provide a device for generating fluorescence imaging data in an environment lacking light. Fluorescence imaging data can be used to identify certain substances, tissues, components, or processes within a body cavity or other environment lacking light. In certain embodiments, fluorescence imaging is provided to a medical practitioner or a computer-implemented program to enable identification of certain structures or tissues in the body. Such fluorescence imaging data can be layered on a black and white or RGB image to provide additional information and context.

[0060] Fluorescence is the emission of light from a substance that has absorbed light or other electromagnetic radiation. Some fluorescent substances can "glow" or emit different colors visible to the human eye when subjected to ultraviolet light or other wavelengths of electromagnetic radiation. When the source of radiation stops, some fluorescent materials will stop emitting light almost immediately.

[0061] Fluorescence occurs when the orbital electrons of a molecule, atom or nanostructure are excited by light or other electromagnetic radiation and then relax to their ground state by emitting photons from an excited state. The specific frequency of the electromagnetic radiation that excites the orbital electrons or is emitted by the photons during relaxation depends on the specific atom, molecule or nanostructure. In most cases, the light emitted by the substance has a longer wavelength than the radiation absorbed by the substance, and therefore has lower energy. However, when the absorbed electromagnetic radiation is intense, one electron can absorb two photons. This two-photon absorption can result in the emission of radiation with a shorter wavelength than the absorbed radiation and therefore with higher energy. In addition, the emitted radiation can also have the same wavelength as the absorbed radiation.

[0062] Fluorescence imaging has many practical applications, including mineralogy, geology, medicine, spectroscopy of chemical sensors, detection of biological processes or signals, etc. Fluorescence can be used in biochemistry and medicine as a non-destructive means for tracking or analyzing biomolecules. Biomolecules, including certain tissues or structures, can be tracked by analyzing the fluorescence emission of biomolecules after being excited by electromagnetic radiation of certain wavelengths. However, relatively few cellular components are naturally fluorescent. In some specific implementations, it may be advantageous to visualize certain tissues, structures, chemical processes or biological processes that are not fluorescent in nature. In such specific implementations, a dye or reagent can be applied to the body, and the dye or reagent may include molecules, proteins or quantum dots with fluorescent properties. Then, the reagent or dye may fluoresce after being excited by electromagnetic radiation of certain wavelengths. Different reagents or dyes may include different molecules, proteins and / or quantum dots that will fluoresce under electromagnetic radiation of a specific wavelength. Therefore, it may be necessary to excite the reagent or dye with electromagnetic radiation of a specific frequency band to achieve fluorescence and identify the desired tissue, structure or process in vivo.

[0063] Fluorescence imaging can provide valuable information in the medical field that can be used for diagnostic purposes and / or can be visualized in real time during medical procedures. Specific reagents or dyes can be applied to the body to make certain tissues, structures, chemical processes or biological processes fluoresce. The fluorescence of the reagent or dye can highlight body structures, such as blood vessels, nerves, specific organs, etc. In addition, the fluorescence of the reagent or dye can highlight conditions or diseases, such as cancer cells or cells that undergo certain biological or chemical processes that may be associated with the condition or disease. Fluorescence imaging can be used in real time by a practicing physician or a computer program to distinguish, for example, cancer cells from non-cancerous cells during surgical tumor extraction. Fluorescence imaging can also be used as a non-destructive means for tracking and visualizing conditions in the body that were originally invisible to the human eye or indistinguishable in RGB images over time.

[0064] Systems, methods and devices for generating fluorescence imaging data can be used in conjunction with reagents or dyes. Some reagents or dyes are known to attach to certain types of tissues and fluoresce at specific wavelengths of the electromagnetic spectrum. In one specific implementation, a reagent or dye is administered to a patient, and the reagent or dye is configured to fluoresce when activated by light of certain wavelengths. The endoscopic imaging system disclosed herein is used to excite reagents or dyes and cause them to fluoresce. The fluorescence of the reagent or dye is captured by the endoscopic imaging system to help identify tissues or structures in a body cavity. In a specific implementation, a plurality of reagents or dyes are administered to a patient, each of which is configured to emit fluorescence of different wavelengths and / or provide indications of different structures, tissues, chemical reactions, biological processes, etc. In such specific implementations, the endoscopic imaging system emits each of the applicable wavelengths to cause each of the applicable reagents or dyes to fluoresce. This can eliminate the need to perform individual imaging procedures for each of the multiple reagents or dyes.

[0065] Imaging agents can enhance imaging capabilities in the pharmaceutical, medical, biotechnology, diagnostic and medical procedure industries. Many imaging techniques such as X-ray, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI) and nuclear medicine mainly analyze anatomical structures and morphology, and cannot detect changes at the molecular level. Fluorescent agents, dyes and probes (including quantum dot nanoparticles and fluorescent proteins) assist medical imaging techniques by providing additional information about certain tissues, structures, chemical processes and / or biological processes present in the imaging area. Cell tracking and / or tracking of certain molecular biomarkers are achieved using fluorescent agent imaging. Fluorescent agents can be used to image cancer, infection, inflammation, stem cell biology, etc. Many fluorescent agents and dyes are being developed and applied to visualize and track biological processes in a non-destructive manner. Such fluorescent agents can be excited by electromagnetic radiation of certain wavelengths or wavelength bands. Similarly, these fluorescent agents can emit relaxation energy of certain wavelengths or wavelength bands when fluorescing, and the relaxation energy emitted can be read by a sensor to determine the position and / or boundary of the agent or dye.

[0066] In one embodiment of the present disclosure, an endoscope system pulses electromagnetic radiation for exciting electrons in a fluorescent agent or dye. The endoscope system can pulse electromagnetic radiation of multiple different wavelengths during a single imaging process for causing multiple different agents or dyes to fluoresce. The endoscope includes an image sensor that is sensitive to the relaxation wavelength of the one or more agents or dyes. The imaging data generated by the image sensor can be used to identify the position and boundaries of the one or more agents or dyes. The endoscope system can also pulse electromagnetic radiation in the red, green, and blue bands of visible light so that fluorescent imaging can be superimposed on an RGB video stream.

[0067] Laser mapping imaging

[0068] In one embodiment, the systems, methods, and devices disclosed herein provide a device for generating laser mapping data using an endoscopic imaging system. The laser mapping data can be used to determine precise measurements and topographical contours of a scene. In a specific implementation, the laser mapping data is used to determine, for example, precise measurements between structures or organs in a body cavity, devices or tools in a body cavity, and / or key structures in a body cavity.

[0069] Laser mapping typically includes a controlled deflection of a laser beam. In the field of three-dimensional object scanning, laser mapping combines controlled steering of a laser beam with a laser rangefinder. By measuring distance in each direction, a laser rangefinder can quickly capture the surface shape of objects, tools, and panoramas. The construction of a full three-dimensional topology can include combining multiple surface models obtained from different perspectives. Various measurement systems and methods exist in the art for applications in archeology, geography, atmospheric physics, autonomous vehicles, and the like. One such system includes light detection and ranging (LIDAR), which is a three-dimensional laser mapping system. LIDAR has been used in navigation systems such as aircraft or satellites to determine the position and orientation of sensors combined with other systems and sensors. LIDAR uses an active sensor to illuminate an object and detects energy reflected from the object and returned to the sensor.

[0070] Laser tracking or tool tracking using lasers measures objects by determining the position of an optical target held relative to those objects. Laser trackers can be accurate to the order of 0.025 mm over distances of several meters. In one embodiment, the endoscopic imaging system pulses light for use in conjunction with a laser tracking system so that positions or tools within a scene can be tracked and measured. In such embodiments, the endoscopic imaging system can pulse a laser tracking pattern on a tool, object, or other structure within a scene imaged by the endoscopic imaging system. A target can be placed on a tool, object, or other structure within the scene. Measurements between the endoscopic imaging system and the target can be triggered and acquired at selected points so that the endoscopic imaging system can track the position of the target (and the tool, object, or other structure to which the target is attached).

[0071] Pulse imaging

[0072] Some specific implementations of the present disclosure include various aspects of sensor and system combination designs that are capable of generating high-definition images with reduced pixel counts in highly constrained lighting environments. This is achieved by pulsing monochromatic wavelengths frame by frame, and switching or alternating each frame between a single different color wavelength using a controlled light source in combination with a high frame capture rate and a specially designed corresponding monochromatic ("color-agnostic") sensor. The pixels are color-agnostic so that each pixel generates data for each electromagnetic radiation pulse, including pulses of red visible light wavelengths, green visible light wavelengths, and blue visible light wavelengths in addition to the relaxation wavelengths, hyperspectral wavelengths, and reflected laser scanning patterns of one or more fluorescent agents. The system disclosed herein can generate an RGB image with additional imaging data stacked thereon. In one embodiment, the system pulses electromagnetic radiation of certain wavelengths to excite fluorescent agents, thereby generating fluorescent imaging data that can be used to identify key tissues and structures. In one embodiment, the system pulses hyperspectral wavelengths of electromagnetic radiation to generate hyperspectral imaging data that can be used to identify key tissues and structures. In one embodiment, the system pulses laser scanning patterns for generating a three-dimensional topology of a scene and / or for measuring the distance and size of objects within a scene. In one embodiment, a combination of fluorescent, hyperspectral and / or laser scanning emissions is deployed to generate a combined image frame that includes RGB image data and also includes hyperspectral, fluorescent and / or laser scanning data.

[0073] For the purpose of promoting understanding of the principles according to the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe these embodiments. However, it should be understood that it is not intended to limit the scope of the present disclosure. Any changes and further modifications to the inventive features shown herein, and any additional applications of the principles of the present disclosure as shown herein (which will generally occur for those skilled in the relevant art and those who understand the present disclosure) will be considered to be within the scope of the disclosure protected by the claims.

[0074] Before disclosing and describing structures, systems and methods for generating images in a light-deficient environment, it should be understood that the present disclosure is not limited to the specific structures, configurations, process steps and materials disclosed herein, because such structures, configurations, process steps and materials may vary to a certain extent. It should also be understood that the terminology used herein is only for the purpose of describing specific embodiments and is not intended to be limiting, because the scope of the present invention will be limited only by the appended claims and their equivalents.

[0075] In describing and claiming the presently disclosed subject matter, the following terminology will be used in accordance with the following definitions.

[0076] It must be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0077] As used herein, the terms "comprising," "including," "characterized by," and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unrecited elements or method steps.

[0078] As used herein, the phrase "consisting of and its grammatical equivalents exclude any elements or steps not listed in a claim.

[0079] As used herein, the phrase "consisting essentially of and its grammatical equivalents limit the scope of a claim to the specified materials or steps and those materials or steps that do not materially affect one or more of the basic and novel characteristics of the claimed disclosure.

[0080] As used herein, the term "proximal side" broadly refers to the concept of a portion close to a starting point.

[0081] As used herein, the term "distal" generally refers to the opposite of the proximal side, and thus refers to the concept of a portion farther from a starting point or a portion farthest from a starting point according to the context.

[0082] As used herein, color sensors or multispectral sensors are those sensors known to have a color filter array (CFA) thereon to filter incident electromagnetic radiation into its individual components. In the visible range of the electromagnetic spectrum, such CFAs may be based on a Bayer template or a modified form thereof to separate the green, red and blue spectral components of light.

[0083] As used herein, a monochrome sensor refers to an imaging sensor without filtering. Because the pixels are color amorphous, their effective spatial resolution is significantly higher than their pixel color (usually filtered with a Bayer pattern) counterparts in traditional single-sensor cameras. Monochrome sensors can also have higher quantum efficiency because fewer incident photons are wasted between individual pixels.

[0084] As used herein, a transmitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of the transmitter can be configured to emit pulses and have very specific frequencies or frequency ranges from within the entire electromagnetic spectrum. The pulses may include wavelengths in the visible range and the invisible range. The transmitter can be cycled on and off to generate pulses, or can generate pulses with a shutter mechanism. The transmitter can have a variable power output level, or can be controlled by auxiliary devices such as apertures or filters. The transmitter can emit wide spectrum or full spectrum electromagnetic radiation that can generate pulses by color filtering or shading. The transmitter may include multiple electromagnetic sources that act alone or in concert.

[0085] Now refer to the accompanying drawings, Figure 1 A schematic diagram of a system 100 for sequential pulse imaging in an aphotic environment is shown. The system 100 can be deployed to generate an RGB image, wherein the laser scan data is superimposed on the RGB image. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a partition of electromagnetic radiation in an aphotic environment 112, and the pixel array 122 senses an instance of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 operate in sequence such that one or more pulses of a partition of electromagnetic radiation produce image data sensed by the pixel array 122.

[0086] It should be noted that, as used herein, the term "light" is both a particle and a wavelength, and is intended to represent electromagnetic radiation that can be detected by the pixel array 122, and may include wavelengths from the visible and invisible spectrum of electromagnetic radiation. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is less than the entire spectrum, or in other words, the wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that is controllable with respect to a portion of the electromagnetic spectrum emitted, or a light source that can manipulate the physical properties of its components, the emission intensity, or the emission duration, or all of the above. An emitter can emit light in any dithered, diffuse, or collimated emission, and can be controlled digitally or by an analog method or system. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and it includes light sources such as lasers, LEDs, incandescent lights, or any light source that can be digitally controlled.

[0087] The pixel array 122 of the image sensor may be electronically paired with the emitter 102 such that the emitter 102 and the pixel array 122 are synchronized during operation for both receiving emissions and adjustments made within the system. The emitter 102 may be tuned to emit electromagnetic radiation in the form of a laser, which may be pulsed to illuminate the light-deficient environment 112. The emitter 102 may pulse at intervals corresponding to the operation and functionality of the pixel array 122. The emitter 102 may pulse light in multiple electromagnetic partitions such that the pixel array receives electromagnetic energy and generates a data set corresponding in time to each specific electromagnetic partition. For example, Figure 1A specific implementation is shown in which the emitter 102 emits four different partitions of electromagnetic radiation, including red 104 wavelengths, green 106 wavelengths, blue 108 wavelengths, and special 110 emissions. The special 110 emissions may include excitation wavelengths for causing the reagent to fluoresce, hyperspectral partitions of the electromagnetic radiation, and / or laser scanning patterns. The special 110 emissions may include multiple individual emissions that are separate and independent of each other. The special 110 emissions may include a combination of excitation wavelengths and laser scanning patterns for causing the reagent to fluoresce, wherein the emissions are separate and independent of each other. The special 110 emissions may include a combination of hyperspectral bands of electromagnetic radiation and laser scanning patterns, wherein the emissions are separate and independent of each other. In one embodiment, the special 110 emissions include any desired combination of emissions that can be combined with data generated by pulsed red 104, pulsed green 106, and pulsed blue 108 emissions.

[0088] The aphotic environment 112 includes structures, tissues, and other elements that reflect a combination of red 114, green 116, and / or blue 118 light. Red 114 tissue is sensitive to the pulsed red 104 light and will be processed by the human eye as having a red 114 hue. The scene within the aphotic environment 112 includes a topology 120 having one or more objects that can be scanned to generate a three-dimensional mapping of the scene.

[0089] The emitter 102 may be a laser emitter capable of emitting pulsed red 104 light for generating sensed red 105 data to identify red 114 elements within the light-starved environment 112. The emitter 102 may also be capable of emitting pulsed green 106 light for generating sensed green 107 data to identify green 116 elements within the light-starved environment. The emitter 102 may also be capable of emitting pulsed blue 108 light for generating sensed blue 109 data to identify blue 118 elements within the light-starved environment. The emitter 102 may also be capable of emitting special 110 emissions for mapping the topology 120 of a scene within the light-starved environment 112. The emitter 102 may be capable of emitting pulsed red 104, pulsed green 106, pulsed blue 108, and pulsed special 110 emissions in any desired order.

[0090] Pixel array 122 senses reflected electromagnetic radiation. Each of sensing red 105, sensing green 107, sensing blue 109, and sensing special 111 data may be referred to as an "exposure frame." Sensing special 111 may result in multiple individual exposure frames that are separate and independent from each other. For example, sensing special 111 may produce a fluorescent exposure frame, a hyperspectral exposure frame, and / or a topological exposure frame including laser scanning data. A specific color or wavelength partition is assigned to each exposure frame, where the assignment is based on the timing of the pulse color or wavelength partition from emitter 102. The combination of an exposure frame and the assigned specific color or wavelength partition may be referred to as a data set. Even if pixel 122 is not a dedicated color, a color may be assigned to any given data set based on a priori information about the emitter.

[0091] For example, during operation, after the pulsed red 104 light is pulsed in the light-starved environment 112, the pixel array 122 senses the reflected electromagnetic radiation. The reflected electromagnetic radiation generates an exposure frame, and the exposure frame is classified as sensed red 105 data because it corresponds in time to the pulsed red 104 light. The exposure frame, along with an indication that it corresponds in time to the pulsed red 104 light, is a "data set." This process is repeated for each partition of electromagnetic radiation emitted by the emitter 102. The data created by the pixel array 122 includes a sensed red 105 exposure frame that identifies the red 114 component in the light-starved environment and corresponds in time to the pulsed red 104 light. The data also includes a sensed green 107 exposure frame that identifies the green 116 component in the light-starved environment and corresponds in time to the pulsed green 106 light. The data also includes a sensed blue 109 exposure frame that identifies the blue 118 component in the light-starved environment and corresponds in time to the pulsed blue 108 light. The data also includes a sensing special 111 exposure frame that identifies the topology 120 and corresponds in time to the special 110 shot.

[0092] In one embodiment, three data sets representing red, green, and blue electromagnetic pulses are combined to form a single image frame. Thus, the information in the red exposure frame, the green exposure frame, and the blue exposure frame are combined to form a single RGB image frame. One or more additional data sets representing other wavelength partitions may be superimposed on the single RGB image frame. The one or more additional data sets may represent, for example, laser scanning data, fluorescence imaging data, and / or hyperspectral imaging data.

[0093] It should be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition without departing from the scope of the present disclosure, and any color combination or any electromagnetic partition can be used to replace RED, GREEN and BLUE, such as cyan, magenta and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible wavelengths and invisible wavelengths. In the figure, the light-deficient environment 112 to be imaged includes a red 114 portion, a green 116 portion and a blue 118 portion, and also includes a topology 120 that can be sensed and mapped into a three-dimensional rendering. As shown, the reflected light from the electromagnetic pulse only contains data for the portion of the object with a specific color corresponding to the pulse color partition. These separate color (or color interval) data sets can then be used to reconstruct an image by combining the data sets at 126. The information in each of the multiple exposure frames (i.e., the multiple data sets) can be combined by a controller 124, a control unit, a camera control unit, an image sensor, an image signal processing pipeline, or some other computing resources that can be configured to process the multiple exposure frames and combine the data sets at 126. The data sets may be combined to generate a single image frame within the endoscopy unit itself or off-site by some other processing resource.

[0094] Figure 2 The invention discloses a system 200 for providing illumination to a light-deficient environment such as for endoscopic imaging. The system 200 may be used in combination with any of the systems, methods or devices disclosed herein. The system 200 includes a transmitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an inner cavity waveguide 210, an inner cavity 212 and an image sensor 214 with accompanying optical components (such as lenses). The transmitter 202 (generally referred to as a "light source") generates light that passes through the jumper waveguide 206 and the inner cavity waveguide 210 to illuminate the scene at the distal end of the inner cavity 212. The transmitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, hyperspectral, fluorescence excitation, laser scanning pulse schemes or other wavelengths. The inner cavity 212 can be inserted into a patient for imaging, such as during surgery or examination. The light is output as shown by the dashed line 216. The scene illuminated by the light can be captured using an image sensor 214 and displayed to a doctor or some other medical personnel. The controller 204 may provide control signals to the emitter 202 to control when illumination is provided to a scene. In one embodiment, the emitter 202 and the controller 204 are located within a camera control unit (CCU) or an external console to which the endoscope is connected. If the image sensor 214 comprises a CMOS sensor, light may be periodically provided to the scene as a series of illumination pulses between readout cycles of the image sensor 214 during so-called blanking periods. Thus, light may be pulsed in a controlled manner to avoid stacking into the readout cycles of image pixels in the pixel array of the image sensor 214.

[0095] In one embodiment, the lumen waveguide 210 includes one or more optical fibers. These optical fibers can be made of low-cost materials (such as plastic) to allow processing of the lumen waveguide 210 and / or other parts of the endoscope. In one embodiment, the lumen waveguide 210 is a single glass fiber with a diameter of 500 microns. The jumper waveguide 206 can be permanently attached to the transmitter 202. For example, the jumper waveguide 206 can receive light from a transmitter in the transmitter 202 and provide light to the lumen waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 106 includes one or more glass optical fibers. The jumper waveguide can include any other type of waveguide for guiding light to the lumen waveguide 210. The connector 208 can selectively couple the jumper waveguide 206 to the lumen waveguide 210 and allow the light in the jumper waveguide 206 to pass through the lumen waveguide 210. In one embodiment, the intracavity waveguide 210 is coupled directly to the light source without any intervening jumper waveguide 206 .

[0096] The image sensor 214 includes a pixel array. In one embodiment, the image sensor 214 includes two or more pixel arrays for generating a three-dimensional image. The image sensor 214 may constitute two additional image sensors, each having an independent pixel array and operable independently of one another. The pixel array of the image sensor 214 includes effective pixels and optically black ("OB") pixels or light-blind pixels. An effective pixel may be a transparent "color-indeterminate" pixel capable of sensing imaging data of electromagnetic radiation of any wavelength. When the pixel array is "reset" or calibrated, the optically black pixels are read during the blanking period of the pixel array. In one embodiment, when reading the optically black pixels, light pulses during the blanking period of the pixel array. After the optically black pixels have been read, the effective pixels are read during the readout period of the pixel array. The effective pixels may be charged by the electromagnetic radiation pulsed during the blanking period, so that the effective pixels are ready to be read by the image sensor during the readout period of the pixel array.

[0097] Figure 2A is a schematic diagram of complementary system hardware such as a special or general-purpose computer. Implementations within the scope of the present disclosure may also include physical and other non-transient computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media that can be accessed by a general or special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a computer storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, a specific implementation of the present disclosure may include at least two distinct types of computer-readable media: a computer storage medium (device) and a transmission medium.

[0098] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general purpose or special purpose computer.

[0099] "Network" refers to one or more data links that enable electronic data to be transmitted between computer systems and / or modules and / or other electronic devices. In one specific implementation, the sensor and camera control unit can be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer reasonably regards the connection as a transmission medium. The transmission medium may include a network and / or data link that can be used to carry the desired program code tools in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer. The above combinations should also be included in the scope of computer-readable media.

[0100] In addition, upon reaching various computer system components, program code tools in the form of computer executable instructions or data structures can be automatically transmitted to computer storage media (devices) by transmission media (or vice versa). For example, computer executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., "NIC") and then ultimately transmitted to the computer system RAM and / or the non-volatile computer storage media (devices) of the computer system. RAM can also include solid state drives (SSDs or PCIx-based real-time memory hierarchical storage devices, such as FusionIO). Therefore, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0101] Computer executable instructions include, for example, instructions and data that cause a general purpose computer, a special purpose computer, or a special purpose processing device to perform certain functions or groups of functions when executed by one or more processors. Computer executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter of the present invention is described linguistically with respect to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or steps described above. Rather, the features and steps described above are disclosed as example forms of implementing the claims.

[0102] It should be understood by those skilled in the art that the present disclosure can be implemented in a network computing environment having various types of computer system configurations, including personal computers, desktop computers, laptop computers, information processors, control units, camera control units, handheld devices, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the above computing devices can be provided by a physical institution or located within a physical institution. The present disclosure can also be implemented in a distributed system environment, in which local and remote computer systems are connected via a network (via a hard-wired data link, a wireless data link, or a combination of a hard-wired data link and a wireless data link), and both can perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0103] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0104] Figure 2A is a block diagram illustrating an exemplary computing device 250. Computing device 250 may be used to execute various programs, such as those discussed herein. Computing device 250 may be used as a server, a client, or any other computing entity. Computing device 250 may perform various monitoring functions discussed herein, and may execute one or more applications, such as the applications described herein. Computing device 250 may be any of a variety of computing devices, such as a desktop computer, a laptop computer, a server computer, a handheld computer, a camera control unit, a tablet computer, and the like.

[0105] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory device 254 and / or the mass storage device 258. The processor 252 may also include various types of computer-readable media, such as cache memory.

[0106] Memory device 254 includes various computer-readable media such as volatile memory (e.g., random access memory (RAM) 264) and / or nonvolatile memory (e.g., read-only memory (ROM) 266). Memory device 254 may also include a rewritable ROM such as flash memory.

[0107] The mass storage device 258 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., flash memory), etc. Figure 2 As shown in FIG. 2 , the particular mass storage device is a hard drive 274. Various drives may also be included in the mass storage device 258 to enable reading and / or writing from various computer readable media. The mass storage device 258 includes removable media 276 and / or non-removable media.

[0108] I / O devices 260 include various devices capable of inputting or retrieving data and / or other information to or from computing device 250. Exemplary I / O devices 260 include digital imaging devices, electromagnetic sensors and transmitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCD or other image capture devices, and the like.

[0109] Display device 280 includes any type of device capable of displaying information to one or more users of computing device 250. Examples of display device 280 include a monitor, a display terminal, a video projection device, and the like.

[0110] Interface 256 includes various interfaces that enable computing device 250 to interact with other systems, devices, or computing environments. Exemplary interface 256 may include any number of different network interfaces 270, such as interfaces to connect local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include user interfaces 268 and peripheral device interfaces 272. Interface 256 may also include one or more user interface elements 268. Interface 256 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touch pad, etc.), keyboards, etc.

[0111] The bus 262 enables the processor 252, the memory device 254, the interface 256, the mass storage device 258, and the I / O device 260 to communicate with each other and with other devices or components coupled to the bus 262. The bus 262 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, and the like.

[0112] For exemplary purposes, the programs and other executable program devices shown herein are discrete blocks, but it should be understood that such programs and devices may reside at various times in different storage devices of the computing device 250 and be executed by the processor 252. Alternatively, the systems and programs described herein may be implemented in hardware, or in a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more of the systems and programs described herein.

[0113] Figure 3A An operating cycle of a sensor used in a rolling readout mode or during a sensor readout 300 is shown. A frame readout may begin at and may be represented by a vertical line 310. A readout cycle is represented by a diagonal or slanted line 302. The sensor may be read out row by row, with the top of the downward sloping edge being the sensor top row 312 and the bottom of the downward sloping edge being the sensor bottom row 314. The time between the last row readout and the next readout cycle may be referred to as a blanking period 316. It should be noted that some of the sensor pixel rows may be covered with a light shield (e.g., a metal coating or any other substantially black layer of another material type). These covered pixel rows may be referred to as optical black rows 318 and 320. Optically black rows 318 and 320 may be used as input to a correction algorithm. As shown Figure 3A As shown, these optically black rows 318 and 320 may be located on the top of the pixel array or on the bottom of the pixel array or on both the top and bottom of the pixel array.

[0114] Figure 3B A method for controlling the amount of electromagnetic radiation (e.g., light) exposed to a pixel so that it is integrated or accumulated by the pixel is shown. It should be understood that photons are the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into an electric charge or current. An electronic shutter or rolling shutter (shown by dashed line 322) can be used to start the integration time by resetting the pixel. The light will then be integrated until the next readout phase. The position of the electronic shutter 322 can be moved between two readout cycles 302 to control the pixel saturation for a given amount of light. It should be noted that this technique allows a constant integration time between two different rows, but introduces a delay when moving from the top row to the bottom row.

[0115] Figure 3C The case is shown where the electronic shutter 322 has been removed. In this configuration, integration of incident light may begin during readout 302 and may end at the next readout cycle 302, which also defines the start of the next integration.

[0116] Figure 3DA configuration is shown without the electronic shutter 322 but with controlled and pulsed light 330 during the blanking period 316. This ensures that all rows see the same light as that emitted from the same light pulse 330. In other words, each row will begin its integration in a dark environment, which may be located at the optically black post row 320 of the readout frame (m) to obtain the maximum light pulse width, and will then receive light through and will end its integration in a dark environment, which may be located at the optically black pre row 318 of the next subsequent readout frame (m+1) to obtain the maximum light pulse width. In e.g. Figure 3D In the blanking period 316, the image generated by the light pulse will be available only during the readout of frame (m+1) without interfering with frames (m) and (m+2). It should be noted that the condition for the light pulse to be read out only in one frame and not interfere with adjacent frames is that a given light pulse is fired during the blanking period 316. Because the optical black rows 318, 320 are not sensitive to light, the optical black post row 320 time of frame (m) and the optical black pre row 318 time of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 330.

[0117] like Figure 3A As shown, the sensor may cycle multiple times to receive data for each pulse color or wavelength (e.g., red, green, blue, or other wavelength on the electromagnetic spectrum). Each cycle may be timed. In one embodiment, the cycle may be timed to operate within an interval of 16.67 ms. In another embodiment, the cycle may be timed to operate within an interval of 8.3 ms. It should be understood that the present disclosure contemplates other timing intervals and is intended to fall within the scope of the present disclosure.

[0118] Figure 4A The operation of an embodiment of an electromagnetic transmitter is graphically illustrated.The transmitter may be timed to correspond to a cycle of the sensor such that electromagnetic radiation is emitted within and / or during a portion of a sensor operating cycle. Figure 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the emitter may pulse during the readout portion 302 of the sensor operating cycle. In one embodiment, the emitter may pulse during the blanking portion 316 of the sensor operating cycle. In one embodiment, the emitter may pulse for a duration that is during a portion of two or more sensor operating cycles. In one embodiment, the emitter may begin pulsing during the blanking portion 316 or during the optically black portion 320 of the readout portion 302, and end pulsing during the readout portion 302 or during the optically black portion 318 of the readout portion 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of the present disclosure as long as the pulsing of the emitter and the cycle of the sensor correspond.

[0119] Figure 4B A graphical representation of varying the duration and magnitude of emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, pulse 3 at 416) to control exposure. An emitter with a fixed output magnitude may be combined with the above Figure 3D and Figure 4A The pulses are pulsed at intervals during any of the cycles described to provide the desired electromagnetic energy to the pixel array. An emitter with a fixed output magnitude may be pulsed at longer intervals to provide more electromagnetic energy to the pixel, or the emitter may be pulsed at shorter intervals to provide less electromagnetic energy. Whether longer or shorter intervals are required depends on the operating conditions.

[0120] Compared to adjusting the time interval of the fixed output magnitude of the emitter pulses, the emission magnitude itself can be increased to provide more electromagnetic energy to the pixel. Similarly, reducing the magnitude of the pulses can provide less electromagnetic energy to the pixel. It should be noted that embodiments of the system can have the ability to adjust both the magnitude and duration if desired. In addition, the sensor can be adjusted to increase its sensitivity and duration as required for optimal image quality. Figure 4B The magnitude and duration of the pulses are shown to be varied. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. In addition, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416, so that the electromagnetic energy provided by the pulse is shown by the area under the pulse shown in the illustration. In the illustration, pulse 2 at 414 has a relatively low magnitude or intensity and a longer duration when compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration when compared to pulse 1 at 412 and pulse 2 at 414.

[0121] Figure 5 According to the principles and teachings of this disclosure, FIG. 3A to FIG. 3D and Figure 4A 302 of the sensor cycle. Figure 4A ) illustrates the possibility or ability to emit electromagnetic energy during the optically black portions 320 and 318 of a read cycle (sensor cycle) 300 in the event that additional time is needed or desired to pulse the electromagnetic energy.

[0122] The exposure frame includes data read by the pixel array of the image sensor during the readout cycle 302. The exposure frame can be combined with an indication of the type of pulse emitted by the emitter before the readout cycle 302. The combination of the exposure frame and the indication of the pulse type can be referred to as a data set. Multiple exposure frames can be combined to generate a black and white or RGB color image. In addition, hyperspectral, fluorescence and / or laser mapping imaging data can be superimposed on the black and white or RGB image.

[0123] In one embodiment, the exposure frame is data sensed by the pixel array during a readout period 302 that occurs after a blanking period 316. The emission of electromagnetic radiation is emitted during the blanking period 316. In one embodiment, a portion of the emission of electromagnetic radiation overlaps the readout period 316. The blanking period 316 occurs when optically black pixels of the pixel array are being read, and the readout period 302 occurs when valid pixels of the pixel array are being read. The blanking period 316 may overlap the readout period 302.

[0124] Fig. 6A and Figure 6B A process for recording an image frame is shown. Multiple image frames may be strung together to generate a video stream. A single image frame may include data from multiple exposure frames, where an exposure frame is data sensed by a pixel array after emitting electromagnetic radiation. Fig. 6A A conventional process is shown that is typically implemented with a color image sensor having a color filter array (CFA) to filter out certain wavelengths of light for each pixel. Figure 6B is a process disclosed herein and can be implemented using a monochromatic "color amorphous" image sensor that receives electromagnetic radiation of all wavelengths.

[0125] Fig. 6A The process shown occurs from time t(0) to time t(1). The process begins with white light emission 602 and sensing white light 604. At 606, an image is processed and displayed based on the sensing at 604.

[0126] Figure 6BThe process shown occurs from time t(0) to time t(1). The process begins by emitting green light 612, and sensing reflected electromagnetic radiation 614 after emitting green light 612. The process continues by emitting red light 616, and sensing reflected electromagnetic radiation 618 after emitting red light 616. The process continues by emitting blue light 620, and sensing reflected electromagnetic radiation 622 after emitting blue light 620. The process continues with one or more emissions of a special 624 emission, and sensing reflected electromagnetic energy 626 after each of the one or more emissions of the special 624 emission. A special emission may include one or more separate emissions, such as an excitation wavelength of a fluorescent agent, a hyperspectral emission, and / or a laser scanning emission. Each of the separate multiple special emissions may be independently sensed by the image sensor to generate a separate and independent exposure frame. An image is processed and displayed at 628 based on each of the sensed reflected electromagnetic energy instances 614, 618, 622, and 626.

[0127] Figure 6B The process shown provides a higher resolution image and provides a means for generating an RGB image that also includes the laser scan data. Figure 6B ), making the sensor sensitive to electromagnetic energy of all wavelengths. Figure 6B In the process shown, the monochrome pixel array is instructed to sense electromagnetic energy from a predetermined partition of the full spectrum of electromagnetic energy in each cycle. Therefore, to form an image, the sensor only needs to cycle with multiple different partitions from within the full spectrum. The final image is assembled based on the multiple cycles. Because the image from each color partition frame cycle has a higher resolution (compared to the CFA pixel array), the resulting image created when the partitioned light frames are combined also has a higher resolution. In other words, because each pixel within the array (rather than, at most, every other pixel in a sensor with a CFA) senses the magnitude of the energy for a given pulse and a given scene, only a period of time apart, a higher resolution image is produced for each scene.

[0128] As in Fig. 6A and Figure 6B In the embodiment shown in FIG. 1 , it can be seen graphically between time t(0) and t(1) that Figure 6B Sensor pair in the partitioned spectral system Fig. 6A Each of the full spectrum systems in the embodiment is cycled at least four times. In one embodiment, the display device (LCD panel) operates at a speed of 50 to 60 frames per second. In such embodiments, Figure 6BThe partitioned optical system in can operate at a speed of 200 to 240 frames per second to maintain the continuity and smoothness of the displayed video. In other embodiments, there may be different capture and display frame rates. In addition, the average capture rate can be any multiple of the display rate.

[0129] In one embodiment, it may be desirable that not all partitions are represented equally within the system frame rate. In other words, not all light sources must be pulsed with the same regularity in order to emphasize and de-emphasize various aspects of the recorded scene as desired by the user. It should also be understood that invisible and visible partitions of the electromagnetic spectrum can be pulsed together within the system, with their corresponding data values ​​stitched into the video output for display to the user.

[0130] Exemplary embodiments may include the following pulse cycle patterns:

[0131] i. Green pulse;

[0132] ii. Red pulse;

[0133] iii. Blue pulse;

[0134] iv. Green pulse;

[0135] v.Red pulse;

[0136] vi.Blue pulse;

[0137] vii. Laser scanning pulse scheme;

[0138] viii. Fluorescence excitation pulse;

[0139] ix. Hyperspectral pulses;

[0140] x.(repeat)

[0141] Additional exemplary embodiments may include the following pulse cycle patterns:

[0142] i. Green pulse;

[0143] ii. Red pulse;

[0144] iii. Blue pulse;

[0145] iv. Fluorescence excitation pulse;

[0146] v.Hyperspectral pulse;

[0147] vi.Green pulse;

[0148] vii.Red pulse;

[0149] viii.Blue pulse;

[0150] ix. Fluorescence excitation pulse;

[0151] x.Hyperspectral pulse;

[0152] xi. Laser scanning pulse scheme;

[0153] xii.(repeat)

[0154] The pulse pattern may be altered to suit the imaging objectives of a particular implementation. An exemplary imaging objective is to obtain hyperspectral imaging data and fluorescence imaging data, and also to obtain laser mapping and / or tool tracking data based on analysis of the hyperspectral and / or fluorescence imaging data. In such examples, the laser mapping and / or tool tracking data may be analyzed for certain areas of the scene that have been highlighted by the hyperspectral and / or fluorescence imaging data. Another exemplary imaging objective is to obtain hyperspectral imaging data or fluorescence imaging data, and also to obtain laser mapping and / or tool tracking data. Another exemplary imaging objective is to obtain laser mapping and / or tool tracking data. Another exemplary imaging objective is to obtain hyperspectral imaging data. Another exemplary imaging objective is to obtain fluorescence imaging data. It should be understood that the imaging objectives may be application-specific, depending on the reason for deploying the imaging system. In addition, the imaging objectives may change during a single imaging session, and the pulse pattern may be altered to match the changing imaging objectives.

[0155] As can be seen in this example, a laser scan partition can be pulsed at a different rate than the other partitions. Doing so can emphasize a certain aspect of the scene, where the laser scan data is simply overlapped with other data in the video output to make the desired emphasis. It should be noted that adding a laser scan partition on top of the red partition, green partition, and blue partition does not necessarily require a serialized system to operate at four times the rate of a full-spectrum non-serial system, because each partition does not have to be equally represented in the pulse pattern. As seen in this embodiment, adding a partition pulse that is represented less in the pulse pattern (laser scan in the above example) will result in an increase of less than 20% of the sensor's cycle speed to accommodate irregular partition sampling.

[0156] In various embodiments, the pulse cycle pattern may also include any of the following wavelengths in any suitable order. Such wavelengths may be particularly suitable for exciting fluorescent agents to generate fluorescent imaging data by sensing the relaxation emission of the fluorescent agent based on the relaxation emission of the fluorescent agent:

[0157] i.770±20nm;

[0158] ii.770±10nm;

[0159] iii.770±5nm;

[0160] iv.790±20nm;

[0161] v.790±10nm;

[0162] vi.790±5nm;

[0163] vii.795±20nm;

[0164] viii.795±10nm;

[0165] ix.795±5nm;

[0166] x.815±20nm;

[0167] xi.815±10nm;

[0168] xii.815±5nm;

[0169] xiii. 770nm to 790nm; and / or

[0170] xiv.795nm to 815nm.

[0171] In various embodiments, the pulse cycle may also include any of the following wavelengths in any suitable order. Such wavelengths may be particularly suitable for generating hyperspectral imaging data:

[0172] i.513±545nm;

[0173] ii.565nm to 585nm;

[0174] iii.900nm to 1000nm;

[0175] iv.513±5nm;

[0176] v.513±10nm;

[0177] vi.513±20nm;

[0178] vii.513±30nm;

[0179] viii.513±35nm;

[0180] ix.545±5nm;

[0181] x.545±10nm;

[0182] xi.545±20nm;

[0183] xii.545±30nm;

[0184] xiii.545±35nm;

[0185] xiv.565±5nm;

[0186] xv.565±10nm;

[0187] xvi.565±20nm;

[0188] xvii.565±30nm;

[0189] xviii.565±35nm;

[0190] xix.585±5nm;

[0191] xx.585±10nm;

[0192] xxi.585±20nm;

[0193] xxii.585±30nm;

[0194] xxiii.585±35nm;

[0195] xxiv.900±5nm;

[0196] xxv.900±10nm;

[0197] xxvi.900±20nm;

[0198] xxvii.900±30nm;

[0199] xxviii.900±35nm;

[0200] xxix.1000±5nm;

[0201] xxx.1000±10nm;

[0202] xxxi.1000±20nm;

[0203] xxxii.1000±30nm; or

[0204] xxxiii.1000±35nm.

[0205] The partition loop can be divided into various imaging and video standards to accommodate or approximate. In one embodiment, the partition loop may include the following: 7A to 7D Pulses of electromagnetic energy in the red, green, and blue spectrum are best shown in FIG. Fig. 7A In , different light intensities have been achieved by modulating the light pulse width or duration within the operating range shown by the vertical grey dashed line. Figure 7BIn , different light intensities have been achieved by modulating the power of an optical or electromagnetic emitter (which may be a laser or LED emitter), but keeping the pulse width or duration constant. Figure 7C The case where both the optical power and optical pulse width are modulated for greater flexibility is shown. The zoned cycles can use cyan magenta yellow (CMY), infrared, ultraviolet, hyperspectral and fluorescent, which use invisible pulse sources mixed with visible pulse sources, and any other color space required to produce an image or approximate a desired video standard currently known or yet to be developed. It should also be understood that the system can switch between color spaces on the fly to provide the desired image output quality.

[0206] When using the color space green-blue-green-red (such as Fig.7D In an embodiment (as shown in ), it may be desirable to pulse the luma component more frequently than the chroma component because users are generally more sensitive to differences in light magnitude than to differences in light color. Fig.7D The monochrome sensor shown in Figure utilises this principle. Fig.7D In the scheme, green, which contains the most brightness information, can be pulsed more frequently or with greater intensity to obtain brightness data in the (GBGRGBGR...) scheme. Such a configuration will create a video stream with more perceptible details without creating and transmitting imperceptible data.

[0207] In one embodiment, duplicate pulses of weaker partitions may be used to produce an output that has been adjusted for the weaker pulses. For example, given the weak sensitivity of blue lasers relative to silicon-based pixels, and the difficulty of producing blue lasers compared to red or green light, it may be pulsed more frequently during a frame cycle to compensate for the weakness of the light. These additional pulses may be done continuously over time, or by using multiple lasers to pulse simultaneously to produce the desired compensating effect. It should be noted that by pulsing during blanking periods (time during which the sensor is not reading out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same kind, and simply accumulates the light for the desired output. In another embodiment, the maximum light pulse range may be different from frame to frame. This may be done in a controlled manner. Fig. 7E , where the light pulses are different from frame to frame. The sensor can be constructed to be able to program different blanking periods in a repeating pattern of two frames or three frames or four frames or n frames. Fig. 7E In FIG. 4 , four different light pulses are shown, and pulse 1 may be repeated, for example, after pulse 4, and may have a pattern of four frames with different blanking periods. This technique may be used to place the most powerful partition at the smallest blanking period, and thus allow the weakest partition to have a wider pulse on one of the subsequent frames without increasing the readout speed. The reconstructed frame may still have a regular pattern from frame to frame, since it is made up of many pulse frames.

[0208] Figure 8 A process flow 800 is shown in which a sensor and / or emitter are adjusted to compensate for differences in energy values ​​of a pulse partition spectrum. In process flow 800, data is obtained from a histogram of a previous frame and analyzed at 802. The sensor is adjusted at 804, and the emitter is adjusted at 806. At 808, an image is determined based on the adjusted sample time from the sensor, and / or at 810, an image is determined based on the adjusted (increased or decreased) emitted light.

[0209] Fig. 9 9 is a process flow 900 for adjusting a sensor and recording a frame based on readings from the adjusted sensor. In the process flow 900, a histogram of a previous frame is obtained at 902 and the sensor is adjusted based on sensitivity at 904. At 906, a frame is recorded based on the readings from the adjusted sensor. In one example, the process flows 800, 900 are implemented because the red light spectrum is more easily detected by the sensor within the system than the blue light spectrum. In this example, because the blue partition has a low quantum efficiency relative to silicon (in Fig. 9 ), so the sensor is adjusted to be less sensitive during the red partition cycle and more sensitive during the blue partition cycle. Similarly, the emitter can be adjusted to provide adjusted partitions (e.g., higher or lower intensity and duration).

[0210] Figure 10 to Figure 11 is a schematic diagram of a method for performing color correction on digital imaging. It is common in digital imaging to manipulate values ​​within the image data to correct the output to meet the expectations of a user or to highlight certain aspects of the imaged object. In systems where light is controlled by the user, it is advantageous to provide light emissions that are known to the user and may be only a portion of the electromagnetic spectrum or multiple portions of the full electromagnetic spectrum. Image correction is important to meet the expectations of the user and to detect faults within the system. One calibration method employs a table of expected values ​​for a given imaging condition that can be compared to the data from the image sensor.

[0211] Fig.101 is a schematic diagram of a method 1000 for performing color correction on digital imaging in a light-deficient environment. The method 1000 includes sampling a neutral color scene or palette (see 1010) at startup by running a full cycle of electromagnetic spectrum partitions at 1002. A lookup table 1008 is generated based on the color neutral scene or palette 1010. The lookup table 1008 is used to determine a histogram for a frame cycle at 1008. At 1006, the histogram is compared to known or expected values ​​based on the neutral color scene or palette 1010 and also based on the lookup table 1008. The method includes adjusting settings on an image signal processor (ISP) at 1012 and / or adjusting an emitter at 1014. The adjustment of the emitter at 1014 may include adjustment of any aspect of the emitted light, such as the magnitude, duration (i.e., on-time), or range within the spectrum partition.

[0212] It should be noted that because each partitioned spectrum of light can have different energy values, the sensor and / or light emitter can be adjusted to compensate for the difference in energy values. For example, in one embodiment, because the blue light spectrum has a lower quantum efficiency than the red light spectrum with respect to silicon-based imagers, the responsiveness of the sensor can then be adjusted to respond less during the red cycle and more during the blue cycle. Conversely, because the quantum efficiency of blue light is lower than that of red light, the emitter can emit blue light at a higher intensity than red light to produce a correctly exposed image.

[0213] Fig.11 1 is a schematic diagram of a method 1100 for performing hierarchical adjustments to an image signal processor (ISP) and / or a transmitter to reduce the amount of noise and artifacts within an output image stream or video. The method 1100 includes transmitting and sensing a full cycle of an electromagnetic spectrum partition at 1102. At 1104, the results of the full cycle of the electromagnetic partition are compared with the desired output. Based on the comparison, the image signal processor (ISP) is adjusted at 1106 and / or the transmitter is adjusted at 1108. The adjustments made to the ISP at 1106 and / or the transmitter at 1108 between frame cycles can be limited at 1110. For example, the transmitter can be adjusted by a portion of its operating range at any time between frames. Similarly, the ISP can be adjusted by a portion of its operating range at any time between frames. In one embodiment, both the transmitter and the ISP can be limited so that only a portion of the respective operating ranges of the transmitter and sensor can be adjusted together at any time between frames. The results of these hierarchical adjustments are compared at 1112, and adjustments are completed based on the comparison.

[0214] In an exemplary embodiment, a stepped adjustment of the ISP and / or emitter may be performed at approximately 0.1 dB of the operating range of the component to correct for the exposure of the previous frame. 0.1 dB is merely an example, and it should be noted that in other embodiments, the allowed adjustment of the component may be any portion of its respective operating range. Components of the system may be changed by intensity or duration adjustments that are typically controlled by the number of bits (resolution) output by the component. Component resolution may typically range from approximately 10 bits to 24 bits, but should not be limited to this range as it is intended to include resolutions of devices yet to be developed in addition to those currently available. For example, after the first frame, if it is determined that the scene is too blue when viewed, the emitter may be adjusted by stepped adjustments as described above (such as approximately 0.1 dB) to reduce the pulse magnitude or duration of blue light during the blue cycle of the system.

[0215] In this exemplary embodiment, more than 10% may be needed, but the system has limited itself to an operating range of 0.1 dB adjustment per system cycle. Therefore, the blue light can be adjusted again during the next system cycle if necessary. The stepped adjustment between cycles can have a damping effect on the output image, and will reduce noise and artifacts when operating the emitters and sensors at their operating limits. Any stepped amount that can determine the operating adjustment range of a component can be used as a limiting factor, or it can be determined that certain embodiments of the system may include components that are adjustable throughout their operating range.

[0216] Additionally, the optical black area of ​​any image sensor can be used to aid in image correction and noise reduction. In one embodiment, the values ​​read from the optical black area can be compared with those of the active pixel area of ​​the sensor to establish a reference point to be used for image data processing. Fig.12 The types of sensor correction methods that can be employed in a color pulse system are shown. CMOS image sensors typically have multiple non-idealities such as fixed pattern noise (FPN) and line noise. Fully controlling the illumination has the following benefits: an entire dark data frame can be periodically acquired and used to correct for pixel and column offsets.

[0217] FPN is dispersion in the sensing element offset, which is usually caused by pixel-to-pixel dispersion caused by random variations in dark current from photodiode to photodiode. Column fixed pattern noise is caused by offsets associated with specific columns of pixels in the readout chain and can produce perceived vertical streaks within the image.

[0218] Row noise is the random temporal variation of pixel offsets within each row. Because row noise is temporal, the correction must be recalculated for each row and each frame. To do this, there are typically a number of light-blind (OB) pixels within each row in the array that must first be sampled to estimate the row offset before sampling the light-sensitive pixels. The row offset is then subtracted during the row noise correction process.

[0219] exist Fig.12 In the example of FIG. 1200 , there are other corrections involving obtaining data in the correct order, monitoring and controlling voltage offsets in the analog domain (black clamping), and identifying / correcting individual defective pixels. The process flow 1200 includes cycling the sensor through each of the electromagnetic partitions at a first emission intensity at 1202. The process flow 1200 includes cycling the sensor through each of the electromagnetic partitions at a second emission intensity at 1204. The process flow 1200 includes combining data from the electromagnetic partitions at the first emission intensity and the second emission intensity at 1206.

[0220] Fig.13 1 is a schematic diagram of a process flow 1300 for increasing the dynamic range of a resulting image. The process flow 1300 includes cycling the sensor through each of the electromagnetic sectors at a first emission intensity at 1302. The process flow 1300 includes cycling the sensor through each of the electromagnetic sectors at a second emission intensity at 1304. The process flow 1300 includes cycling the sensor through “n” electromagnetic sectors at an emission intensity of “m” at 1306, and may be repeated any suitable number of times. The process flow 1300 includes cycling the sensor through “n+i” electromagnetic sectors at an emission intensity of “m+j” at 1308. The process flow 1300 includes combining data from each of the cycled emission intensities at 1310.

[0221] In one embodiment, the exposure input may be input at different levels over time and combined to produce a greater dynamic range. A greater dynamic range may be particularly desirable due to the limited spatial environment in which the imaging device is used. In a limited spatial environment that lacks light or is dark except for the light provided by the light source, and where the light source is close to the light emitter, exposure has an exponential relationship with distance. For example, objects close to the light source and the optical opening of the imaging device tend to be overexposed, while objects farther away tend to be extremely underexposed because there is very little (any) ambient light.

[0222] As in Fig.13As can be seen in FIG. 1 , the cycle of a system having electromagnetic energy emission in multiple partitions can be continuously cycled according to the partitions of the electromagnetic spectrum. For example, in an embodiment in which the emitter emits lasers in different red partitions, different blue partitions, different green partitions, and different laser scanning partitions, the two cycle data sets to be combined can be of the following form:

[0223] i. Red with an intensity of one at 1302;

[0224] ii. Red with an intensity of two at 1304;

[0225] iii. Blue with an intensity of one at 1302;

[0226] iv. Blue with an intensity of two at 1304;

[0227] v. Green with an intensity of one at 1302;

[0228] vi. Green with an intensity of two at 1304;

[0229] vii. A laser scan at an intensity of one at 1302; and

[0230] viii. Laser scan at 1304 with an intensity of two.

[0231] Alternatively, the system can loop as follows:

[0232] i. Red with an intensity of one at 1302;

[0233] ii. Blue with an intensity of one at 1302;

[0234] iii. Green with an intensity of one at 1302;

[0235] iv. A laser scan with an intensity of one at 1302;

[0236] v. Red with intensity two at 1304;

[0237] vi. Blue with an intensity of two at 1304;

[0238] vii. Green at an intensity of two at 1304; and

[0239] viii. Laser scan at 1304 with an intensity of two.

[0240] In such embodiments, the first image may be derived from intensity one values ​​and the second image may be derived from intensity two values ​​and then combined or processed into a complete image dataset at 1310 rather than at its component parts.

[0241] It is contemplated within the scope of the present invention that any number of transmit partitions may be used in any order. Fig.13 As shown in , "n" is used as a variable to represent any number of electromagnetic partitions, and "m" is used to represent any level of strength of the "n" partitions. Such a system can be looped as follows:

[0242] i. n with intensity m at 1306;

[0243] ii. n+1 with strength m+1;

[0244] iii. n+2 with strength m+2; and

[0245] iv. n+i with intensity m+j at 1308.

[0246] Thus, any sequenced cyclic pattern may be used to produce the desired image correction, where "i" and "j" are additive values ​​within the operating range of the imaging system.

[0247] Fig.14 A process flow 1400 is shown to be implemented by a controller and / or a monochrome image signal processor (ISP) for generating a video stream of RGB images with laser scan data superimposed thereon. How an image signal processor (ISP) chain may be assembled for the purpose of generating a sequence of sRGB images from raw sensor data produced in the presence of a GRGB-special light pulse scheme. In process flow 1400, the first stage involves performing a correction (see Fig.14 Receive data from the sensor at 1402, reorder at 1404, and sensor correction at 1406) to account for any non-ideal factors in the sensor technology that are best suited to work in the raw data domain. In the next stage, multiple frames (e.g., green frame 1408a, red-blue frame 1408b, and special frame 1408c) are buffered because each final frame derives data from multiple raw frames. Frame reconstruction at 1264 continues by sampling data from the current frame and two buffered frames (see 1408a, 1408b and / or 1408c). The reconstruction process produces a full-color frame including hyperspectral image data in linear RGB color space. In this example, before converting to YCbCr space at 1422, white balance coefficients at 1268 and color correction matrices at 1420 are applied for subsequent edge enhancement at 1424. After edge enhancement at 1424, the image is converted back to linear RGB at 1426 for scaling at 1428 (if applicable). Finally, a gamma transfer function is applied at 1430 to convert the data to the sRGB domain at 1432 .

[0248] Fig.151504 and 1505. The color fusion hardware 1500 uses a memory writer 1502 to store an RGBG-special video data stream in a memory 1504 and converts the video data stream into a parallel RGB+special video data stream at 1505. The bit width on the input side may be, for example, 12 bits per color. The output width of this example will be at least 36 bits per pixel. Other embodiments may have different initial bit widths and a number 3 times the output width. The memory writer 1502 block takes the RGBG-special video stream as its input and writes each frame to its correct frame memory 1504 (the memory writer triggers the same pulse generator that runs the laser source (see 1510)). The memory 1504 may store the exposure frame data in a pattern such as that shown, i.e., red, green 1, blue, green 2, special, and then start back again with red. The memory reader 1506 reads three frames at a time to construct an RGB pixel. Each pixel is three times the bit width of a single color component. At 1510, the memory reader 1506 also triggers the laser pulse generator. In one embodiment, the memory reader 1506 waits until the red frame, green 1 frame, and blue frame have been written, and then continues to read them out in parallel while the writer continues to write green 2, special, and begins returning to red. When red is complete, the reader begins reading from blue, green 2, special, and red. This pattern continues indefinitely.

[0249] Fig.16 16 is a schematic diagram of a process flow 1600 for a sensor correction process. The process flow 1600 can be used in color and hyperspectral pulse systems as discussed herein. The process flow 1600 can be used to counteract non-ideal factors in CMOS image sensors, such as fixed pattern noise (FPN) and row noise. Fixed pattern noise is the dispersion in the offset of the sensing element. Typically, most of the FPN is inter-pixel dispersion, which originates from random variations in dark current from photodiode to photodiode and other sources. The system disclosed herein maintains full control over the illumination source, and this enables dark data to be collected and used to correct pixel and column offsets. In the illustrated example, a single frame buffer can be used to perform a sliding average over the entire frame in the absence of light using, for example, a simple exponential smoothing filter. During normal operation, this dark average frame can be subtracted from each illumination frame. Row noise is the random temporal variation of pixel offsets within each row. Because row noise is temporal, corrections are calculated for each row and each frame. To this end, there are typically many light-blind (OB) pixels within each row in the pixel array. Before sampling the photosensitive pixels, the OB pixels must first be sampled to evaluate the line offset. The row offset is then subtracted during the row noise correction process.

[0250] The process flow 1600 includes performing top deserialization 1602 and bottom deserialization 1603. The process flow 1600 includes performing line reordering at the top port at 1604 and performing line reordering at the bottom port at 1605. Upon completion of the line reordering, the information may be stored in a separate database 1632, 1634 or other memory device. The process flow 1600 includes performing black clamp calculations on the top ADC at 1606 and performing black clamp calculations on the bottom ADC at 1607. The information exits the process flow 1600 on a first-in, first-out (FIFO) basis. The process flow 1600 includes performing row noise correction at the top ADC at 1608 and performing row noise correction at the bottom ADC at 1609. The process flow 1600 includes performing full row reordering at 1610 and performing dark frame accumulation at 1612. The information may be stored in a database 1630 or other storage device before performing fixed pattern noise (FPN) correction. The process flow includes performing fixed pattern noise (FPN) correction at 1614 and performing pixel defect correction at 1616. The process flow 1600 includes performing programmable digital gain at 1618 before the video stream leaves the process flow 1600 to be provided to a user.

[0251] Fig.17 is a schematic diagram of the pattern reconstruction process. Fig.17 The exemplary pattern shown includes red, green, blue and special light pulses each lasting a duration of T1. In various embodiments, the light pulses may have the same duration or different durations. The red exposure frame, the green exposure frame, the blue and special exposure frames are combined to generate an RGB image with special data stacked thereon. The special data may include one or more of fluorescence data, hyperspectral data and / or laser scanning data. A single image frame including a red exposure frame, a green exposure frame, a blue exposure frame and a special exposure frame requires a time period of 4*T1 to be generated. Fig.17 The durations shown are exemplary only and may vary for different specific implementations. In other embodiments, different pulse schemes may be used. For example, an embodiment may be based on the timing of each color component or frame (T1), and the reconstructed frame has a period that is twice the period of the input color frame (2×T1). Different frames within a sequence may have different frame periods, and the average capture rate may be any multiple of the final frame rate.

[0252] In one embodiment, the dynamic range of the system is increased by changing the pixel sensitivity of pixels within the pixel array of the image sensor. Some pixels may sense reflected electromagnetic radiation at a first sensitivity level, other pixels may sense reflected electromagnetic radiation at a second sensitivity level, and so on. Different pixel sensitivities may be combined to increase the dynamic range provided by the pixel configuration of the image sensor. In one embodiment, adjacent pixels are set with different sensitivities so that each cycle includes data generated by pixels that are more sensitive and less sensitive relative to each other. When multiple sensitivities are recorded in a single cycle of the pixel array, the dynamic range is increased. In one embodiment, a wide dynamic range can be achieved by having multiple global TXs, each TX firing only on different groups of pixels. For example, in global mode, the global TX1 signal is firing pixel group 1, the global TX2 signal is firing pixel group 2, the global TXn signal is firing pixel group n, and so on.

[0253] Fig.18A An example of the timing of two different pixel sensitivities (dual pixel sensitivity) in a pixel array is shown. In this case, the global TX1 signal fires half of the pixels of the array and the global TX2 fires the other half of the pixels. Because global TX1 and global TX2 have different "on" to "off" edge locations, the integrated light is different between TX1 pixels and TX2 pixels.

[0254] Fig.18B Different embodiments of timing for dual pixel sensitivity are shown. In this case, the light pulse is modulated twice (pulse duration and / or amplitude). The TX1 pixel integrates the P1 pulse, and the TX2 pixel integrates the P1+P2 pulse. Separating the global TX signal can be done in a variety of ways, including distinguishing a TX row from each row, and sending multiple TX rows per row, where each TX row addresses a different group of pixels.

[0255] FIG. 19A to FIG. 19C The use of pulsed and / or synchronized or constant white light emission and corresponding color sensors is shown. Fig.19A As can be seen in FIG. 1 , the white light emitter can be configured to emit a light beam during the blanking period of the corresponding sensor to provide a controlled light source in a controlled light environment. The light source can emit a light beam of constant magnitude and vary the duration of the pulse, such as Fig.19A Alternatively, the pulse may be kept constant while varying the magnitude to achieve correctly exposed data, such as Fig.19B shown. Fig.19C Shown in is a graphical representation of a constant light source that can be modulated with a varying current controlled by and synchronized with a sensor.

[0256] In one embodiment, white light or multi-spectral light is emitted as pulses to provide data for use within the system (in FIG. 19A to FIG. 19C ). White light emission combined with partitioning of the electromagnetic spectrum can be used to emphasize and de-emphasize certain aspects within a scene. Such an embodiment may use the following pulse pattern:

[0257] i. Green pulse;

[0258] ii. Red pulse;

[0259] iii. Blue pulse;

[0260] iv. Special pulse;

[0261] v.Green pulse;

[0262] vi.Red pulse;

[0263] vii. Blue pulse;

[0264] viii.Special pulses;

[0265] ix. White light (multi-spectrum) pulses;

[0266] x.(repeat)

[0267] Any system using an image sensor cycle at least twice faster than white light cycle is intended to fall within the scope of the present disclosure.It should be understood that any combination of partitions of the electromagnetic spectrum are contemplated herein, whether from the visible or invisible spectrum of the full electromagnetic spectrum.

[0268] FIG. 20A to FIG. 20C Each shows a light source 2000 having multiple emitters. The emitters include a first emitter 2002, a second reflector 2004, and a third emitter 2006. Additional emitters may be included, as discussed further below. The emitters 2002, 2004, and 2006 may include one or more laser generators that emit light having different wavelengths. For example, the first emitter 2002 may emit a wavelength consistent with a blue laser, the second emitter 2004 may emit a wavelength consistent with a green laser, and the third emitter 2006 may emit a wavelength consistent with a red laser. For example, the first emitter 2002 may include one or more blue lasers, the second emitter 2004 may include one or more green lasers, and the third emitter 2006 may include one or more red lasers. The lasers 2002, 2004, 2006 emit laser beams toward a collection area 2008, which may be a waveguide, a lens, or a device for collecting light and / or transmitting light to a waveguide (such as Figure 2 The jumper waveguide 206 or the intracavity waveguide 210) provides a location for other optical components of the light.

[0269] In a specific implementation where a reagent or dye has been administered to a patient that aids in identifying certain tissues, structures, chemical reactions, biological processes, etc., the emitters 2002, 2004, and 2006 may emit wavelengths for causing the reagent or dye to fluoresce. Such wavelengths may be determined based on the reagent or dye administered to the patient. In such embodiments, the emitters may need to be highly accurate in order to emit the desired wavelengths to cause certain reagents or dyes to fluoresce or activate.

[0270] In one specific implementation, the emitters 2002, 2004, and 2006 emit hyperspectral wavelengths of electromagnetic radiation. Certain hyperspectral wavelengths can penetrate tissue and enable a medical practitioner to "see through" tissue in the foreground to identify chemical processes, structures, compounds, biological processes, etc., located behind the tissue in the foreground. Hyperspectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc., which are known to have specific spectral responses.

[0271] In one specific implementation, the transmitters 2002, 2004, and 2006 transmit laser scan patterns for mapping the topology of a scene and / or for calculating the size and distance between objects in the scene. In one embodiment, the endoscopic imaging system is used in conjunction with multiple tools such as scalpels, retractors, clamps, etc. In such embodiments, each of the transmitters 2002, 2004, and 2006 can emit a laser scan pattern such that the laser scan pattern is projected onto each tool individually. In such embodiments, the laser scan data of each of the tools can be analyzed to identify the distance between the tool and other objects in the scene.

[0272] exist Fig. 20B In an embodiment, emitters 2002, 2004, 2006 each deliver laser light to collection area 2008 at different angles. Changes in angle can result in changes in the location of the electromagnetic energy in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at collection area 2008, the changing angles can result in different amounts of light entering different fibers. For example, the angles can result in intensity variations across collection area 2008. In addition, light from different emitters may not be mixed evenly, so some fibers may receive different amounts of different colors of light. Variations in the color or intensity of light in different fibers can result in non-optimal illumination of the scene. For example, variations in the delivered light or light intensity can result in both the scene and the captured image.

[0273] In one embodiment, an intermediate optical element may be placed between the fiber bundle and the transmitter 2002, 2004, 2006 to mix light of different colors (wavelengths) before entering the fiber or other waveguide. Exemplary intermediate optical elements include diffusers, mixing rods, one or more lenses, or other optical components for mixing light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. This mixing can result in the same color in each fiber, but, in some embodiments, can still result in different total brightness delivered to different fibers. In one embodiment, the intermediate optical element can also spread or evenly spread the light over the collection area so that each fiber carries the same total amount of light (e.g., the light may be spread out in a top hat profile). Diffusers or mixing rods can cause light loss.

[0274] Although the collection area in 2008 Fig. 20A 2008 is represented as a physical component, but the collection region 2008 may simply be the region that delivers light from the emitters 2002, 2004, and 2006. In some cases, the collection region 2008 may include optical components such as diffusers, mixing rods, lenses, or any other intervening optical components between the emitters 2002, 2004, 2006 and the output waveguides.

[0275] Fig. 20C An embodiment of a light source 2000 is shown having emitters 2002, 2004, 2006 that provide light to a collection area 2008 at the same or substantially the same angle. The light is provided at an angle substantially perpendicular to the collection area 2008. The light source 2000 includes a plurality of dichroic mirrors, including a first dichroic mirror 2010, a second dichroic mirror 2012, and a third dichroic mirror 2014. The dichroic mirrors 2010, 2012, 2014 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 2014 may reflect blue laser light provided by a third emitter, while being transparent to red and green light provided by each of the first and second emitters 2002, 2004. The second dichroic mirror 2012 may be transparent to light from the first emitter 2002, but reflective to light from the second emitter 2004. If other colors or wavelengths are included, the dichroic mirrors may be selected to reflect light corresponding to at least one emitter and be transparent to other emitters. For example, the third dichroic mirror 2014 reflects light from the third emitter 2006, but is transparent to emitters "behind" it, such as the first emitter 2002 and the second emitter 2004. In embodiments where there are dozens or hundreds of emitters, each dichroic mirror may reflect the corresponding emitter and the emitter in front of it, while being transparent to the emitter behind it. This may allow dozens or hundreds of emitters to emit electromagnetic energy to the collection area 2008 at substantially the same angle.

[0276] Because these dichroic mirrors allow other wavelengths to be transmitted or passed through, each of these wavelengths can arrive at the collection area 2008 from the same angle and / or with the same center point or focus. Providing light from the same angle and / or the same focus / center point can significantly improve reception and color mixing at the collection area 2008. For example, a particular fiber can receive different colors in the same proportions as they are transmitted / reflected by the emitters 2002, 2004, 2006 and the mirrors 2010, 2012, 2014. Fig. 20B Compared to the embodiment of the present invention, the light mixing can be significantly improved at the collection area. In one embodiment, any optical component discussed herein can be used at the collection area 2008 to collect light before providing it to the fiber or fiber bundle.

[0277] Fig. 20C An embodiment of a light source 2000 having emitters 2002, 2004, 2006 that also provide light to a collection area 2008 at the same or approximately the same angle is shown. For example, the light incident on the collection area 2008 is offset from the vertical. Angle 2016 indicates the angle of offset from the vertical. In one embodiment, the laser emitters 2002, 2004, 2006 may have a Gaussian cross-sectional intensity profile. As previously described, the improved distribution of light energy between fibers can be achieved by forming a flatter or top-hat-shaped intensity profile. In one embodiment, as the angle 2016 increases, the intensity across the collection area 2008 approaches a top-hat profile. For example, by increasing the angle 2016 until the profile is sufficiently flat, the top-hat profile can even approximate a non-flat output beam. The top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intermediate optical components between the emitters 2002, 2004, 2006 and the output waveguide, fiber or fiber bundle.

[0278] Fig.21 21 is a schematic diagram showing a single optical fiber 2102 outputted via a diffuser 2104 at the output. In one embodiment, the optical fiber 2102 has a diameter of 500 microns, a numerical aperture of 0.65, and emits a light cone 2106 of about 70 or 80 degrees without the diffuser 2104. With the diffuser 2104, the light cone 2106 may have an angle of about 110 or 120 degrees. The light cone 2106 may be the bulk of where all the light arrives and is evenly distributed. The diffuser 2104 may allow for a more even distribution of the electromagnetic energy of the scene observed by the image sensor.

[0279] In one embodiment, the intracavity waveguide 210 comprises a single plastic or glass optical fiber of about 500 microns. Plastic fibers are relatively low in cost, but their width may allow the fiber to carry a sufficient amount of light to the scene through coupling, diffusion, or other losses. For example, a smaller fiber may not be able to carry as much light or power as a larger fiber. The intracavity waveguide 210 may comprise a single or multiple optical fibers. The intracavity waveguide 210 may receive light directly from a light source or via a jumper waveguide. A diffuser may be used to widen the light output 206 to obtain a desired field of view for an image sensor 214 or other optical component.

[0280] Although in FIG. 20A to FIG. 20C Three emitters are shown, but in some embodiments, a number ranging from one to hundreds or more emitters may be used. The emitters may have different wavelengths or spectra of light that they emit, and these lights may be used to continuously cover the desired portion of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra). The emitters may be configured to be able to emit visible light such as red, green, and blue light, and may also be configured to be able to emit hyperspectral emissions of electromagnetic radiation, fluorescence excitation wavelengths for causing agents to fluoresce, and / or laser mapping patterns for calculating parameters and distances between objects in a scene.

[0281] Fig. 22 A portion of the electromagnetic spectrum 2200 is shown divided into twenty different sub-spectra. The number of sub-spectra is exemplary only. In at least one embodiment, the spectrum 2200 can be divided into hundreds of sub-spectra, each with a small band. The spectrum can extend from the infrared spectrum 2202, through the visible spectrum 2204, and into the ultraviolet spectrum 2206. The sub-spectra each have a band 2208 covering a portion of the spectrum 2200. Each band can be defined by an upper wavelength and a lower wavelength.

[0282] In one embodiment, at least one emitter (such as a laser emitter) is included in a light source (such as light source 202, 2000) to provide complete and continuous coverage of the entire spectrum 2200. For example, a light source for providing coverage of the sub-spectrum shown may include at least 20 different emitters, at least one emitter for each sub-spectrum. In one embodiment, each emitter covers a spectrum covering 40 nanometers. For example, one emitter may emit light in a band from 500nm to 540nm, while another emitter may emit light in a band from 540nm to 580nm. In another embodiment, the emitters may cover bands of other sizes, depending on the type of available emitters or imaging needs. For example, a plurality of emitters may include a first emitter covering a band from 500nm to 540nm, a second emitter covering a band from 540nm to 640nm, and a third emitter covering a band from 640nm to 650nm. Each emitter may cover a different segment of the electromagnetic spectrum ranging from far infrared, mid-infrared, near infrared, visible light, near ultraviolet, and / or far ultraviolet. In some cases, multiple emitters of the same type or wavelength may be included to provide sufficient output power for imaging. The number of emitters required for a particular wavelength band may depend on the sensitivity of the monochromatic sensor to the wavelength band and / or the power output capability of the emitters in that wavelength band.

[0283] The bandwidth and coverage provided by the emitter can be selected to provide any desired combination of spectra. For example, continuous coverage of spectra using very small bandwidths (e.g., 10 nm or less) can allow highly selective hyperspectral and / or fluorescence imaging. The bandwidth can allow selective emission of excitation wavelengths of one or more specific fluorescent agents. In addition, the bandwidth can allow selective emission of certain portions of hyperspectral electromagnetic radiation for identification of specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths come from emitters that can be selectively activated, extreme flexibility in causing one or more specific fluorescent agents to fluoresce during an inspection can be achieved. In addition, extreme flexibility in identifying one or more objects or processes through hyperspectral imaging can be achieved. Therefore, more fluorescence and / or hyperspectral information can be achieved in less time and within a single inspection, which would otherwise require multiple inspections, delays due to the application of dyes or stains, etc.

[0284] Fig.232300 is a schematic diagram showing the timing diagram 2300 for generating the emission and readout of the image. The solid line represents the readout (peak 2302) and blanking period (valley) for capturing a series of exposure frames 2304-2314. A series of exposure frames 2304-2314 may include a series of repeated exposure frames that can be used to generate laser scanning, hyperspectral and / or fluorescence data that can be stacked on an RGB video stream. In one embodiment, a single image frame includes information from a plurality of exposure frames, wherein one exposure frame includes red image data, another exposure frame includes green image data, and another exposure frame includes blue image data. In addition, a single image frame may include one or more of hyperspectral image data, fluorescence image data, and laser scanning data. The plurality of exposure frames are combined to produce a single image frame. A single image frame is an RGB image with fluorescence imaging data. The series of exposure frames includes a first exposure frame 2304, a second exposure frame 2306, a third exposure frame 2308, a fourth exposure frame 2310, a fifth exposure frame 2312, and an Nth exposure frame 2326.

[0285] In addition, hyperspectral image data, fluorescent image data, and laser scanning data can be used in combination to identify key tissues or structures and also measure the size of those key tissues or structures. For example, hyperspectral image data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, cancerous tissues, etc. The location and identification of the key structure can be received from the corresponding system and can also be used to generate the topology of the key structure using laser scanning data. For example, the corresponding system determines the location of a cancerous tumor based on hyperspectral imaging data. Because the location of the cancerous tumor is known based on the hyperspectral imaging data, the topology and distance of the cancerous tumor can be calculated based on the laser scanning data. This example can also be applicable when identifying a cancerous tumor or other structure based on fluorescent imaging data.

[0286] In one embodiment, each exposure frame is generated based on at least one pulse of electromagnetic energy. The electromagnetic energy pulses are reflected and detected by the image sensor and then read out in a subsequent readout (2302). Thus, each blanking period and readout results in an exposure frame for a particular spectrum of electromagnetic energy. For example, a first exposure frame 2304 may be generated based on the spectrum of the first one or more pulses 2316, a second exposure frame 2306 may be generated based on the spectrum of the second one or more pulses 2318, a third exposure frame 2308 may be generated based on the spectrum of the third one or more pulses 2320, a fourth exposure frame 2310 may be generated based on the spectrum of the fourth one or more pulses 2322, a fifth exposure frame 2312 may be generated based on the spectrum of the fifth one or more pulses 2324, and an Nth exposure frame 2326 may be generated based on the spectrum of the Nth one or more pulses 2326.

[0287] Pulses 2316-2326 may include energy from a single emitter or from a combination of two or more emitters. For example, a spectrum included in a single readout cycle or within multiple exposure frames 2304-2314 may be selected for the desired inspection or detection of a specific tissue or condition. According to one embodiment, one or more pulses may include visible spectrum light for generating an RGB or black and white image, while emitting one or more additional pulses to fluoresce a fluorescent agent. For example, pulse 2316 may include red light, pulse 2318 may include blue light, and pulse 2320 may include green light, while the remaining pulses 2322-2326 may include wavelengths and spectra for detecting a specific tissue type, causing the agent to fluoresce, and / or marking the topology of the scene. For another example, the pulses of a single readout cycle include spectra (e.g., different fragments of the electromagnetic spectrum) generated by multiple different emitters that can be used to detect a specific tissue type. For example, if a combination of wavelengths causes a pixel to have a value that exceeds or is below a threshold, the pixel may be classified as corresponding to a specific type of tissue. Each frame can also be used to narrow down the type of tissue present at that pixel (e.g., and every pixel in the image) to provide a very specific classification of the tissue and / or the state of the tissue (disease / healthy) based on the spectral response of the tissue and / or the presence or absence of fluorescent agents at the tissue.

[0288] Multiple frames 2304-2314 are shown with readout periods of varying lengths and pulses of varying lengths or intensities. Blanking periods, pulse lengths or intensities, etc. may be selected based on the sensitivity of the monochromatic sensor to a particular wavelength, the power output capability of the transmitter, and / or the carrying capacity of the waveguide.

[0289] In one embodiment, dual image sensors may be used to obtain a three-dimensional image or video feed.Three-dimensional examination may allow for an improved understanding of the three-dimensional structure of the examination region and mapping of different tissue or material types within the region.

[0290] In an exemplary implementation, a fluorescent agent is provided to a patient, and the fluorescent agent is configured to be able to attach to cancer cells. It is known that the fluorescent agent fluoresces when irradiated by a specific partition of electromagnetic radiation. The relaxation wavelength of the fluorescent agent is also known. In an exemplary implementation, the patient is imaged using an endoscopic imaging system as discussed herein. The endoscopic imaging system pulses partitions of light of red, green, and blue wavelengths to generate an RGB video stream of the inside of the patient's body. In addition, the endoscopic imaging system pulses the electromagnetic radiation excitation wavelength of the fluorescent agent applied to the patient. In the example, the patient has cancer cells, and the fluorescent agent attaches to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength of the fluorescent agent, the fluorescent agent will fluoresce and emit a relaxation wavelength. If cancer cells are present in a scene imaged by the endoscopic imaging system, the fluorescent agent will also be present in the scene and emit its relaxation wavelength after fluorescing due to the emission of the excitation wavelength. The endoscopic imaging system senses the relaxation wavelength of the fluorescent agent, thereby sensing the presence of the fluorescent agent in the scene. Because the fluorescent agent is known to attach to cancer cells, the presence of the fluorescent agent also indicates the presence of cancer cells in the scene. The endoscopic imaging system thereby identifies the location of cancer cells within the scene. The endoscopic imaging system may also emit a laser scanning pulse scheme for generating the topology of the scene and calculating the size of objects within the scene. The location of the cancer cells (as identified by the fluorescence imaging data) may be combined with the topological and size information calculated based on the laser scanning data. Thus, the precise location, size, dimensions, and topology of the cancer cells may be identified. This information may be provided to a medical practitioner to aid in the removal of cancer cells. Additionally, this information may be provided to a robotic surgical system to enable the surgical system to remove cancer cells.

[0291] In another exemplary implementation, an endoscopic imaging system is used to image a patient to identify quantitative diagnostic information about the patient's tissue pathology. In this example, the patient is suspected or known to have a disease that can be tracked using hyperspectral imaging to observe the progression of the disease in the patient's tissue. The endoscopic imaging system pulses partitions of red, green, and blue wavelengths of light to generate an RGB video stream inside the patient's body. In addition, the endoscopic imaging system pulses light of one or more hyperspectral wavelengths, which allows the system to "see through" some tissues and generate imaging of tissues affected by the disease. The endoscopic imaging system senses the reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue, thereby identifying the location of the diseased tissue in the patient's body. The endoscopic imaging system can also emit a laser scanning pulse scheme for generating the topology of the scene and calculating the size of the object in the scene. The location of the diseased tissue (identified by the hyperspectral imaging data) can be combined with the topological and size information calculated using the laser scanning data. Therefore, the precise location, size, size, and topology of the diseased tissue can be identified. This information can be provided to a practicing physician to help remove, image, or study the diseased tissue. Additionally, this information may be provided to a robotic surgical system to enable the surgical system to resect diseased tissue.

[0292] Fig.24 24 is a schematic diagram of an imaging system 2400 having a single cutoff filter. System 2400 may be particularly suitable for implementations of fluorescent agent imaging. System 2400 includes an endoscope 2406 or other suitable imaging device having a light source 2408 for use in a light-deficient environment. Endoscope 2406 includes an image sensor 2404 and a filter 2402 for filtering out unwanted wavelengths of light or other electromagnetic radiation before reaching image sensor 2404. Light source 2408 transmits light that can illuminate surface 2262 in a light-deficient environment such as a body cavity. Light 2260 reflects from surface 2262 and passes through filter 2402 before striking image sensor 2404.

[0293] Filter 2402 can be used in specific implementations where a fluorescent agent or dye has been applied. In such embodiments, light source 2408 emits an excitation wavelength for causing the fluorescent agent or dye to fluoresce. Typically, the relaxation wavelength emitted by the fluorescent agent or dye will have a wavelength different from the excitation wavelength. Filter 2402 can be selected to filter out the excitation wavelength and only allow the relaxation wavelength to pass through the filter and be sensed by image sensor 2404.

[0294] In one embodiment, the filter 2402 is configured to filter out the electromagnetic radiation excitation wavelength that causes the reagent or dye to fluoresce, so that only the expected relaxation wavelength of the fluorescing reagent or dye is allowed to pass through the filter 2402 and reach the image sensor 2404. In one embodiment, the filter 2402 filters out the fluorescent reagent excitation wavelength at least between 770nm and 790nm. In one embodiment, the filter 2402 filters out the fluorescent reagent excitation wavelength at least between 795nm and 815nm. In one embodiment, the filter 2402 filters out the fluorescent reagent excitation wavelength at least between 770nm and 790nm and between 795nm and 815nm. In these embodiments, the filter 2402 filters out the excitation wavelength of the reagent and allows only the relaxation wavelength of the fluorescent reagent to be read by the image sensor 2404. The image sensor 2404 can be a wavelength-indeterminate image sensor, and the filter 2402 can be configured to allow the image sensor 2404 to receive only the relaxation wavelength of the fluorescent reagent and not the emission excitation wavelength of the reagent. The data determined by the image sensor 2404 may then indicate the presence of key body structures, tissues, biological processes, or chemical processes as determined by the location of the agent or dye.

[0295] Filter 2402 may also be used in implementations where fluorescent agents or dyes have not been administered. Filter 2402 may be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read by image sensor 2404. Image sensor 2404 may be a monochrome image sensor, such that pixels of a captured image that are above a threshold or below a threshold may be characterized as corresponding to certain spectral responses or fluorescent emissions. The spectral response or fluorescent emissions determined by the pixels captured by image sensor 2404 may indicate the presence of certain body tissues or structures, certain conditions, certain chemical processes, and the like.

[0296] Fig.25 2500 is a schematic diagram of an imaging system 2500 having multiple cut-off filters. System 2400 may be particularly suitable for implementation of fluorescent agent imaging. System 2500 includes an endoscope 2506 or other suitable imaging device having a light source 2508 for a light-deficient environment. Endoscope 2506 includes an image sensor 2504 and two filters 2502a, 2502b. It should be understood that in alternative embodiments, system 2500 may include any number of filters, and the number of filters and the type of filters may be selected for certain purposes, for example, for collecting imaging information of specific body tissues, body conditions, chemical processes, etc. Filters 2502a, 2502b are configured to prevent image sensor 2504 from sensing unwanted wavelengths of light or other electromagnetic radiation. Filters 2502a, 2502b may be configured to filter out unwanted wavelengths from white light or other electromagnetic radiation that may be emitted by light source 2508.

[0297] Relative to Fig.24 The present invention is also described, and filters 2502a, 2502b can be used in specific implementations where fluorescent reagents or dyes have been applied. Filters 2502a, 2502b can be configured to block the emission excitation wavelength of the reagent or dye, and allow the image sensor 2504 to read only the relaxation wavelength of the reagent or dye. In addition, filters 2502a, 2502b can be used in specific implementations where fluorescent reagents or dyes have not been applied. In such specific implementations, filters 2502a, 2502b can be selected to allow wavelengths corresponding to the desired spectral response to pass through the image sensor 2504 and be read by the image sensor.

[0298] Multiple filters 2502a, 2502b can each be configured to filter out different ranges of wavelengths of the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than a desired wavelength range, and an additional filter can be configured to filter out wavelengths shorter than a desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 2504.

[0299] In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 513nm and 545nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 565nm and 585nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 900nm and 1000nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 425nm and 475nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 520nm and 545nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 625nm and 645nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 760nm and 795nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 795nm and 815nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 370nm and 420nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are tailored to allow electromagnetic radiation between 600nm and 670nm to contact the image sensor 2504. In one embodiment, the filters 2502a, 2502b are configured to allow only certain fluorescent relaxation emissions to pass through the filters 2502a, 2502b and contact the image sensor 2504.

[0300] In one embodiment, system 2500 includes multiple image sensors 2504, and may specifically include two image sensors for generating three-dimensional images. Image sensors 2504 may be color / wavelength agnostic and configured to read electromagnetic radiation of any wavelength reflected from surface 2512. In one embodiment, image sensors 2504 are each color-dependent or wavelength-dependent and configured to read electromagnetic radiation of a specific wavelength reflected from surface 2512 and returned to image sensor 2504. Alternatively, image sensor 2504 may include a single image sensor having multiple different pixel sensors configured to read light of different wavelengths or colors, such as a Bayer filter color filter array. Alternatively, image sensor 2504 may include one or more color-agnostic image sensors that may be configured to read electromagnetic radiation of any wavelength reflected from surface 2512 according to a pulse schedule (such as Figures 5 to 7E and Figure 15 to Figure 16 Those shown in ) read electromagnetic radiation of different wavelengths.

[0301] Fig.26 26 is a schematic diagram showing a system 2600 for mapping a surface and / or tracking an object in an environment lacking light. In one embodiment, an endoscope 2602 pulses a grid array 2606 (which may be referred to as a laser mapping pattern) onto a surface 2604 in an environment lacking light. Fig.26 In one embodiment shown in FIG. 2 , the grid array 2606 includes vertical hashes 2608 and horizontal hashes 2610. It should be understood that the grid array 2606 may include any suitable array for mapping the surface 2604, including, for example, a raster grid of discrete points, an occupancy grid map, a dot array, etc. In addition, the endoscope 2602 may pulse multiple grid arrays 2606, and may pulse one or more separate grid arrays on each of multiple objects or structures, for example, in an environment lacking light.

[0302] In one embodiment, the system 2600 pulses a grid array 2606 that can be used to map the three-dimensional topology of a surface and / or track the position of an object such as a tool or another device in a light-deficient environment. In one embodiment, the system 2600 provides data to a third-party system or computer algorithm for determining surface size and configuration by way of light detection and ranging (LIDAR) mapping. The system 2600 can pulse any suitable wavelength of light or electromagnetic radiation in the grid array 2606, including, for example, ultraviolet light, visible light, light, and / or infrared or near infrared light. Surfaces 2604 and / or objects within an environment can be mapped and tracked at very high resolution and with very high accuracy and precision.

[0303] In one embodiment, system 2600 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. System 2600 may include a light engine having an emitter and an inner cavity, the emitter generating one or more electromagnetic radiation pulses, the inner cavity transmitting the one or more electromagnetic radiation pulses to a distal end of an endoscope in a light-deficient environment such as a body cavity. In one embodiment, at least a portion of the one or more electromagnetic radiation pulses includes a laser mapping pattern emitted onto a surface in the light-deficient environment (such as, a surface of body tissue in the body cavity and / or a surface of a tool or other device). Endoscope 2602 may include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations in the light-deficient environment.

[0304] In one embodiment, system 2600 includes a processor for determining the distance between an endoscope or tool and an object, such as surface 2604. The processor may also determine the angle between the endoscope or tool and the object. The processor may also determine surface area information about the object, including, for example, the size of the surgical tool, the size of the structure, the size of the anatomical structure, position information, and other position data and metrics. System 2600 may include one or more image sensors that provide image data output to a control system for determining the distance between an endoscope or tool and an object, such as surface 2604. The image sensor may output information to the control system for determining the angle between the endoscope or tool and the object. In addition, the image sensor may output information to the control system for determining surface area information about the object, the size of the surgical tool, the size of the structure, the size of the anatomical structure, position information, and other position data and metrics.

[0305] In one embodiment, the grid array 2606 is pulsed by the transmitter of the endoscope 2602 at a sufficient speed so that the grid array 2606 is not visible to the user. In various specific implementations, the grid array 2606 may be distracting to the user during the endoscopic imaging procedure and / or endoscopic surgery. The grid array 2606 may be pulsed at a sufficiently short period so that the grid array 2606 cannot be detected by the human eye. In an alternative embodiment, the endoscope 2602 pulses the grid array 2606 at a sufficient repetition frequency so that the grid array 2606 can be viewed by the user. In such an embodiment, the grid array 2606 may be superimposed on the image of the surface 2604 on the display. The grid array 2606 may be superimposed on the black and white or RGB image of the surface 2604 so that the grid array 2606 may be visible to the user during the use of the system 2600. The user of the system 2600 may indicate whether the grid array 2606 should be superimposed on the image of the surface 2604 and / or whether the grid array 2606 should be visible to the user. System 2600 may include a display that provides real-time measurement of the distance from endoscope 2602 to surface 2604 or another object in the light-deficient environment. The display may also provide real-time surface area information about surface 2604 and / or any object, structure, or tool in the light-deficient environment. The accuracy of the measurement may be accurate to less than one millimeter.

[0306] The endoscope 2602 can pulse electromagnetic radiation according to a pulse schedule (such as those shown herein), for example, which can also include pulsing the grid array 2606 and pulsing red, green, and blue light for generating an RGB image and also generating a grid array 2606 that can be overlaid on the RGB image and / or used to map and track surfaces 2604 and objects in an aphotic environment. The grid array 2606 can additionally be pulsed in conjunction with hyperspectral or fluorescence excitation wavelengths of electromagnetic radiation. Data from each of RGB imaging, laser scanning imaging, hyperspectral imaging, and fluorescence imaging can be combined to identify the location, size, and surface topology of key structures in the body.

[0307] In one embodiment, the endoscope 2602 includes one or more color-agnostic image sensors. In one embodiment, the endoscope 2602 includes two color-agnostic image sensors for generating a three-dimensional image or mapping of an aphotic environment. The image sensor can generate an RGB image of an aphotic environment according to a pulse schedule as disclosed herein. In addition, the image sensor can determine data for mapping an aphotic environment and tracking one or more objects within the aphotic environment based on data determined when pulsing the grid array 2606. In addition, the image sensor can determine spectral or hyperspectral data and fluorescence imaging data according to the pulse schedule, which can be modified by the user to adapt to the specific needs of the imaging procedure. In one embodiment, the pulse schedule includes red, green and blue pulses and pulses of the grid array 2606 and / or pulses for generating hyperspectral image data and / or fluorescence image data. In various specific implementations, the pulse schedule may include any suitable combination of electromagnetic radiation pulses according to user needs. The recurrence frequency of electromagnetic radiation of different wavelengths can be determined based on, for example, the energy of certain pulses, the needs of the user, whether certain data (e.g., hyperspectral data and / or fluorescence imaging data) need to be continuously updated or can be updated less frequently, etc.

[0308] The pulse schedule may be modified in any suitable manner, and certain electromagnetic radiation pulses may be repeated at any suitable frequency, as desired by a user or a computer-implemented program for certain imaging procedures. For example, in embodiments where surface tracking data generated based on grid array 2606 is provided to a computer-implemented program for use, for example, in robotic surgery, grid array 2606 may be pulsed more frequently than where surface tracking data is provided to a user visualizing a scene during an imaging procedure. In such embodiments where surface tracking data is used for robotic surgery, the surface tracking data may need to be updated more frequently or may need to be extremely accurate so that the computer-implemented program can perform the robotic surgery with precision and accuracy.

[0309] In one embodiment, the system 2600 is configured to generate an occupancy grid map including an array of cells divided into a grid. The system 2600 is configured to store height values ​​of each of the corresponding grid cells to determine a surface map of a three-dimensional environment in an illuminant environment.

[0310] Fig.27A and Fig.27BA perspective view and a side view, respectively, of an implementation of a monolithic sensor 2700 having multiple pixel arrays for producing a three-dimensional image in accordance with the teachings and principles of the present disclosure are shown. Such an implementation may be desirable for three-dimensional image capture, where the two pixel arrays 2702 and 2704 may be offset during use. In another implementation, the first pixel array 2702 and the second pixel array 2704 may be dedicated to receiving electromagnetic radiation of a predetermined wavelength range, where the first pixel array is dedicated to electromagnetic radiation of a different wavelength range than the second pixel array.

[0311] Fig.28A and Fig.28B A perspective view and a side view of a specific implementation of an imaging sensor 2800 built on multiple substrates are shown respectively. As shown, multiple pixel columns 2804 forming the pixel array are located on a first substrate 2802 and multiple circuit columns 2808 are located on a second substrate 2806. The figure also shows the electrical connection and communication between a pixel column and its associated or corresponding circuit column. In a specific implementation, the image sensor may have a pixel array separated from all or most of the supporting circuits, and it may be otherwise manufactured so that its pixel array and supporting circuits are on a single, monolithic substrate / chip. The present disclosure may use at least two substrates / chips, which will be stacked together using three-dimensional stacking technology. The first 2802 of the two substrates / chips may be processed using an image CMOS process. The first substrate / chip 2802 may consist of only a pixel array, or may consist of a pixel array surrounded by a limited circuit. The second or subsequent substrate / chip 2806 may be processed using any process, rather than necessarily from an image CMOS process. The second substrate / chip 2806 may be, but is not limited to, a high density digital process for integrating various and multiple functions into a very limited space or area on the substrate / chip, or a mixed mode or analog process for integrating, for example, precise analog functions, or an RF process for implementing wireless capabilities, or a MEMS (micro-electro-mechanical system) for integrating MEMS devices. The image CMOS substrate / chip 2802 may be stacked with the second or subsequent substrate / chip 2806 using any three-dimensional technology. The second substrate / chip 2806 may support a large portion or a large portion of the circuitry that would otherwise be implemented in the first image CMOS chip 2802 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increase the overall system area while keeping the pixel array size constant and optimized as much as possible. The electrical connection between the two substrates / chips may be accomplished through interconnects, which may be bonding wires, lugs, and / or TSVs (through silicon vias).

[0312] Fig.29A and Fig.29BA perspective view and a side view of a specific implementation of an imaging sensor 2900 having multiple pixel arrays for producing a three-dimensional image are shown, respectively. The three-dimensional image sensor can be constructed on multiple substrates and can include multiple pixel arrays and other associated circuits, wherein multiple pixel columns 2904a forming a first pixel array and multiple pixel columns 2904b forming a second pixel array are located on respective substrates 2902a and 2902b, respectively, and multiple circuit columns 2908a and 2908b are located on a separate substrate 2906. Electrical connections and communications between pixel columns and associated or corresponding circuit columns are also shown.

[0313] The multiple pixel arrays can sense information at the same time, and the information from the multiple pixel arrays can be combined to generate a three-dimensional image. In one embodiment, the endoscopic imaging system includes two or more pixel arrays that can be deployed to generate three-dimensional imaging. The endoscopic imaging system may include a transmitter for emitting electromagnetic radiation pulses during the blanking period of the pixel array. The pixel arrays can be synchronized so that for the two or more pixel arrays, optical black pixels are read at the same time (i.e., a blanking period occurs). The transmitter can emit electromagnetic radiation pulses for charging each of the two or more pixel arrays. The two or more pixel arrays can read their corresponding charged pixels at the same time, so that the readout cycles of the two or more pixel arrays occur simultaneously or approximately simultaneously. In one embodiment, the endoscopic imaging system includes a plurality of transmitters, each transmitter being individually synchronized with one or more pixel arrays in the plurality of pixel arrays. Information from the plurality of pixel arrays can be combined to generate a three-dimensional image frame and a video stream.

[0314] It should be understood that the teachings and principles of the present disclosure can be used for reusable device platforms, limited-use device platforms, reconfigurable device platforms, or single-use / disposable device platforms without departing from the scope of the present disclosure. It should be understood that in a reusable device platform, the end user is responsible for the cleaning and disinfection of the device. In a limited-use device platform, the device can be used a specified number of times before becoming inoperable. A typical new device has been sterilized before delivery and should be cleaned and disinfected by the end user before other uses if it is to be used for other purposes. In a reconfigurable device platform, a third party can reprocess the device (e.g., clean, package, and disinfect) a single-use device for additional use at a lower cost than a new unit. In a single-use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0315] Example

[0316] The following examples relate to preferred features of further embodiments:

[0317] Embodiment 1 is a system. The system includes an emitter and an image sensor, the emitter being used to emit electromagnetic radiation pulses, and the image sensor including a pixel array for sensing reflected electromagnetic radiation. The system includes a controller, the controller including a processor in electrical communication with the image sensor and the emitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor. The system is such that at least a portion of the electromagnetic radiation pulses emitted by the emitter include electromagnetic radiation red wavelengths, electromagnetic radiation green wavelengths, and electromagnetic radiation blue wavelengths, and also includes one or more of the following: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm; electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm; electromagnetic radiation excitation wavelengths that cause reagents to fluoresce; or a laser scanning pattern.

[0318] Embodiment 2 is a system according to embodiment 1, wherein the transmitter pulses the laser scanning pattern with a duration and frequency such that the laser scanning pattern is not visible to a user of the system.

[0319] Embodiment 3 is a system according to any one of embodiments 1 to 2, wherein the image sensor is configured to be capable of generating a plurality of exposure frames, wherein each exposure frame of the plurality of exposure frames corresponds to an electromagnetic radiation pulse emitted by the transmitter, and a data set corresponding in time to each electromagnetic radiation pulse is generated to generate a plurality of data sets corresponding to the plurality of exposure frames.

[0320] Embodiment 4 is the system of any one of embodiments 1 to 3, wherein the plurality of exposure frames and the plurality of data sets are combined to form an image frame.

[0321] Embodiment 5 is a system according to any one of embodiments 1 to 4, wherein the pixel array of the image sensor is a dual-sensitivity pixel array, and the dual-sensitivity pixel array includes a plurality of pixels sensitive to long exposure and a plurality of pixels sensitive to short exposure.

[0322] Embodiment 6 is the system of any one of embodiments 1 to 5, further comprising an optical filter that prevents the electromagnetic radiation excitation wavelength from being sensed by the image sensor.

[0323] Embodiment 7 is a system according to any one of embodiments 1 to 6, wherein the transmitter is configured to be capable of transmitting a plurality of electromagnetic radiation sub-pulses having sub-durations shorter than the pulse duration during a pulse duration.

[0324] Embodiment 8 is a system according to any one of embodiments 1 to 7, wherein one or more of the electromagnetic radiation pulses emitted by the transmitter includes electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

[0325] Embodiment 9 is a system according to any one of embodiments 1 to 8, wherein at least one of the electromagnetic radiation pulses emitted by the transmitter produces an exposure frame generated by the image sensor, and wherein the system also includes a display for displaying two or more exposure frames as a stacked image.

[0326] Embodiment 10 is a system according to any one of embodiments 1 to 9, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter is an excitation wavelength for causing the reagent to fluoresce, and wherein pulsing the excitation wavelength causes the image sensor to generate a fluorescence exposure frame indicating the position of the reagent within the scene.

[0327] Embodiment 11 is a system according to any one of embodiments 1 to 10, wherein the controller is further configured to provide the fluorescent exposure frame to a corresponding system, and the corresponding system determines the position of the key tissue structure based on the fluorescent exposure frame.

[0328] Embodiment 12 is a system according to any one of embodiments 1 to 11, wherein the controller is further configured to be capable of: receiving the position of the critical tissue structure from the corresponding system; generating an overlay frame including the position of the critical tissue structure within the scene; and combining the overlay frame with a color image frame depicting the scene to indicate the position of the reagent within the scene.

[0329] Embodiment 13 is a system according to any one of embodiments 1 to 12, wherein the image sensor is configured to sense reflected electromagnetic radiation generated by the laser scanning pattern to generate a topological exposure frame, and wherein the controller is further configured to: provide the topological exposure frame to a corresponding system, the corresponding system determines the topology of the scene and / or the size of one or more objects within the scene; provide the position of the key tissue structure to the corresponding system; and receive the topology and / or size of the key tissue structure from the corresponding system.

[0330] Embodiment 14 is a system according to any one of embodiments 1 to 13, wherein the critical tissue structure is one or more of a nerve, a ureter, a blood vessel, an artery, blood flow, cancerous tissue, or a tumor.

[0331] Embodiment 15 is a system according to any one of embodiments 1 to 14, wherein the blanking period of the image sensor corresponds to the time between the readout of the last row of the pixel array and the beginning of the next readout cycle of the pixel array.

[0332] Embodiment 16 is a system according to any one of embodiments 1 to 15, wherein the controller is further configured to be able to adjust the sequence of electromagnetic radiation pulses emitted by the emitter based on a threshold, wherein the threshold determines the appropriate illumination of a scene in a light-deficient environment.

[0333] Embodiment 17 is a system according to any one of embodiments 1 to 16, further comprising a display for displaying a video stream captured by the image sensor, wherein the video stream is allocated 8 bits or 16 bits or n bits of visible color for the display.

[0334] Embodiment 18 is a system according to any one of embodiments 1 to 17, wherein the transmitter is configured to repeatedly transmit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, each exposure frame corresponding to an electromagnetic radiation pulse.

[0335] Embodiment 19 is a system according to any one of embodiments 1 to 18, wherein the transmitter transmits the electromagnetic radiation pulse during the blanking period of the image sensor so that the pixel array is charged and ready to be read during the readout period of the image sensor.

[0336] Embodiment 20 is a system according to any one of embodiments 1 to 19, wherein the image sensor is configured to generate a topological exposure frame by sensing reflected electromagnetic radiation generated by the transmitter pulsing the laser scanning pattern, wherein the topological exposure frame includes information for determining real-time measurement, the information including one or more of the following: the distance from the endoscope to the object; the angle between the endoscope and the object; or surface topological information about the object.

[0337] Embodiment 21 is a system according to any one of embodiments 1 to 20, wherein the topological exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 centimeters.

[0338] Embodiment 22 is the system of any one of embodiments 1 to 21, wherein the topological exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

[0339] Embodiment 23 is a system according to any one of embodiments 1 to 22, further comprising a plurality of tools, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter comprises a tool-specific laser scanning pattern for each tool in the plurality of tools.

[0340] Embodiment 24 is a system according to any one of embodiments 1 to 23, wherein the laser scanning pattern emitted by the transmitter includes a first output and a second output that are independent of each other, wherein the first output is used for light illumination and the second output is used for tool tracking.

[0341] It should be appreciated that the various features disclosed herein provide significant advantages and advances in the art. The following claims are examples of some of those features.

[0342] In the above-mentioned detailed description of the present disclosure, for the purpose of simplifying the present disclosure, the various features of the present disclosure are concentrated in a single embodiment. The method of the present disclosure should not be understood as embodying such an intention: the disclosure protected by the claims requires more features than those explicitly listed in each claim. On the contrary, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0343] It should be understood that any features of the above-described arrangements, embodiments and implementations may be combined in a single embodiment comprising a combination of features obtained from any of the disclosed arrangements, embodiments and implementations.

[0344] It should be understood that the above configuration is only an exemplary application of the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, those skilled in the art may design many modifications and alternative configurations, and the appended claims are intended to cover these modifications and configurations.

[0345] Thus, while the present disclosure is shown in the drawings and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications may be made without departing from the principles and concepts described herein, including but not limited to changes in size, material, shape, form, function and mode of operation, assembly and use.

[0346] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0347] For the purpose of illustration and description, the above-mentioned specific embodiments have been provided. These specific embodiments are not intended to be exhaustive or to limit the present disclosure to the disclosed specific forms. According to the above-mentioned teachings, many modifications and changes can be made to the present disclosure. In addition, it should be noted that any or all of the aforementioned alternative specific implementations can be used in any desired combination to form other mixed specific implementations of the present disclosure.

[0348] In addition, although specific implementations of the present disclosure have been described and illustrated, the present disclosure is not limited to the specific forms or arrangements of parts as described and illustrated. The scope of the present disclosure will be defined by the claims appended hereto, any future claims filed here and in different applications, and their equivalents.

Claims

1. A system comprising: A transmitter for transmitting electromagnetic radiation pulses; an image sensor comprising an array of pixels for sensing reflected electromagnetic radiation; and a controller including a processor in electrical communication with the image sensor and the transmitter; wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor; and wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter include electromagnetic radiation red wavelengths, electromagnetic radiation green wavelengths, and electromagnetic radiation blue wavelengths, and further include one or more of the following to generate hyperspectral imaging data: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm; Electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; or, Electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm; The electromagnetic radiation pulse further comprises: an electromagnetic radiation excitation wavelength that causes the agent to fluoresce to generate fluorescence imaging data; and laser scanning the pattern to generate laser scanning imaging data; Therein, a combination of fluorescence imaging and / or hyperspectral imaging with laser scanning imaging is used to identify key structures in the body cavity.

2. The system according to claim 1, wherein: The transmitter pulses the laser scan pattern with a duration and frequency such that the laser scan pattern is not visible to a user of the system.

3. The system according to claim 1, wherein: The image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to an electromagnetic radiation pulse emitted by the emitter, and to generate a data set corresponding in time to each electromagnetic radiation pulse to generate a plurality of data sets corresponding to the plurality of exposure frames.

4. The system according to claim 3, wherein: The plurality of exposure frames are combined to form an image frame.

5. The system according to claim 1, wherein: The pixel array of the image sensor is a dual-sensitivity pixel array including a plurality of pixels sensitive to long exposure and a plurality of pixels sensitive to short exposure.

6. The system of claim 1, further comprising a filter that prevents the electromagnetic radiation excitation wavelength from being sensed by the image sensor.

7. The system according to claim 1, wherein: The transmitter is configured to be capable of transmitting, during a pulse duration, a plurality of electromagnetic radiation sub-pulses having sub-durations shorter than the pulse duration.

8. The system according to claim 1, wherein: One or more of the electromagnetic radiation pulses emitted by the emitter comprises electromagnetic radiation emitted simultaneously as a single pulse or a single sub-pulse at two or more wavelengths.

9. The system according to claim 1, wherein: At least one of the electromagnetic radiation pulses emitted by the emitter produces an exposure frame generated by the image sensor, wherein the system further comprises a display for displaying two or more exposure frames as a stacked image.

10. The system according to claim 1, wherein: At least a portion of the electromagnetic radiation pulses emitted by the emitter are of an excitation wavelength for causing an agent to fluoresce, and wherein pulsing the excitation wavelength causes the image sensor to generate a fluorescence exposure frame indicative of a location of the agent within a scene.

11. The system according to claim 10, wherein: The controller is further configured to provide the fluorescence exposure frame to a corresponding system, and the corresponding system determines the location of the key tissue structure based on the fluorescence exposure frame.

12. The system according to claim 11, wherein: The controller is also configured to: receiving the location of the key organizational structure from the corresponding system; generating a stacked frame including the location of the key tissue structure within the scene; as well as The overlay frame is combined with a color image frame depicting the scene to indicate the location of the agent within the scene.

13. The system according to claim 12, wherein: The image sensor is configured to sense reflected electromagnetic radiation generated by the laser scanning pattern to generate a topological exposure frame, and wherein the controller is further configured to: providing the topological exposure frame to a corresponding system, the corresponding system determining the topology of the scene and / or the size of one or more objects within the scene; providing the location of the key organizational structure to the corresponding system; as well as A topology and / or size of the key tissue structure is received from the corresponding system.

14. The system according to claim 13, wherein: The critical tissue structure is one or more of a nerve, a ureter, a blood vessel, an artery, blood flow, cancerous tissue, or a tumor.

15. The system of claim 1, wherein: The blanking period of the image sensor corresponds to the time between the readout of the last row of the pixel array and the start of the next readout cycle of the pixel array.

16. The system of claim 1, wherein: The controller is also configured to be capable of adjusting the sequence of electromagnetic radiation pulses emitted by the emitter based on a threshold value, wherein the threshold value determines appropriate illumination of a scene in a light-deficient environment.

17. The system of claim 1, further comprising a display for displaying a video stream captured by the image sensor, wherein the video stream is allocated 8 bits or 16 bits or n bits of visible color for the display.

18. The system of claim 1, wherein: The transmitter is configured to repeatedly transmit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, each exposure frame corresponding to a pulse of electromagnetic radiation.

19. The system of claim 1, wherein: The emitter emits the pulses of electromagnetic radiation during the blanking period of the image sensor such that the pixel array is charged and ready to be read during a readout period of the image sensor.

20. The system of claim 1, wherein: The image sensor is configured to generate a topological exposure frame by sensing reflected electromagnetic radiation generated by the transmitter pulsing the laser scanning pattern, wherein the topological exposure frame includes information for determining real-time measurements, the information including one or more of the following: distance from the endoscope to the subject; the angle between the endoscope and the object; or Surface topological information about the object.

21. The system of claim 20, wherein: The topological exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 centimeters.

22. The system of claim 20, wherein: The topological exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

23. The system of claim 1, further comprising a plurality of tools, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter comprises a tool-specific laser scan pattern for each tool in the plurality of tools.

24. The system of claim 1, wherein: The laser scanning pattern emitted by the emitter includes a first output and a second output that are independent of each other, wherein the first output is used for light illumination and the second output is used for tool tracking.

Images