Hyperspectral, fluorescence, and laser mapping imaging with fixed pattern noise removal

Abstract

Hyperspectral, fluorescence, and laser mapping imaging with reduced fixed pattern noise are disclosed. A method includes actuating an emitter to emit a plurality of electromagnetic radiation pulses, and sensing reflected electromagnetic radiation generated by the plurality of electromagnetic radiation pulses through a pixel array of an image sensor. The method includes reducing fixed pattern noise in the exposure frame by subtracting a reference frame from the exposure frame. The method causes at least a portion of the electromagnetic radiation pulses emitted by the emitter to include one or more of: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm, from about 565 nm to about 585 nm, from about 900 nm to about 1000 nm, an electromagnetic radiation excitation wavelength that causes a reagent to fluoresce, or a laser mapping mode.

Description

Technical Field

[0001] The present disclosure relates to digital imaging, and in particular to hyperspectral imaging, fluorescence imaging and / or laser mapping imaging in light-deficient environments. Background Art

[0002] Advances in technology have provided advances in medical imaging capabilities. Endoscopes can be used to view the inside of the body and examine the inside of an organ or cavity of the body. Endoscopes are used to investigate a patient's symptoms, confirm a diagnosis, or provide medical treatment. Medical endoscopes can be used to view a variety of body systems and parts, such as the gastrointestinal tract, respiratory tract, urinary tract, abdominal cavity, etc. Endoscopes can also be used for surgical procedures, such as plastic surgery, surgery on a joint or bone, surgery on the nervous system, surgery within the abdominal cavity, etc.

[0003] In some cases of endoscopic imaging, it may be advantageous or necessary to observe the color space. A digital color image includes at least three layers or "color channels" that accumulate to form an image with a range of tones. Each of the color channels measures the intensity and chromaticity of light in a spectral band. Typically, a digital color image includes color channels for a red spectral band, a green spectral band, and a blue spectral band (this may be referred to as a red-green-blue or RGB image). Each of the red, green, and blue color channels includes brightness information for a red, green, or blue spectral band. The brightness information of the individual red, green, and blue layers is combined to generate a color image. Because the color image is composed of individual layers, a conventional digital camera image sensor includes a color filter array that allows red, green, and blue visible wavelengths to hit the selected pixel sensor. Each individual pixel sensor element is sensitive to a red, green, or blue wavelength and will only return image data for that wavelength. Image data from the total array of pixel sensors is combined to generate an RGB image. At least three different types of pixel sensors take up a lot of physical space, so that the complete pixel array cannot be assembled in the smaller distal end of the endoscope.

[0004] Because conventional image sensors cannot be fitted in the distal end of an endoscope, the image sensor is traditionally located in the handpiece unit of the endoscope, which is held by the endoscope operator and is not placed in the body cavity. In such endoscopes, light is transmitted from the handpiece unit along the length of the endoscope to the distal end of the endoscope. This configuration has significant limitations. Endoscopes with this configuration are delicate and can be easily misaligned or damaged when they are bumped or impacted during regular use. This can significantly reduce the quality of the image and require frequent repair or replacement of the endoscope.

[0005] Conventional endoscopes with image sensors placed in the handpiece unit are further limited to capturing only color images. However, in some embodiments, in addition to color image data, it may be desirable to capture images using fluorescence data, hyperspectral data, and / or laser mapping data. Fluorescence imaging captures the emission of light by a substance that has absorbed electromagnetic radiation and "glows" as it emits relaxation wavelengths. Hyperspectral imaging can be used to identify different materials, biological processes, and chemical processes by emitting different partitions of electromagnetic radiation and evaluating the spectral response of the material. Laser mapping imaging can capture the surface shape of objects and landscapes and measure the distance between objects within the scene. Laser mapping imaging can also cover tool tracking, where the distance and / or size of tools within the scene can be tracked relative to each other, relative to the imaging device, and / or relative to structures within the scene. In some embodiments, it may be desirable to combine one or more of fluorescence imaging, hyperspectral imaging, and / or laser mapping imaging when imaging a scene.

[0006] However, the application of fluorescence imaging techniques, hyperspectral imaging techniques, and laser mapping techniques known in the art generally requires highly specialized equipment that may not be available for many applications. In addition, such techniques provide a limited view of the environment and must generally be used in conjunction with multiple individual systems and multiple individual image sensors that are made sensitive to specific bands of electromagnetic radiation. It is therefore desirable to develop an imaging system that can be used in a spatially confined environment to generate fluorescence imaging data, hyperspectral imaging data, and / or laser mapping imaging data.

[0007] In view of the above, this article describes systems, methods and devices for fluorescence imaging, hyperspectral imaging and laser mapping imaging in a light-deficient environment. Such systems, methods and devices can provide multiple data sets for identifying key structures in the body and providing accurate and valuable information about body cavities. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] A non-limiting and non-exhaustive embodiment of the present disclosure is described with reference to the following drawings, wherein like reference numerals refer to like parts in the various views unless otherwise specified. The advantages of the present disclosure will be better understood with reference to the following description and drawings, wherein:

[0009] Figure 1 is a schematic diagram of a system for digital imaging in a light-deficient environment having paired emitters and a pixel array;

[0010] Figure 2 It is a system used to provide illumination to light-deficient environments for endoscopic imaging;

[0011] Figure 2A is a schematic diagram of the complementary system hardware;

[0012] FIG. 3A to FIG. 3D is a diagrammatic representation of the operating cycle of the sensor for constructing an exposure frame;

[0013] Figure 4A is a graphical representation of the operation of one embodiment of an electromagnetic transmitter;

[0014] Figure 4B is a graphical representation of varying the duration and magnitude of emitted electromagnetic pulses to provide exposure control;

[0015] Figure 5 Yes FIG. 3A to FIG. 4B A graphical representation of one embodiment of the present disclosure of an operating cycle of a sensor, an electromagnetic transmitter and an electromagnetic pulse transmitted in combination, showing the imaging system during operation;

[0016] Fig. 6A is a schematic diagram of a process for recording a video with full spectrum light during the time period from t(0) to t(1);

[0017] Figure 6B is a schematic diagram of the process of recording a video by pulsing a partitioned spectral light in the time period from t(0) to t(1);

[0018] 7A to 7E A schematic diagram showing a process for recording video frames of both full spectrum light and partitioned spectrum light within a certain time interval;

[0019] Fig. 8A A conventional image sensor having optical black pixels disposed around a recording area of ​​active pixels is shown;

[0020] Figure 8B An embodiment of a minimum area image sensor is shown;

[0021] Fig. 9 is a readout sequence of an embodiment of an image sensor;

[0022] Fig.10 is a readout sequence for an implementation of a minimum area image sensor;

[0023] Fig.11 is a graphical representation of simple exponentially smoothed pixel shift estimates with small and large values ​​of w;

[0024] Fig.12 is a graphical representation of the modulation of a simple exponential smoothing indicator after a gain change;

[0025] Fig.13 is a schematic diagram of the process flow for eliminating fixed pattern noise;

[0026] Fig.14is a schematic diagram of a process flow to be implemented by a controller and / or a monochrome image signal processor for generating a video stream having an RGB image with application specific data superimposed thereon;

[0027] Fig.15 is a schematic diagram illustrating methods and hardware for use with a zoned light system;

[0028] Fig.16 is a schematic diagram of a process flow for performing corrections and adjustments on digital image data;

[0029] Fig.17 is a schematic diagram of a pattern reconstruction process for generating an RGB image having application specific data superimposed thereon by pulsed partitioned spectral light;

[0030] 18A to 18C A light source having a plurality of emitters is shown;

[0031] Fig.19 A single optical fiber is shown outputting at the output via a diffuser to illuminate a scene in a light-deficient environment;

[0032] Fig. 20 shows a portion of an electromagnetic spectrum divided into a plurality of different sub-spectra that may be emitted by an emitter of a light source in accordance with the principles and teachings of the present disclosure;

[0033] Fig.21 is a schematic diagram showing emission and readout timing for generating an image frame including a plurality of exposure frames produced by different partitions of pulsed light;

[0034] Fig. 22 An imaging system including a single cut-off filter for filtering wavelengths of electromagnetic radiation is shown;

[0035] Fig.23 An imaging system including a plurality of cut-off filters for filtering wavelengths of electromagnetic radiation is shown;

[0036] Fig.24 An exemplary laser mapping pattern that can be pulsed by the imaging system is shown;

[0037] Fig.25A and Fig.25B Specific implementations with multiple pixel arrays for generating three-dimensional images according to the principles and teachings of the present disclosure are shown;

[0038] Fig.26A and 26Bperspective and side views, respectively, of a specific implementation of an imaging sensor constructed on multiple substrates, wherein multiple columns of pixels forming a pixel array are located on a first substrate, and multiple columns of circuitry are located on a second substrate, and electrical connections and communications between a column of pixels and its associated or corresponding column of circuitry are shown; and

[0039] Fig.27A and Fig.27B A perspective view and a side view, respectively, of a specific implementation of an imaging sensor having multiple pixel arrays for producing a three-dimensional image are shown, wherein the multiple pixel arrays and the image sensor are built on multiple substrates. DETAILED DESCRIPTION

[0040] Disclosed herein are systems, methods and apparatus for digital imaging that may be primarily suitable for medical applications such as medical endoscopic imaging.An embodiment of the present disclosure is an endoscopic system for fluorescence imaging, hyperspectral imaging and / or laser mapping imaging in a light-absent environment.

[0041] An embodiment of the present disclosure is an endoscopic imaging system with a CMOS (complementary metal oxide semiconductor) image sensor. A CMOS image sensor has multiple sources of image noise. Image noise is random variations in brightness or color information and can be caused by electronic noise. Image noise is generally undesirable and can reduce the quality of an image. Image noise can range from barely perceptible specks on a digital image acquired under good lighting conditions to an optical image that is almost completely noisy, from which only a small amount of information can be derived by complex image processing. The amount of noise in an image generated by a CMOS image sensor depends on various physical conditions. Pure Poisson or Gaussian temporal noise without coherent components (such as photon scattering noise or source follower read noise) can look natural within a video stream. Other forms of perceptible noise will reduce image quality and may be unacceptable in some embodiments.

[0042] One form of noise that can cause problems with CMOS image sensors is spatial noise (SN). Spatial noise has at least two sources, including pixel fixed pattern noise (pixel FPN) and column fixed pattern noise (column FPN). Pixel fixed pattern noise is primarily due to variations in photodiode leakage current (which may be referred to as "dark signal") from pixel to pixel. This source may depend on junction temperature and exposure time. Column fixed pattern noise is a result of a readout architecture in which pixels from within the same column are directed through a common analog readout element. In many specific implementations, it is desirable to cancel image noise and specifically cancel fixed pattern noise. In light of the foregoing, systems, methods, and apparatus for imaging in a light-deficient environment and canceling fixed pattern noise in the resulting image are disclosed herein.

[0043] Conventional endoscopes are designed so that the image sensor is placed at the proximal end of the device within the handpiece unit. This configuration requires that the incident light travel the length of the endoscope through precision coupled optical elements. The precision optical elements are prone to misalignment during normal use, and this can result in image distortion or image loss. The embodiments of the present disclosure place the image sensor within the distal end of the endoscope itself. This provides greater optical simplicity compared to specific implementations known in the art. However, an acceptable solution to this approach is non-trivial and introduces its own set of engineering challenges, especially that the image sensor must fit within a highly constrained area. Disclosed herein are systems, methods, and devices for digital imaging in light-deficient environments that employ image sensors of minimal area and can be configured for hyperspectral imaging, fluorescence imaging, and / or laser mapping imaging, as well as color imaging.

[0044] In some cases, it is desirable to generate endoscopic imaging with multiple data types or multiple images superimposed on each other. For example, it may be desirable to generate a color ("RGB") image that also includes hyperspectral imaging data, fluorescence imaging data, and / or laser mapping imaging data superimposed on the RGB image. Superimposed images of this nature can enable a practitioner or computer program to identify the highly accurate dimensions and three-dimensional topology of key body structures, and further identify the distance between the tool and other structures in a light-deficient environment based on the laser mapping data. In the past, this would require the use of multiple sensor systems, including image sensors for color imaging and one or more additional image sensors for hyperspectral imaging, fluorescence imaging, or laser mapping imaging. In such systems, multiple image sensors will have multiple types of pixel sensors that are each sensitive to different ranges of electromagnetic radiation. In systems known in the art, this includes three separate types of pixel sensors for generating RGB color images, as well as additional sensors and systems for generating hyperspectral data, fluorescence data, and laser mapping data. These multiple different sensors take up too much physical space and cannot be located at the distal end of the endoscope. In systems known in the art, one or more cameras are not placed at the distal end of the endoscope, but are placed in the endoscope handpiece or robotic unit. This causes many disadvantages and renders the endoscope very fragile. When the fragile endoscope is hit or impacted during use, it may damage the endoscope and reduce the image quality. In view of the above, systems, methods and devices for endoscopic imaging in a light-deficient environment are disclosed herein. The systems, methods and devices disclosed herein provide a way to employ multiple imaging techniques in a single imaging session while allowing one or more image sensors to be disposed in the distal end of the endoscope.

[0045] The fluorescence imaging techniques discussed herein can be used in combination with one or more fluorescent agents or fluorescent dyes. The location of the agent can be identified by emitting an electromagnetic radiation excitation wavelength that causes the agent to fluoresce. The relaxation wavelength emitted by the agent can be read by an image sensor to identify the location of the agent within the scene. Depending on the type of agent used, the location of the agent can further indicate the location of key structures such as certain types of tissues, cancer cells and non-cancerous cells, etc.

[0046] The hyperspectral imaging techniques discussed herein can be used to "see through" the tissue layers in the foreground of a scene to identify specific types of tissue and / or specific biological or chemical processes. Hyperspectral imaging can be used in medical settings to quantitatively track the course of a disease and determine tissue pathology. Additionally, hyperspectral imaging can be used to identify key structures such as neural tissue, muscle tissue, cancer cells, etc. In one embodiment, partitions of electromagnetic radiation are pulsed, and data about the spectral responses of different types of tissues are collected in response to the partitions of electromagnetic radiation. A data store of spectral responses can be generated and analyzed to evaluate a scene and predict which tissues are present within the scene based on the sensed spectral responses.

[0047] The laser mapping imaging techniques discussed herein can be evaluated to generate a three-dimensional landscape map of a scene and calculate distances between objects within the scene. The laser mapping data can be used in conjunction with fluorescence imaging and / or hyperspectral imaging to calculate the precise location and size of key structures. For example, fluorescence imaging and / or hyperspectral imaging can be used to identify the location and boundaries of key structures. The location of key structures, the size of key structures, and precise measurements of distances from key structures to other objects can then be calculated based on the laser mapping data.

[0048] Hyperspectral imaging

[0049] In one embodiment, the systems, methods, and devices disclosed herein provide a means for generating hyperspectral imaging data in a light-deficient environment. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from conventional cameras that only capture light across three wavelengths (including red, green, and blue wavelengths) based on the visible spectrum that can be distinguished by the human eye to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands.

[0050] Hyperspectral imaging was originally developed for applications in mining and geology. Unlike normal camera images, which provide limited information to the human eye, hyperspectral imaging can identify specific minerals based on the spectral signatures of different minerals. Hyperspectral imaging is useful even when captured in aerial images and can provide information about, for example, oil or gas leaks from pipelines or natural wells and their impact on nearby vegetation. This information is gathered based on the spectral signatures of certain materials, objects, or processes that can be identified through hyperspectral imaging.

[0051] Hyperspectral imaging includes spectroscopy and digital photography. In one embodiment of hyperspectral imaging, a complete spectrum or some spectral information is collected at each pixel in the image plane. The goal of hyperspectral imaging can vary for different applications. In one application, the goal of hyperspectral imaging is to obtain the entire electromagnetic spectrum for each pixel in the image scene. This can make it possible to find certain objects that might not be recognizable in the visible light wavelength band. This can make it possible to accurately identify certain substances or tissues, although these substances or tissues may not be recognizable in the visible light wavelength band. In addition, this can make it possible to detect certain processes by capturing images across all wavelengths of the electromagnetic spectrum.

[0052] In one embodiment of the present disclosure, the endoscope system illuminates the source and pulses electromagnetic radiation for spectral or hyperspectral imaging. Spectral imaging uses multiple bands across the electromagnetic spectrum. This is different from conventional cameras that only capture light across three wavelengths (including red, green, and blue wavelengths) based on the visible spectrum that can be distinguished by the human eye to generate RGB images. Spectral imaging can use any wavelength band in the electromagnetic spectrum, including infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can stack imaging generated based on invisible bands (e.g., infrared) on top of imaging based on visible bands (e.g., standard RGB images) to provide additional information that can be easily distinguished by humans or computer algorithms.

[0053] Hyperspectral imaging has many advantages over conventional imaging. The information obtained by hyperspectral imaging enables practitioners and / or computer-implemented programs to accurately identify certain tissues or conditions that may not be identifiable using RGB imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, enabling practitioners to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells that contrast with typical healthy cells, identify certain tissues or conditions, identify critical structures, etc. Hyperspectral imaging provides specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0054] In medical applications, hyperspectral imaging can provide specific advantages over conventional imaging. The information obtained by hyperspectral imaging can enable practitioners and / or computer-implemented programs to accurately identify certain tissues or conditions that may not be diagnosed or may be diagnosed less accurately when using conventional imaging such as RGB imaging. In addition, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, which can enable practitioners to, for example, view tissues located behind certain tissues or fluids, identify atypical cancer cells that contrast with typical healthy cells, identify certain tissues or conditions, identify key structures, etc. Hyperspectral imaging can provide specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated with conventional imaging.

[0055] In various applications and implementations of the present disclosure, endoscopic hyperspectral imaging can present advantages over conventional imaging. In medical implementations, endoscopic hyperspectral imaging can allow a practitioner or a computer-implemented program to distinguish, for example, neural tissue, muscle tissue, various blood vessels, blood flow direction, etc. Hyperspectral imaging can enable atypical cancer tissue to be accurately distinguished from typical healthy tissue, and thus can enable a practitioner or a computer-implemented program to distinguish the boundaries of cancerous tumors during surgery or research imaging. In addition, hyperspectral imaging in a light-deficient environment as disclosed herein can be combined with the use of reagents or dyes to allow further distinction of certain tissues or substances. In such embodiments, the reagent or dye can fluoresce at a specific wavelength band in the electromagnetic spectrum, thereby providing information specific to the purpose of the reagent or dye. The systems, methods, and devices disclosed herein can enable pulsing of any number of wavelength bands, so that one or more reagents or dyes can fluoresce at different times, and further enable one or more partitions of pulsable electromagnetic radiation for hyperspectral imaging in the same imaging session. In certain implementations, this enables identification or study of multiple medical conditions during a single imaging procedure.

[0056] Fluorescence imaging

[0057] The systems, methods and devices disclosed herein provide a means for generating fluorescence imaging data in an environment lacking light. Fluorescence imaging data can be used to identify certain materials, tissues, components or processes within a body cavity or other environment lacking light. In certain embodiments, fluorescence imaging can be provided to a medical practitioner or a computer-implemented program to enable identification of certain structures or tissues in the body. Such fluorescence imaging data can be layered on a black and white or RGB image to provide additional information and context.

[0058] Fluorescence is the emission of light by a substance that has absorbed light or other electromagnetic radiation. Some fluorescent substances can "glow" or emit different colors visible to the human eye when subjected to ultraviolet light or other wavelengths of electromagnetic radiation. When the radiation source stops, some fluorescent materials will stop emitting light almost immediately.

[0059] Fluorescence occurs when the orbital electrons of a molecule, atom or nanostructure are excited by light or other electromagnetic radiation and then relax to their ground state by emitting photons from an excited state. The specific frequency of the electromagnetic radiation that excites the orbital electrons or is emitted by the photons during relaxation depends on the specific atom, molecule or nanostructure. In most cases, the light emitted by the substance has a longer wavelength than the radiation absorbed by the substance, and therefore has lower energy. However, when the absorbed electromagnetic radiation is intense, one electron can absorb two photons. This two-photon absorption can result in the emission of radiation with a shorter wavelength than the absorbed radiation and therefore with higher energy. In addition, the emitted radiation can also have the same wavelength as the absorbed radiation.

[0060] Fluorescence imaging has many practical applications, including mineralogy, geology, medicine, spectroscopy of chemical sensors, detection of biological processes or signals, etc. Fluorescence can be used in biochemistry and medicine as a non-destructive means for tracking or analyzing biomolecules. Biomolecules, including certain tissues or structures, can be tracked by analyzing the fluorescence emission of biomolecules after being excited by electromagnetic radiation of certain wavelengths. However, relatively few cellular components are naturally fluorescent. In some specific implementations, it may be advantageous to visualize certain tissues, structures, chemical processes or biological processes that are not fluorescent in nature. In such specific implementations, a dye or reagent can be applied to the body, and the dye or reagent may include molecules, proteins or quantum dots with fluorescent properties. Then, the reagent or dye may fluoresce after being excited by electromagnetic radiation of certain wavelengths. Different reagents or dyes may include different molecules, proteins and / or quantum dots that will fluoresce under electromagnetic radiation of a specific wavelength. Therefore, it may be necessary to excite the reagent or dye with electromagnetic radiation of a specific frequency band to achieve fluorescence and identify the desired tissue, structure or process in vivo.

[0061] Fluorescence imaging can provide valuable information in the medical field that can be used for diagnostic purposes and / or can be visualized in real time during medical procedures. Specific reagents or dyes can be applied to the body to make certain tissues, structures, chemical processes or biological processes fluoresce. The fluorescence of the reagent or dye can highlight body structures, such as blood vessels, nerves, specific organs, etc. In addition, the fluorescence of the reagent or dye can highlight conditions or diseases, such as cancer cells or cells that undergo certain biological or chemical processes that may be associated with the condition or disease. Fluorescence imaging can be used in real time by a practicing physician or a computer program to distinguish, for example, cancer cells from non-cancerous cells during surgical tumor extraction. Fluorescence imaging can also be used as a non-destructive means for tracking and visualizing conditions in the body that were originally invisible to the human eye or indistinguishable in RGB images over time.

[0062] Systems, methods and devices for generating fluorescence imaging data can be used in conjunction with reagents or dyes. Some reagents or dyes are known to attach to certain types of tissues and fluoresce at specific wavelengths of the electromagnetic spectrum. In one specific implementation, a reagent or dye configured to fluoresce when activated by light of certain wavelengths is administered to a patient. The reagent or dye is excited and fluoresced using the endoscopic imaging system disclosed herein. The fluorescence of the reagent or dye is captured by the endoscopic imaging system to help identify tissues or structures in a body cavity. In a specific implementation, a plurality of reagents or dyes are administered to a patient, each of which is configured to emit fluorescence of different wavelengths and / or provide indications of different structures, tissues, chemical reactions, biological processes, etc. In such specific implementations, the endoscopic imaging system emits each applicable wavelength in the applicable wavelengths to cause each of the applicable reagents or dyes to fluoresce. This can eliminate the need to perform a single imaging procedure for each of the multiple reagents or dyes.

[0063] Imaging agents can enhance imaging capabilities in the pharmaceutical, medical, biotechnology, diagnostic and medical procedure industries. Many imaging techniques such as X-ray, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI) and nuclear medicine mainly analyze anatomical structures and morphology, and cannot detect changes at the molecular level. Fluorescent agents, dyes and probes (including quantum dot nanoparticles and fluorescent proteins) assist medical imaging techniques by providing additional information about certain tissues, structures, chemical processes and / or biological processes present in the imaging area. Imaging using fluorescent agents can achieve cell tracking and / or tracking of certain molecular biomarkers. Fluorescent agents can be used to image cancer, infection, inflammation, stem cell biology, etc. Many fluorescent agents and dyes are being developed and applied to visualize and track biological processes in a non-destructive manner. Such fluorescent agents can be excited by electromagnetic radiation of certain wavelengths or wavelength bands. Similarly, these fluorescent agents can emit relaxation energy of certain wavelengths or wavelength bands when fluorescing, and the relaxation energy emitted can be read by a sensor to determine the position and / or boundary of the agent or dye.

[0064] In one embodiment of the present disclosure, an endoscopic imaging system pulses electromagnetic radiation to excite electrons in a fluorescent agent or dye. The endoscopic imaging system can pulse electromagnetic radiation of multiple different wavelengths during a single imaging session to cause multiple different agents or dyes to fluoresce. The endoscope includes an image sensor that is sensitive to the relaxation wavelength of one or more agents or dyes. The imaging data generated by the image sensor can be used to identify the location and boundaries of the one or more agents or dyes. The endoscopic system can also pulse electromagnetic radiation in the red band, green band, and blue band of visible light so that fluorescent imaging can be superimposed on an RGB video stream.

[0065] Laser mapping imaging

[0066] In one embodiment, the systems, methods, and devices disclosed herein provide a means for generating laser mapping data using an endoscopic imaging system. The laser mapping data can be used to determine precise measurements and topological contours of a scene. In a specific implementation, the laser mapping data is used to determine, for example, precise measurements between structures or organs in a body cavity, devices or tools in a body cavity, and / or key structures in a body cavity. As described herein, the term "laser mapping" may cover techniques referred to as laser mapping, laser scanning, topological scanning, three-dimensional scanning, laser tracking, tool tracking, and the like. As discussed herein, a laser mapping exposure frame may include topological data of a scene, dimensions between objects or structures within a scene, dimensions or distances of tools or objects within a scene, and the like.

[0067] Laser mapping typically involves controlled deflection of a laser beam. In the field of three-dimensional object scanning, laser mapping combines controlled steering of a laser beam with a laser rangefinder. By measuring distance in each direction, a laser rangefinder can quickly capture the surface shape of objects, tools, and landscapes. The construction of a full three-dimensional topology can include combining multiple surface models obtained from different perspectives. Various measurement systems and methods exist in the art for applications in archaeology, geography, atmospheric physics, autonomous vehicles, and the like. One such system includes light detection and ranging (LIDAR), which is a three-dimensional laser mapping system. LIDAR has been used in navigation systems such as aircraft or satellites to determine the position and orientation of sensors combined with other systems and sensors. LIDAR uses an active sensor to illuminate an object and detects energy reflected from the object and returned to the sensor.

[0068] As used herein, the term "laser mapping" includes laser tracking. Laser tracking, or the use of lasers for tool tracking, measures objects by determining the position of an optical target held relative to those objects. Laser trackers can be accurate to the order of 0.025 mm over a distance of several meters. In one embodiment, an endoscopic imaging system pulses light for use in conjunction with a laser tracking system so that positions or tools within a scene can be tracked and measured. In such embodiments, the endoscopic imaging system can pulse a laser tracking pattern on a tool, object, or other structure within a scene imaged by the endoscopic imaging system. A target can be placed on a tool, object, or other structure within the scene. Measurements between the endoscopic imaging system and the target can be triggered and acquired at selected points so that the endoscopic imaging system can track the position of the target (and the tool, object, or other structure attached to the target).

[0069] Pulse imaging

[0070] Some specific implementations of the present disclosure include various aspects of sensor and system combination designs that are capable of generating high-definition images with reduced pixel counts in restricted lighting environments. This is achieved by pulsing a monochromatic wavelength frame by frame and switching or alternating between a single different color wavelength each frame using a controlled light source in conjunction with a high frame capture rate and a specially designed corresponding monochromatic sensor. The pixels can be color amorphous so that each pixel generates data for each electromagnetic radiation pulse that includes pulses of red, green, and blue visible light wavelengths as well as other wavelengths for laser mapping imaging.

[0071] For the purpose of promoting understanding of the principles according to the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe these embodiments. However, it should be understood that it is not intended to limit the scope of the present disclosure. Any changes and further modifications to the inventive features shown herein, and any additional applications of the principles of the present disclosure as shown herein (which will generally occur for those skilled in the relevant art and those who understand the present disclosure) will be considered to be within the scope of the disclosure protected by the claims.

[0072] Before disclosing and describing structures, systems and methods for generating images in a light-deficient environment, it should be understood that the present disclosure is not limited to the specific structures, configurations, process steps and materials disclosed herein, because such structures, configurations, process steps and materials may vary to a certain extent. It should also be understood that the terminology used herein is only for the purpose of describing specific embodiments and is not intended to be limiting, because the scope of the present invention will be limited only by the appended claims and their equivalents.

[0073] In describing and claiming the presently disclosed subject matter, the following terminology will be used in accordance with the following definitions.

[0074] It must be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0075] As used herein, the terms "comprising," "including," "characterized by," and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unrecited elements or method steps.

[0076] As used herein, the phrase "consisting of and its grammatical equivalents exclude any elements or steps not listed in a claim.

[0077] As used herein, the phrase "consisting essentially of and its grammatical equivalents limit the scope of a claim to the specified materials or steps and those materials or steps that do not materially affect one or more of the basic and novel characteristics of the claimed disclosure.

[0078] As used herein, the term "proximal side" broadly refers to the concept of a portion close to a starting point.

[0079] As used herein, the term "distal" generally refers to the opposite of the proximal side, and thus refers to the concept of a portion farther from a starting point or a portion farthest from a starting point according to the context.

[0080] As used herein, color sensors or multispectral sensors are those sensors known to have a color filter array (CFA) thereon to filter incident electromagnetic radiation into its individual components. In the visible range of the electromagnetic spectrum, such CFAs may be based on a Bayer template or a modified form thereof to separate the green, red and blue spectral components of light.

[0081] As used herein, a monochrome sensor refers to an imaging sensor without filtering. Because the pixels are color amorphous, their effective spatial resolution is significantly higher than their pixel color (usually filtered with a Bayer pattern) counterparts in traditional single-sensor cameras. Monochrome sensors can also have higher quantum efficiency because fewer incident photons are wasted between individual pixels.

[0082] As used herein, a transmitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of the transmitter can be configured to emit pulses and have very specific frequencies or frequency ranges from within the entire electromagnetic spectrum. The pulses may include wavelengths from the visible range and the invisible range. The transmitter can be cycled on and off to generate pulses, or a shutter mechanism can be used to generate pulses. The transmitter can have a variable power output level, or can be controlled using auxiliary devices such as apertures or filters. The transmitter can emit wide spectrum or full spectrum electromagnetic radiation that can be pulsed by color filtering or shutter opening and closing. The transmitter may include multiple electromagnetic sources that act alone or in concert.

[0083] It should be noted that, as used herein, the term "light" is both a particle and a wavelength, and is intended to represent electromagnetic radiation that can be detected by the pixel array 122, and may include wavelengths of electromagnetic radiation from the visible spectrum and the invisible spectrum. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is less than the entire spectrum, or in other words, the wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that is controllable with respect to a portion of the electromagnetic spectrum emitted, or a light source that can manipulate the physical properties of its components, the emission intensity, or the emission duration, or all of the above. An emitter can emit light in any dithered, diffuse, or collimated emission, and can be controlled digitally or by an analog method or system. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and it includes light sources such as lasers, LEDs, incandescent lights, or any light source that can be digitally controlled.

[0084] Now referring to the accompanying drawings, Figure 1A schematic diagram of a system 100 for sequential pulse imaging in an environment lacking light is shown. The system 100 can be deployed to generate an RGB image with application specific data layered thereon. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a partition of electromagnetic radiation in an environment lacking light 112, and the pixel array 122 senses an instance of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 operate in sequence so that one or more pulses of a partition of electromagnetic radiation produce image data sensed by the pixel array 122.

[0085] It should be noted that, as used herein, the term "light" is both a particle and a wavelength, and is intended to represent electromagnetic radiation that can be detected by the pixel array 122, and may include wavelengths of electromagnetic radiation from the visible spectrum and the invisible spectrum. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is less than the entire spectrum, or in other words, the wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a light source that is controllable with respect to a portion of the electromagnetic spectrum emitted, or a light source that can manipulate the physical properties of its components, the emission intensity, or the emission duration, or all of the above. An emitter can emit light in any dithered, diffuse, or collimated emission, and can be controlled digitally or by an analog method or system. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and it includes light sources such as lasers, LEDs, incandescent lights, or any light source that can be digitally controlled.

[0086] The pixel array 122 of the image sensor may be electronically paired with the emitter 102 such that the emitter 102 and the pixel array 122 are synchronized during operation for both receiving emissions and for adjustments made within the system. The emitter 102 may be tuned to emit electromagnetic radiation in the form of a laser, which may be pulsed to illuminate the light-deficient environment 112. The emitter 102 may pulse at intervals corresponding to the operation and functionality of the pixel array 122. The emitter 102 may pulse light in multiple electromagnetic partitions such that the pixel array receives electromagnetic energy and generates a data set corresponding in time to each specific electromagnetic partition. For example, Figure 1One implementation is shown in which the emitter 102 emits four different partitions of electromagnetic radiation, including red 104 wavelengths, green 106 wavelengths, blue 108 wavelengths, and a dedicated 110 emission. The dedicated 110 emission may include an excitation wavelength for causing an agent to fluoresce, a hyperspectral partition of the electromagnetic radiation, and / or a laser mapping mode. The dedicated 110 emission may include multiple individual emissions that are separate and independent of each other. The dedicated 110 emission may include a combination of an excitation wavelength for causing an agent to fluoresce and a laser mapping mode, wherein the emissions are separate and independent of each other. The data generated by the individual emissions may be analyzed in tandem to identify key structures within the scene based on the fluorescence imaging data, and also to identify the size or location of the key structures based on the combination of the laser mapping data and the fluorescence imaging data. The dedicated 110 emission may include a combination of electromagnetic radiation and a laser mapping mode in a hyperspectral band, wherein the emissions are separate and independent of each other. The data generated by the individual emissions may be analyzed in tandem to identify key structures within the scene based on the hyperspectral imaging data, and also to identify the size or location of the key structures based on the combination of the laser mapping data and the hyperspectral imaging data. In one embodiment, the dedicated 110 transmissions include any desired combination of transmissions that can be combined with the data obtained from the pulsed red 104 transmission, the pulsed green 106 transmission, and the pulsed blue 108 transmission. The dedicated 110 transmissions can be dispersed within the pulse pattern so that pulses of different types of dedicated 110 transmissions are less frequent than pulses of pulsed red 104 transmission, pulsed green 106 transmission, and pulsed blue 108 transmission.

[0087] Light-starved environment 112 includes structures, tissues, and other elements that reflect a combination of red 114 light, green 116 light, and / or blue 118 light. Structures perceived as red 114 will reflect pulsed red 104 light. Reflection from red structures causes pixel array 122 to sense red 105 after pulsed red 104 emission. Data sensed by pixel array 122 generates a red exposure frame. Structures perceived as green 116 will reflect pulsed green 106 light. Reflection from green structures causes pixel array 122 to sense green 107 after pulsed green 106 emission. Data sensed by pixel array 122 generates a green exposure frame. Structures perceived as blue 118 will reflect pulsed blue 108 light. Reflection from blue structures causes pixel array 122 to sense blue 109 after pulsed blue 108 emission. Data sensed by pixel array 122 generates a blue exposure frame.

[0088] When the structure is a combination of colors, the structure will reflect a combination of pulsed red 104 emission, pulsed green 106 emission, and / or pulsed blue 108 emission. For example, a structure perceived as purple will reflect light from both pulsed red 104 emission and pulsed blue 108 emission. The resulting data sensed by pixel array 122 will indicate that light is reflected in the same area after the pulsed red 104 emission and the pulsed blue 108 emission. When the resulting red exposure frame and blue exposure frame are combined to form an RGB image frame, the RGB image frame will indicate that the structure is purple.

[0089] In embodiments where the light-deficient environment 112 includes a fluorescent agent or fluorescent dye or includes one or more fluorescent structures, tissues, or other elements, the pulse scheme may include the emission of certain fluorescence excitation wavelengths. Certain fluorescence excitation wavelengths may be selected to cause known fluorescent agents, fluorescent dyes, or other structures to fluoresce. The fluorescent structure will be sensitive to the fluorescence excitation wavelength and will emit a fluorescence relaxation wavelength. After emitting the fluorescence excitation wavelength, the fluorescence relaxation wavelength will be sensed by the pixel array 122. The data sensed by the pixel array 122 generates a fluorescence exposure frame. The fluorescence exposure frame may be combined with a plurality of other exposure frames to form an image frame. The data in the fluorescence exposure frame may be stacked on an RGB image frame including data from a red exposure frame, a green exposure frame, and a blue exposure frame.

[0090] In embodiments where the aphotic environment 112 includes structures, tissues, or other materials that emit spectral responses to certain partitions of the electromagnetic spectrum, the pulse scheme may include emission of hyperspectral partitions of electromagnetic radiation for inducing spectral responses from structures, tissues, or other materials present in the aphotic environment 112. The spectral response includes emission or reflection of electromagnetic radiation of certain wavelengths. The spectral response may be sensed by the pixel array 122 and a hyperspectral exposure frame may be generated. The hyperspectral exposure frame may be combined with a plurality of other exposure frames to form an image frame. The data in the hyperspectral exposure frame may be overlaid on an RGB image frame that includes data from a red exposure frame, a green exposure frame, and a blue exposure frame.

[0091] In one embodiment, the pulse scheme includes the emission of a laser marking mode or a tool tracking mode. After the emission of the laser marking mode or the tool tracking mode, the reflected electromagnetic radiation sensed by the pixel array 122 generates a laser marking exposure frame. The data in the laser marking exposure frame can be provided to a corresponding system to identify, for example, the distance between tools present in the light-poor environment 112, the three-dimensional surface topology of the scene in the light-poor environment 112, the distance, size or position of structures or objects within the scene, etc. The data can be superimposed on the RGB image frame or provided to the user of the system in other ways.

[0092] The emitter 102 may be a laser emitter capable of emitting pulsed red 104 light for generating sensed red 105 data to identify red 114 elements within the light-starved environment 112. The emitter 102 may also be capable of emitting pulsed green 106 light for generating sensed green 107 data to identify green 116 elements within the light-starved environment. The emitter 102 may also be capable of emitting pulsed blue 108 light for generating sensed blue 109 data to identify blue 118 elements within the light-starved environment. The emitter 102 may also be capable of emitting dedicated 110 emissions for mapping the topology 120 of a scene within the light-starved environment 112. The emitter 102 may be capable of emitting pulsed red 104 emissions, pulsed green 106 emissions, pulsed blue 108 emissions, and pulsed dedicated 110 emissions in any desired order.

[0093] The pixel array 122 senses the reflected electromagnetic radiation. Each of the sensed red 105 data, the sensed green 107 data, the sensed blue 109 data, and the sensed dedicated 111 data may be referred to as an "exposure frame". The sensed dedicated 111 may produce multiple individual exposure frames that are separate and independent of each other. For example, the sensed dedicated 111 may produce a fluorescent exposure frame, a hyperspectral exposure frame, and / or a laser mapping exposure frame that includes laser mapping data. Each exposure frame is assigned a specific color partition or wavelength partition, where the assignment is based on the timing of the pulse color or wavelength partition from the emitter 102. The combination of the exposure frame and the assigned specific color or wavelength partition may be referred to as a data set. Even if the pixels 122 are not color-specific, a color may be assigned to any given data set based on a priori information about the emitter.

[0094] For example, during operation, after the pulsed red 104 light is pulsed in the light-starved environment 112, the pixel array 122 senses the reflected electromagnetic radiation. The reflected electromagnetic radiation produces an exposure frame, and the exposure frame is classified as sensed red 105 data because it corresponds in time to the pulsed red 104 light. The combination of the exposure frame and its indication that it corresponds in time to the pulsed red 104 light is a "data set". This process is repeated for each partition of the electromagnetic radiation emitted by the emitter 102. The data formed by the pixel array 122 includes a sensed red 105 exposure frame, which is used to identify the red 114 component in the light-starved environment and corresponds in time to the pulsed red 104 light. The data also includes a sensed green 107 exposure frame, which is used to identify the green 116 component in the light-starved environment and corresponds in time to the pulsed green 106 light. The data also includes a sensed blue 109 exposure frame, which is used to identify the blue 118 component in the light-starved environment and corresponds in time to the pulsed blue 108 light. The data also includes sensed dedicated 111 exposure frames that are used to identify the topology 120 and correspond in time to the dedicated 110 transmissions.

[0095] In one embodiment, three data sets representing RED electromagnetic pulses, GREEN electromagnetic pulses, and BLUE electromagnetic pulses are combined to form a single image frame. Thus, the information in the red exposure frame, the green exposure frame, and the blue exposure frame are combined to form a single RGB image frame. One or more additional data sets representing other wavelength partitions may be superimposed on the single RGB image frame. One or more additional data sets may represent, for example, laser mapping data, fluorescence imaging data, and / or hyperspectral imaging data.

[0096] It should be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition without departing from the scope of the present disclosure, and any color combination or any electromagnetic partition can be used to replace RED, GREEN and BLUE, such as cyan, magenta and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible wavelengths and invisible wavelengths. In the figure, the light-deficient environment 112 to be imaged includes a red 114 portion, a green 116 portion and a blue 118 portion, and also includes a topology 120 that can be sensed and mapped into a three-dimensional rendering. As shown, the reflected light from the electromagnetic pulse only contains data for the portion of the object having a specific color corresponding to the pulse color partition. These separate color (or color interval) data sets can then be used to reconstruct an image by combining the data sets at 126. The information in each of the multiple exposure frames (i.e., multiple data sets) can be combined by a controller 124, a control unit, a camera control unit, an image sensor, an image signal processing pipeline, or some other computing resources that can be configured to process multiple exposure frames and combine data sets at 126. The data sets may be combined to generate a single image frame within the endoscopy unit itself or off-site by some other processing resource.

[0097] Figure 2The invention discloses a system 200 for providing illumination to a light-deficient environment, such as for endoscopic imaging. The system 200 may be used in combination with any of the systems, methods, or devices disclosed herein. The system 200 includes a transmitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an inner cavity waveguide 210, an inner cavity 212, and an image sensor 214 with accompanying optical components (such as a lens). The transmitter 202 (generally referred to as a "light source") generates light that travels through the jumper waveguide 206 and the inner cavity waveguide 210 to illuminate the scene at the distal end of the inner cavity 212. The transmitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared wavelengths, ultraviolet wavelengths, hyperspectral wavelengths, fluorescence excitation wavelengths, or other wavelengths. The inner cavity 212 can be inserted into a patient for imaging, such as during surgery or examination. The light is output as shown by the dashed line 216. The scene illuminated by the light can be captured using an image sensor 214 and displayed to a doctor or some other medical personnel. The controller 204 may provide control signals to the emitter 202 to control when illumination is provided to a scene. In one embodiment, the emitter 202 and the controller 204 are located within a camera control unit (CCU) or an external console to which the endoscope is connected. If the image sensor 214 comprises a CMOS sensor, light may be periodically provided to the scene in a series of illumination pulses between readout cycles of the image sensor 214 during so-called blanking periods. Thus, light may be pulsed in a controlled manner to avoid stacking into the readout cycles of image pixels in the pixel array of the image sensor 214.

[0098] In one embodiment, the lumen waveguide 210 includes one or more optical fibers. Considering the disposal of the lumen waveguide 210 and / or other parts of the endoscope, these optical fibers can be made of low-cost materials (such as plastic). In one embodiment, the lumen waveguide 210 is a single glass fiber with a diameter of 500 microns. The jumper waveguide 206 can be permanently attached to the transmitter 202. For example, the jumper waveguide 206 can receive light from a transmitter in the transmitter 202 and provide light to the lumen waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 106 includes one or more glass optical fibers. The jumper waveguide can include any other type of waveguide for guiding light to the lumen waveguide 210. The connector 208 can selectively couple the jumper waveguide 206 to the lumen waveguide 210 and allow the light in the jumper waveguide 206 to pass through the lumen waveguide 210. In one embodiment, the intracavity waveguide 210 is coupled directly to the light source without any intervening jumper waveguide 206 .

[0099] The image sensor 214 includes a pixel array. In one embodiment, the image sensor 214 includes two or more pixel arrays for generating a three-dimensional image. The image sensor 214 may constitute more than two image sensors, each having an independent pixel array and operable independently of each other. The pixel array of the image sensor 214 includes effective pixels and optically black ("OB") pixels or optically blind pixels. The effective pixels may be transparent "color-indeterminate" pixels capable of sensing imaging data for electromagnetic radiation of any wavelength. When the pixel array is "reset" or calibrated, the optically black pixels are read during the blanking period of the pixel array. In one embodiment, when the optically black pixels are read, light pulses during the blanking period of the pixel array. After the optically black pixels are read, the effective pixels are read during the readout period of the pixel array. The effective pixels may be charged by the electromagnetic radiation pulsed during the blanking period so that the effective pixels are ready to be read by the image sensor during the readout period of the pixel array.

[0100] Figure 2A is a schematic diagram of complementary system hardware such as a special or general-purpose computer. Specific implementations within the scope of the present disclosure may also include physical media and other non-transitory computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media that can be accessed by a general or special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a computer storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, a specific implementation of the present disclosure may include at least two distinct types of computer-readable media: a computer storage medium (device) and a transmission medium.

[0101] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store the desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer.

[0102] "Network" refers to one or more data links that enable electronic data to be transmitted between computer systems and / or modules and / or other electronic devices. In one specific implementation, the sensor and camera control unit can be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer reasonably regards the connection as a transmission medium. The transmission medium may include a network and / or data link that can be used to carry the desired program code tools in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer. The above combinations should also be included in the scope of computer-readable media.

[0103] In addition, upon reaching various computer system components, program code tools in the form of computer executable instructions or data structures can be automatically transmitted to computer storage media (devices) by transmission media (or vice versa). For example, computer executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., "NIC") and then ultimately transmitted to the computer system RAM and / or the non-volatile computer storage media (devices) of the computer system. RAM can also include solid state drives (SSDs or PCIx-based real-time memory hierarchical storage devices, such as FusionIO). Therefore, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0104] Computer executable instructions include instructions and data that, when executed by one or more processors, cause a general purpose computer, a special purpose computer, or a special purpose processing device to perform certain functions or groups of functions. Computer executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter of the present invention is described linguistically with respect to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or steps described above. Rather, the features and steps described above are disclosed as example forms of implementing the claims.

[0105] It should be understood by those skilled in the art that the present disclosure can be implemented in a network computing environment having various types of computer system configurations, including personal computers, desktop computers, laptop computers, information processors, control units, camera control units, handheld devices, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the above computing devices can be provided by a physical institution or located within a physical institution. The present disclosure can also be implemented in a distributed system environment, in which local and remote computer systems are connected via a network (via a hard-wired data link, a wireless data link, or a combination of a hard-wired data link and a wireless data link), and both can perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0106] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0107] Figure 2A is a block diagram illustrating an exemplary computing device 250. Computing device 250 may be used to execute various programs, such as those discussed herein. Computing device 250 may be used as a server, a client, or any other computing entity. Computing device 250 may perform various monitoring functions discussed herein, and may execute one or more applications, such as the applications described herein. Computing device 250 may be any of a variety of computing devices, such as a desktop computer, a laptop computer, a server computer, a handheld computer, a camera control unit, a tablet computer, and the like.

[0108] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory device 254 and / or the mass storage device 258. The processor 252 may also include various types of computer-readable media, such as cache memory.

[0109] Memory device 254 includes various computer-readable media such as volatile memory (e.g., random access memory (RAM) 264) and / or nonvolatile memory (e.g., read-only memory (ROM) 266). Memory device 254 may also include a rewritable ROM such as flash memory.

[0110] The mass storage device 258 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., flash memory), etc. Figure 2 As shown in FIG. 2 , the particular mass storage device is a hard drive 274. Various drives may also be included in the mass storage device 258 to enable reading and / or writing from various computer readable media. The mass storage device 258 includes removable media 276 and / or non-removable media.

[0111] I / O devices 260 include various devices capable of inputting data and / or retrieving data and / or other information to or from computing device 250. Exemplary I / O devices 260 include digital imaging devices, electromagnetic sensors and transmitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCD or other image capture devices, and the like.

[0112] Display device 280 includes any type of device capable of displaying information to one or more users of computing device 250. Examples of display device 280 include a monitor, a display terminal, a video projection device, and the like.

[0113] Interface 256 includes various interfaces that enable computing device 250 to interact with other systems, devices, or computing environments. Exemplary interface 256 may include any number of different network interfaces 270, such as interfaces for connecting local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include user interfaces 268 and peripheral device interfaces 272. Interface 256 may also include one or more user interface elements 268. Interface 256 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touch pad, etc.), keyboards, etc.

[0114] The bus 262 enables the processor 252, the memory device 254, the interface 256, the mass storage device 258, and the I / O device 260 to communicate with each other and with other devices or components coupled to the bus 262. The bus 262 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, etc.

[0115] For exemplary purposes, the programs and other executable program devices shown herein are discrete blocks, but it should be understood that such programs and devices may reside at various times in different storage devices of the computing device 250 and be executed by the processor 252. Alternatively, the systems and programs described herein may be implemented in hardware, or in a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more of the systems and programs described herein.

[0116] Figure 3A An operating cycle of a sensor used in a rolling readout mode or during a sensor readout 300 is shown. A frame readout may begin at and may be represented by a vertical line 310. A readout cycle is represented by a diagonal or slanted line 302. The active pixels of the pixel array of the image sensor may be read out row by row, with the top of the downward sloping edge being the sensor top row 312 and the bottom of the downward sloping edge being the sensor bottom row 314. The time between the last row readout and the next readout cycle may be referred to as a blanking period 316. It should be noted that some of the sensor pixel rows may be covered with a light shield (e.g., a metal coating or any other substantially black layer of another material type). These covered pixel rows may be referred to as optical black rows 318 and 320. The optical black rows 318 and 320 may be used as input to a correction algorithm. As shown Figure 3A As shown, these optically black rows 318 and 320 may be located on the top of the pixel array or on the bottom of the pixel array or on both the top and bottom of the pixel array.

[0117] Figure 3B A method for controlling the amount of electromagnetic radiation (e.g., light) exposed to a pixel so that it is integrated or accumulated by the pixel is shown. It should be understood that photons are the elementary particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into an electric charge or current. An electronic shutter or rolling shutter (shown by dashed line 322) can be used to start the integration time by resetting the pixel. The light will then be integrated until the next readout phase. The position of the electronic shutter 322 can be moved between two readout cycles 302 to control the pixel saturation of a given amount of light. It should be noted that this technique allows a constant integration time between two different rows, but introduces a delay when moving from the top row to the bottom row.

[0118] Figure 3C The case is shown where the electronic shutter 322 has been removed. In this configuration, integration of incident light may begin during readout 302 and may end at the next readout cycle 302, which also defines the start of the next integration.

[0119] Figure 3DA configuration is shown without the electronic shutter 322 but with controlled and pulsed light 330 during the blanking period 316. This ensures that all rows see the same light as that emitted from the same light pulse 330. In other words, each row will begin its integration in a dark environment, which may be located at the optically black post row 320 of the readout frame (m) to obtain the maximum light pulse width, and will then receive light through and will end its integration in a dark environment, which may be located at the optically black pre row 318 of the next subsequent readout frame (m+1) to obtain the maximum light pulse width. In e.g. Figure 3D In the blanking period 316, the image generated by the light pulse will be available only during the readout of frame (m+1) without interfering with frames (m) and (m+2). It should be noted that the condition for the light pulse to be read out only in one frame and not interfere with adjacent frames is that a given light pulse is fired during the blanking period 316. Because the optical black rows 318, 320 are not sensitive to light, the optical black post row 320 time of frame (m) and the optical black pre row 318 time of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 330.

[0120] like Figure 3A As shown, the sensor may cycle multiple times to receive data for each pulse color or wavelength (e.g., red, green, blue or other wavelength on the electromagnetic spectrum). Each cycle may be timed. In one embodiment, the cycle may be timed to operate within an interval of 16.67 ms. In another embodiment, the cycle may be timed to operate within an interval of 8.3 ms. It should be understood that the present disclosure contemplates other timing intervals and is intended to fall within the scope of the present disclosure.

[0121] Figure 4A The operation of an embodiment of an electromagnetic transmitter is graphically illustrated.The transmitter may be timed to correspond to a cycle of the sensor such that electromagnetic radiation is emitted within and / or during a portion of a sensor operating cycle. Figure 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the emitter may pulse during the readout period 302 of the sensor operating cycle. In one embodiment, the emitter may pulse during the blanking portion 316 of the sensor operating cycle. In one embodiment, the emitter may pulse for a duration that is during a portion of two or more sensor operating cycles. In one embodiment, the emitter may begin pulsing during the blanking portion 316 or during the optically black portion 320 of the readout period 302, and end pulsing during the readout period 302 or during the optically black portion 318 of the readout period 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of the present disclosure as long as the pulsing of the emitter and the cycle of the sensor correspond.

[0122] Figure 4B A graphical representation of varying the duration and magnitude of emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, pulse 3 at 416) to control exposure. An emitter with a fixed output magnitude may be combined with the above Figure 3D and Figure 4A The pulses are pulsed at intervals during any of the cycles described to provide the desired electromagnetic energy to the pixel array. An emitter with a fixed output magnitude may be pulsed at longer intervals to provide more electromagnetic energy to the pixel, or the emitter may be pulsed at shorter intervals to provide less electromagnetic energy. Whether longer or shorter intervals are required depends on the operating conditions.

[0123] Compared to adjusting the time interval at which the emitter pulses a fixed output magnitude, the emission magnitude itself can be increased to provide more electromagnetic energy to the pixel. Similarly, decreasing the magnitude of the pulses can provide less electromagnetic energy to the pixel. It should be noted that embodiments of the system can have the ability to adjust both magnitude and duration if desired. In addition, the sensor can be adjusted to increase its sensitivity and duration as needed for optimal image quality. Figure 4B The magnitude and duration of the pulses are shown to be varied. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. In addition, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416, so that the electromagnetic energy provided by the pulse is shown by the area under the pulse shown in the illustration. In the illustration, pulse 2 at 414 has a relatively low magnitude or intensity and a longer duration when compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration when compared to pulse 1 at 412 and pulse 2 at 414.

[0124] Figure 5 It is a combination of the principles and teachings of this disclosure FIG. 3A to FIG. 3D 4, an operation cycle, an electromagnetic emitter, and a graphical representation of an embodiment of the present disclosure of an electromagnetic emitter and transmitting electromagnetic pulses to display an imaging system during operation. As can be seen in the figure, the electromagnetic emitter pulses primarily during the blanking period 316 of the image sensor so that the pixels will be charged and ready to be read during the readout period 302 of the image sensor cycle. Figure 5 The dashed lines in the figure represent electromagnetic radiation pulses (from Figure 4A ). The electromagnetic radiation pulses are primarily emitted during the blanking period 316 of the image sensor, but may overlap with the readout period 302 of the image sensor.

[0125] The exposure frame includes data read by the pixel array of the image sensor during the readout cycle 302. The exposure frame can be combined with an indication of what type of pulse was emitted by the transmitter before the readout cycle 302. The combination of the exposure frame and the indication of the type of pulse can be referred to as a data set. Multiple exposure frames can be combined to generate a black and white or RGB color image. In addition, hyperspectral imaging data, fluorescence imaging data, and / or laser mapping imaging data can be superimposed on the black and white or RGB image.

[0126] In one embodiment, an RGB image frame is generated based on three exposure frames, including a red exposure frame generated by an image sensor after red emission, a green exposure frame generated by an image sensor after green emission, and a blue exposure frame generated by an image sensor after blue emission. Fluorescence imaging data may be superimposed on the RGB image frame. Fluorescence imaging data may be extracted from one or more fluorescence exposure frames. The fluorescence exposure frame includes data generated by the image sensor during the readout cycle 302 after emitting electromagnetic radiation of an excitation wavelength for exciting a fluorescent agent. The data sensed by the pixel array after exciting the fluorescent agent may be a relaxation wavelength emitted by the fluorescent agent. The fluorescence exposure frame may include a plurality of fluorescence exposure frames, each of which is generated by the image sensor after a different type of fluorescence excitation emission. In one embodiment, the fluorescence exposure frame includes a plurality of fluorescence exposure frames, including a first fluorescence exposure frame generated by the image sensor after emitting electromagnetic radiation having a wavelength from about 770nm to about 790nm, and a second fluorescence exposure frame generated by the image sensor after emitting electromagnetic radiation having a wavelength from about 795nm to about 815nm. The fluorescence exposure frames may include further additional fluorescence exposure frames generated by the image sensor after other fluorescence excitation emissions of light as needed based on the imaging application.

[0127] In one embodiment, the exposure frame is data sensed by the pixel array during a readout period 302 that occurs after a blanking period 316. The emission of electromagnetic radiation is emitted during the blanking period 316. In one embodiment, a portion of the emission of electromagnetic radiation is overlapped with the readout period 316. The blanking period 316 occurs when optically black pixels of the pixel array are read, and the readout period 302 occurs when valid pixels of the pixel array are read. The blanking period 316 can be overlapped with the readout period 302.

[0128] Fig. 6A and Figure 6B A process for recording an image frame is shown. Multiple image frames may be strung together to generate a video stream. A single image frame may include data from multiple exposure frames, where an exposure frame is data sensed by a pixel array after emitting electromagnetic radiation. Fig. 6AA conventional process is shown that is typically implemented with a color image sensor having a color filter array (CFA) to filter out certain wavelengths of light for each pixel. Figure 6B is a process disclosed herein and can be implemented using a monochrome "color-indeterminate" image sensor capable of receiving electromagnetic radiation of all wavelengths.

[0129] Fig. 6A The process shown occurs from time t(0) to time t(1). The process begins by emitting white light 602 and sensing white light 604. At 606, an image is processed and displayed based on the sensing at 604.

[0130] Figure 6B The process shown occurs from time t(0) to time t(1). The process first emits green light 612 and senses reflected electromagnetic radiation 614 after emitting green light 612. The process continues to emit red light 616 and sense reflected electromagnetic radiation 618 after emitting red light 616. The process continues to emit blue light 620 and sense reflected electromagnetic radiation 622 after emitting blue light 620. The process continues with one or more of the dedicated 624 emissions and senses reflected electromagnetic energy 626 after each of the one or more of the dedicated 624 emissions. The dedicated emissions may include one or more separate emissions, such as excitation wavelengths of fluorescent agents, hyperspectral emissions, and / or laser marking emissions. Each of the separate multiple dedicated emissions can be independently sensed by the image sensor to generate a separate and independent exposure frame. At 628, an image is processed and displayed based on each of the sensed reflected electromagnetic energy instances 614, 618, 622, and 626.

[0131] Figure 6B The process shown provides a higher resolution image and provides a means for generating an RGB image that also includes special data. Figure 6B ), making the sensor sensitive to electromagnetic energy of all wavelengths. Figure 6B In the process shown, the monochrome pixel array is instructed to sense electromagnetic energy from a predetermined partition of the full spectrum electromagnetic energy in each cycle. Therefore, in order to form an image, the sensor only needs to cycle through multiple different partitions within the full spectrum light. The final image is combined based on multiple cycles. Because the image from each color partition frame cycle has a higher resolution (compared to the CFA pixel array), the resulting image formed when the partitioned light frames are combined also has a higher resolution. In other words, because every pixel within the array (rather than, at most, every other pixel in a sensor with a CFA) is sensing an amount of energy for a given pulse and a given scene, a higher resolution image is formed for each scene only at intervals.

[0132] As in Fig. 6A and Figure 6B In the embodiment shown in FIG. 1 , it can be seen graphically between time t(0) and t(1) that Figure 6B Sensor pair in the partitioned spectral system Fig. 6A Each system in the full spectrum system in the embodiment is cycled at least four times. In one embodiment, the display device (LCD panel) operates at a speed of 50 to 60 frames per second. In such embodiments, Figure 6B The partitioned optical system in can operate at a rate of 200 frames / second to 240 frames / second to maintain the continuity and smoothness of the displayed video. In other embodiments, there may be different capture and display frame rates. In addition, the average capture rate can be any multiple of the display rate.

[0133] In one embodiment, it may be desirable that not all partitions are represented equally within the system frame rate. In other words, not all light sources must be pulsed with the same regularity in order to emphasize and de-emphasize various aspects of the recorded scene as desired by the user. It should also be understood that invisible and visible partitions of the electromagnetic spectrum can be pulsed together within the system, with their corresponding data values ​​stitched into the video output for display to the user.

[0134] An exemplary embodiment may include the following pulse cycle pattern:

[0135] i. Green pulse;

[0136] ii. Red pulse;

[0137] iii. Blue pulse;

[0138] iv. Green pulse;

[0139] v.Red pulse;

[0140] vi. Blue pulse;

[0141] vii. Laser mapping pulse scheme;

[0142] viii. Fluorescence excitation pulse;

[0143] ix. Hyperspectral pulses;

[0144] x. (repeat)

[0145] Another exemplary embodiment may include the following pulse cycle pattern:

[0146] i. Green pulse;

[0147] ii. Red pulse;

[0148] iii. Blue pulse;

[0149] iv. Fluorescence excitation pulse;

[0150] v.Hyperspectral pulse;

[0151] vi. Green pulse;

[0152] vii. Red pulse;

[0153] viii. Blue pulse;

[0154] ix. Fluorescence excitation pulse;

[0155] x.Hyperspectral pulse;

[0156] xi. Laser mapping pulse scheme;

[0157] xii. (repeat)

[0158] The pulse mode may be changed to accommodate the imaging objectives of a particular implementation. One exemplary imaging objective is to obtain hyperspectral imaging data and fluorescence imaging data, and also to obtain laser mapping data and / or tool tracking data based on an analysis of the hyperspectral imaging data and / or fluorescence imaging data. In such examples, the laser mapping data and / or tool tracking data may be analyzed for certain areas of the scene that have been highlighted by the hyperspectral imaging data and / or fluorescence imaging data. Another exemplary imaging objective is to obtain hyperspectral imaging data or fluorescence imaging data, and also to obtain laser mapping data and / or tool tracking data. Another exemplary imaging objective is to obtain laser mapping data and / or tool tracking data. Another exemplary imaging objective is to obtain hyperspectral imaging data. Another exemplary imaging objective is to obtain fluorescence imaging data. It should be understood that the imaging objectives may be specialized, depending on the reason for deploying the imaging system. In addition, the imaging objectives may change during a single imaging session, and the pulse mode may be changed to match the changing imaging objectives.

[0159] As can be seen in this example, a laser marked partition can be pulsed at a different rate than the other partitions are pulsed. Doing so can emphasize a certain aspect of the scene, with the laser marked data simply overlaid with other data in the video output to make the desired emphasis. It should be noted that adding laser marked partitions on top of the red, green, and blue partitions does not necessarily require the serialized system to operate at four times the rate of a full spectrum non-serial system, as each partition does not have to be represented equally in the pulse pattern. As seen in this embodiment, adding a partition pulse that is represented less in the pulse pattern (laser marking in the above example) will result in an increase of less than 20% of the sensor's cycle speed to accommodate irregular partition sampling.

[0160] In various embodiments, the pulse cycle pattern may also include any of the following wavelengths in any suitable order. Such wavelengths may be particularly suitable for exciting fluorescent agents to generate fluorescent imaging data by sensing the relaxation emission of the fluorescent agent based on the relaxation emission of the fluorescent agent:

[0161] i.770±20nm;

[0162] ii. 770±10nm;

[0163] iii. 770±5nm;

[0164] iv. 790±20nm;

[0165] v.790±10nm;

[0166] vi. 790±5nm;

[0167] vii. 795±20nm;

[0168] viii. 795±10nm;

[0169] ix. 795±5nm;

[0170] x.815±20nm;

[0171] xi. 815±10nm;

[0172] xii. 815±5nm;

[0173] xiii. 770nm to 790nm; and / or

[0174] xiv. 795nm to 815nm.

[0175] In various embodiments, the pulse cycle may also include any of the following wavelengths in any suitable order. Such wavelengths may be particularly suitable for generating hyperspectral imaging data:

[0176] i. 513nm to 545nm;

[0177] ii. 565nm to 585nm;

[0178] iii. 900nm to 1000nm;

[0179] iv. 513±5nm;

[0180] v.513±10nm;

[0181] vi. 513±20nm;

[0182] vii. 513±30nm;

[0183] viii. 513±35nm;

[0184] ix. 545±5nm;

[0185] x.545±10nm;

[0186] xi. 545±20nm;

[0187] xii. 545±30nm;

[0188] xiii. 545±35nm;

[0189] xiv. 565±5nm;

[0190] xv. 565±10nm;

[0191] xvi. 565±20nm;

[0192] xvii. 565±30nm;

[0193] xviii. 565±35nm;

[0194] xix. 585±5nm;

[0195] xx. 585±10nm;

[0196] xxi. 585±20nm;

[0197] xxii. 585±30nm;

[0198] xxiii. 585±35nm;

[0199] xxiv. 900±5nm;

[0200] xxv. 900±10nm;

[0201] xxvi. 900±20nm;

[0202] xxvii. 900±30nm;

[0203] xxviii. 900±35nm;

[0204] xxix. 1000±5nm;

[0205] xxx. 1000±10nm;

[0206] xxxi. 1000±20nm;

[0207] xxxii. 1000±30nm; or

[0208] xxxiii. 1000±35nm.

[0209] The partition loop can be divided into various imaging and video standards to accommodate or approximate. In one embodiment, the partition loop may include the following: 7A to 7D Pulses of electromagnetic energy in the red, green, and blue spectrum are best shown in Figure 1. Fig. 7A In , different light intensities have been achieved by modulating the light pulse width or duration within the operating range shown by the vertical grey dashed line. Figure 7B In , different light intensities have been achieved by modulating the power of an optical or electromagnetic emitter (which may be a laser or LED emitter), but keeping the pulse width or duration constant. Figure 7C The case where both the light power and light pulse width are modulated for greater flexibility is shown. The zoned loop can use cyan magenta yellow (CMY), infrared, ultraviolet, hyperspectral and fluorescent, which uses invisible pulse sources mixed with visible pulse sources, and any other color space required to produce an image or approximate a desired video standard currently known or yet to be developed. It should also be understood that the system can switch between color spaces on the fly to provide the desired image output quality.

[0210] When using the color space green-blue-green-red (such as Fig.7D In an embodiment (as shown in ), it may be desirable to pulse the luma component more frequently than the chroma component because users are generally more sensitive to differences in light magnitude than to differences in light color. Fig.7D The monochrome sensor shown in Figure utilises this principle. Fig.7D In the scheme, green, which contains the most brightness information, can be pulsed more frequently or with greater intensity to obtain brightness data in the (GBGRGBGR...) scheme. Such a configuration will create a video stream with more perceptible details without creating and transmitting imperceptible data.

[0211] In one embodiment, duplicate pulses of weaker partitions may be used to produce an output that has been adjusted for the weaker pulses. For example, given the weak sensitivity of blue lasers relative to silicon-based pixels, and the difficulty of producing blue lasers compared to red or green light, it may be pulsed more frequently during a frame cycle to compensate for the weakness of the light. These additional pulses may be done continuously over time, or by using multiple lasers to pulse simultaneously to produce the desired compensating effect. It should be noted that by pulsing during blanking periods (time during which the sensor is not reading out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same kind, and simply accumulates the light for the desired output. In another embodiment, the maximum light pulse range may be different from frame to frame. This may be done in a controlled manner. Fig. 7E , where the light pulses are different from frame to frame. The sensor can be constructed to be able to program different blanking periods in a repeating pattern of 2 frames or 3 frames or 4 frames or n frames. Fig. 7E In FIG. 4 , four different light pulses are shown, and pulse 1 may be repeated, for example, after pulse 4 and may have a pattern of four frames with different blanking periods. This technique can be used to place the most powerful partition on the smallest blanking period, and thus allow the weakest partition to have a wider pulse on one of the subsequent frames without increasing the readout speed. The reconstructed frame can still have a regular pattern from frame to frame, because it is composed of many pulse frames.

[0212] Reference now Fig. 8A and Figure 8B , showing a specific implementation of an image sensor. Fig. 8A The image sensor shown is a conventional image sensor 800 including a pixel array as is common in the prior art. Figure 8B The illustrated minimum area image sensor 820 is a pixel array according to the teachings and principles of the present disclosure. The minimum area image sensor 820 can be used in the distal end of an endoscope. The distal end of an endoscope must be small because the distal end is usually inserted into a small area (such as a body cavity). The distal end of an endoscope is usually too small to accommodate a traditional image sensor 800.

[0213] In spatially constrained environments, such as the distal tip of an endoscope, it may be desirable to reduce the non-sensitive portion of the image sensor to maximize the area of ​​the image sensor dedicated to light sensing elements. This can reduce the overall area of ​​the image sensor while maintaining image quality. One specific implementation for reducing the non-light sensing elements in the image sensor is to eliminate or reduce the number of optical black ("OB") rows in the image sensor. However, eliminating optical black rows may also eliminate the possibility of implementing OB-based column fixed pattern noise elimination methods commonly used in conventional imaging systems. Column fixed pattern noise elimination requires black pixel data, where there are typically 10 to 100 black pixels per column. Before the light sensing pixels are read out, the black pixels are corrected for column-wise random offsets. Disclosed herein are alternative methods, systems, and apparatus for correcting fixed pattern noise when using a minimum area image sensor 820.

[0214] The image sensor may incorporate dedicated, light-blind, or optically black ("OB") rows and / or columns. Optically black pixels may alternatively be referred to as manufacturing buffer pixels or dummy pixels. Optically black rows 804 are located at the top and / or bottom of recording area 802. Recording area 802 includes an array of imaging pixels. Optically black columns 806 are located to the right and / or left of the pixel array. Optically black rows and columns are used for calibration of offset recording area 802.

[0215] The exemplary layout of the conventional image sensor 800 includes optical black rows 804 on the top and bottom of the conventional recording area 802. The conventional image sensor 800 also includes optical black columns 806 on the left and right sides of the recording area 802. The layout of the minimum area image sensor 820 includes optical black columns 806 on the left and right sides of the recording area 802. The minimum area image sensor 820 does not include any optical black rows 804. Each of the exemplary embodiments may include a guard ring 808 around the periphery of the image sensor 800, 820.

[0216] The optical black row 804 may be used to monitor the analog pixel black level for the purpose of the optical black clamping algorithm. The optical black row 804 may also be used by the digital algorithm for the purpose of canceling column fixed pattern noise (CFPN). The optical black column 806 may be used to evaluate line offset as a way to cancel line noise in the recording area 802. Because line noise may be temporal, the line offset may be recalculated for each line of the recording area 802 in each frame.

[0217] The minimum area image sensor 820 provides an overall reduction in the size of the image sensor by removing the optical black rows 804 from the top and bottom sides of the recording area 802. When the minimum area image sensor 820 is deployed, the optical black columns 806 are used for the optical black clamping algorithm instead of any optical black rows 804. In one embodiment, all fixed pattern noise including column fixed pattern noise can be cancelled by acquiring a dark data frame. This eliminates the need for a dedicated CFPN correction and its associated optical black rows 804.

[0218] Depending on the space constraints, the number of optical black columns 806 can typically be 100 or more. The more optical black columns 806 are available, the greater the line offset accuracy can be. Greater accuracy means lower line noise after correction. Typically, all available physical optical black pixels are read for each line, such as Fig. 9 A further degree of pixel array size reduction can be achieved if (given some accuracy target) a smaller number of physical optical black pixels are implemented during the horizontal readout process of the recording area 802 and the optical black columns 806 and the physical optical black pixels are resampled multiple times, rather than having a necessary number of physical optical black pixels. Fig.10 Shown in.

[0219] In an alternative embodiment, the minimum area image sensor includes optically black rows 804 but does not include optically black columns 806. In such an embodiment, the optically black rows 804 may be read before and after reading the active pixels in the recording area 802 of the pixel array.

[0220] The disclosure provided herein is applicable to all forms of fixed pattern noise and is not necessarily limited to cancelling column fixed pattern noise. Other types of fixed pattern noise can be canceled by acquiring a dark data frame to eliminate the need for a dedicated column fixed pattern noise correction and its associated optical black row. In one embodiment, a closed sensor frame of memory is used to store calibration data for each pixel and then cancel all fixed pattern noise sources at the pixel level. Although more memory may be required for black frame calibration compared to a column fixed pattern noise algorithm, there may be advantages because all fixed pattern noise sources can be eliminated or significantly reduced by the methods, systems and devices disclosed herein. Exemplary fixed pattern noise sources that can be eliminated or reduced by the disclosure herein include column fixed pattern noise, pixel fixed pattern noise, and row fixed pattern noise.

[0221] Fig. 9 FIG. 9 shows a readout sequence 908 of a pixel array commonly used in the prior art. Fig. 9 As shown, the readout sequence 908 begins with a column readout 906 for each of the optically black columns 904. The optically black columns 904 may be similar to Fig. 8Aand Figure 8B After completing the column readout 906 for the optical black column 904, the readout sequence 908 begins the column readout 906 for the recording area 910. The recording area 910 may be similar to Fig. 8A and Figure 8B The recording area 802 is shown. The recording area 910 may include transparent pixels 902 or color-indeterminate pixels. The readout sequence 908 continues until the entire recording area 910 has been read. After the recording area 910 has been read, the optically black column 904 on the opposite side is read. The readout sequence 908 continues line by line until each row of the pixel array has been read.

[0222] Fig.10 A readout sequence 1008 of a pixel array with reduced area size having a reduced number of optically black pixels is shown. Fig.10 In the readout sequence 1008 shown, the optically black column 1004 undergoes column readout 1006 multiple times. Fig.10 As shown, the optical black column 1004 readout may be resampled m times or n times. The recording area may include transparent pixels 1002.

[0223] The raw CMOS image sensor data present at the output of the digitizer may be far from ideal. It may often be the case that the optimal order for reading out the horizontal pixel rows does not equal the actual physical order within the array. In addition, the raw data often reveals undesirable artifacts that reflect the nature of the readout architecture. These readout artifacts may include column fixed pattern noise caused by column-by-column offset variations, and may also include timeline noise resulting from circuit resets associated with the horizontal readout process.

[0224] Another characteristic of CMOS (complementary metal oxide semiconductor) sensors is that a certain degree of dark signal can be generated by the photodiode within the pixel. The amount of integrated signal caused by the dark signal depends on the exposure time and temperature. Because the dark signal may be indistinguishable from the light signal, changes in the dark signal translate into changes in the signal reference in the analog domain. Therefore, it may be important to sample and adjust the dark signal so that the available dynamic range of the analog-to-digital converter can be fully utilized.

[0225] Fig.11 is a graphical representation of a simple exponential smoothing (SES) pixel shift estimate with small and large values ​​of w. The SES pixel shift estimate may be implemented to reduce or eliminate fixed pattern noise in an exposure frame.

[0226] Exponential smoothing is a time series prediction method. Exponential smoothing produces a prediction that is a weighted sum of past observations, where the weights decay exponentially as the past observations become older. In one embodiment, SES is used to generate a reference frame based on multiple dark exposure frames. Each dark exposure frame in the multiple dark exposure frames is generated by a pixel array and includes dark frame data sensed when the emitter does not emit electromagnetic radiation. The reference frame can be applied to reduce fixed pattern noise in the exposure frames and / or image frames generated by the image sensor. SES is used to increase the accuracy of the reference frame. In one embodiment, the SES algorithm explicitly uses exponentially decreasing weights for considering previously captured dark exposure frames.

[0227] In one embodiment, SES is applied to dark frame data from multiple dark exposure frames. SES is used to generate a reference frame based on the dark frame data captured in multiple dark exposure frames. The reference frame does not include imaging data displayed on a screen or otherwise provided to a user, but includes data for removing fixed pattern noise from an exposure frame and / or an image frame. The reference frame is calculated by applying SES to multiple dark exposure frames. Multiple dark exposure frames are captured over time, and when calculating the reference frame, older dark exposure frames are given less weight. The reference frame includes reference data that is subtracted from the exposure frame and / or the image frame to facilitate reducing fixed pattern noise. When the light-deficient environment is dark, each of the multiple dark exposure frames is sensed by a pixel array, and the emitter does not pulse to emit electromagnetic radiation.

[0228] In one embodiment, the emitter emits pulses of electromagnetic radiation in a pattern including electromagnetic radiation of different wavelengths. In one embodiment, the pattern includes electromagnetic radiation of red, green, blue and laser calibration wavelengths. The pattern may also include an instance in which the emitter does not emit electromagnetic radiation. The pixel array may be configured to sense dark frame data when the emitter does not emit electromagnetic radiation. When the pixel array senses dark frame data, this results in a dark exposure frame. In one embodiment, the pattern of electromagnetic radiation of different wavelengths also includes an instance in which the emitter does not emit electromagnetic radiation. The pixel array generates an exposure frame in response to each emission of electromagnetic radiation, and further generates a dark exposure frame in response to an instance in which the emitter does not emit electromagnetic radiation. The exposure frame and the dark exposure frame may be captured in a repeating pattern over time, so that the dark exposure frame is interspersed between (non-dark) exposure frames. A series of exposure frames may be combined to generate an image frame, and in one embodiment, there is a dark exposure frame corresponding to each image frame or a grouping of image frames. A series of dark exposure frames are used to generate a reference frame, and the reference frame is used to reduce the fixed pattern noise present in the exposure frame (and the image frame). SES is applied to a series of dark exposure frames such that older dark exposure frames have less weight than newer dark exposure frames.

[0229] In one embodiment, each time a dark exposure frame is sensed by the pixel array, a single reference frame is incrementally adjusted. Each pixel sample acquired in the dark exposure frame is divided by the appropriate binary number (2 w ), then added to the buffer contents and multiplied by (2 w -1) / 2 w . Fig.11 The effect on the pixel offset estimate is shown for small and large values ​​of w. Fig.11 As shown, high values ​​of w lead to greater statistical accuracy over time in stable scenarios. Lower values ​​of w render the SES correction more reactive to rapid changes at the expense of accuracy and stability. In one embodiment, the value of w may be tuned via a control register in communication with the endoscope system.

[0230] In one embodiment, the dark current temperature dependency can be overcome by using a running average calibration as described herein. In rolling shutter operation, the pixel integration time can be changed on a frame-by-frame basis to accommodate changes in the light environment controlled by the auto-exposure algorithm. For example, FIG. 3A to FIG. 3D The frame readout sequence shown shows an electronic shutter that is adjusted within the frame to control the integration time. Because calibration can be performed at a given integration time, the dark frame correction algorithm can lose its effectiveness and become inaccurate when attempting to correct frames with different integration times. One specific implementation of the present disclosure for overcoming this problem is to introduce a second calibration point for each pixel. This may require a second frame of memory. The second calibration point for each pixel may be collected at different integration times. Linear interpolation may be used to calculate pixel calibration data at a given integration time. However, the use of linear interpolation may introduce interpolation errors and complicate back-end processing of the image data.

[0231] When pulsing light in a light-deficient environment, the pixel integration time can be substantially constant from frame to frame. Figure 3D , where the light intensity is adjusted by changing the light pulse width or level. In this specific implementation, only one integration time calibration may be required, and interpolation may not be required. In one embodiment, dark frame subtraction can be used to adjust the average offset of each pixel. This can suppress all forms of fixed pattern noise. Because the pixel offset has temperature dependence, the correction can be a running average process by acquiring sample dark frames at regular intervals and updating the stored correction data based on the samples. The resulting quality of this correction can depend on the sampling statistics.

[0232] In one embodiment, the uncertainty of the pixel offset estimate is equal to the temporal noise of the pixel divided by the square root of the number of samples. This uncertainty can be directly converted to the corrected pixel fixed pattern noise and is independent of the original fixed pattern noise. In an embodiment where the video stream is captured at a rate of 60 image frames per second, the pixel fixed pattern noise can be less than one quarter of the pixel temporal noise to be imperceptible to a human viewer. The statistical value of the perceptual criterion depends only on the pixel temporal noise, as follows:

[0233]

[0234]

[0235] ∴N f ≥16

[0236] where σ T is the pixel temporal noise. Therefore, the frame correction process should be effective as long as there are at least 16 dark frames to calculate the average.

[0237] In one embodiment, SES capture for dark frames only is calculated according to:

[0238] b i，j =d i，j (j=0)

[0239]

[0240] where b i，j is the dark frame correction buffer content of pixel i following dark frame number j, and d i，j is the raw dark data of pixel i obtained from dark frame j. w is a tunable integer.

[0241] In one embodiment, SES is applied to non-dark exposure frames according to:

[0242] x′ i =x i -b i +B

[0243] where x i is the raw data input for pixel i in any non-dark frame, and b i Is the current dark frame buffer content. x' i is the output and B is the black clamp target level.

[0244] Fig.12 is a graphical representation of the modulation of the simple exponential smoothing (SES) indicator after a gain change. In one embodiment, the value of w can be tuned via a control register that communicates with the endoscope system. When T JWhen the gain or exposure time changes most significantly, or in some embodiments, after a change in exposure time, the value of w can be automatically adjusted to ensure that the correction is reactive when it is activated. For example, if a large change in gain or exposure time occurs, the value of w can be reduced and then incrementally restored to the baseline on a frame-by-frame basis. The incremental adjustment can change linearly or in a roughly exponential manner, such as Fig.12 shown.

[0245] Fig.13 13 is a schematic diagram of a process flow 1300 for eliminating fixed pattern noise. The process flow 1300 includes sampling a first pixel at 1302 and resetting a frame buffer to zero. The process flow 1300 includes determining whether the sampled pixel is a dark pixel (which may be referred to as an optically black pixel) at 1304. If it is determined at 1304 that the sampled pixel is a dark pixel, the process flow 1300 includes determining whether the buffer of the pixel is zero at 1306. If it is determined at 1306 that the buffer of the dark pixel is zero, then at 1308, the pixel value is written directly to the buffer location of the dark pixel. At 1310, the next pixel is sampled. If it is determined at 1306 that the buffer of the dark pixel is not equal to zero, then at 1314, the buffer location of the dark pixel is read and multiplied by Add the result to the new pixel sample and divide by 2 w And then written back to the buffer. At 1310 the next pixel is sampled.

[0246] If it is determined at 1304 that the sampled pixel is not a dark pixel, process flow 1300 includes reading a buffer of valid (non-dark) pixels and subtracting the contents from the data at 1312. Process flow 1300 may also include adding a constant black level if necessary. At 1310, the next pixel is sampled.

[0247] Fig.14 A process flow 1400 is shown to be implemented by a controller and / or a monochrome image signal processor (ISP) for generating a video stream having RGB images with application specific data layered thereon. The application specific data may include, for example, hyperspectral imaging data, fluorescence imaging data, and / or laser mapping or topographic imaging data. An image signal processor (ISP) chain may be assembled for the purpose of generating a sequence of sRGB images from raw sensor data produced in the presence of a GRGB-laser mapping light pulse scheme. In process flow 1400, the first stage involves performing a correction (see Fig.141408a, 1408b, and sensor correction at 1406) to account for any non-ideal factors in the sensor technology that are best suited to work in the raw data domain. In the next stage, multiple frames (e.g., green frame 1408a, red-blue frame 1408b, and special frame 1408c) are buffered because each final frame derives data from multiple raw frames. Special frame 1408c may include one or more special exposure frames, such as hyperspectral exposure frames, fluorescence exposure frames, and / or topological or laser mapping exposure frames. Frame reconstruction at 1264 continues by sampling data from the current frame and two buffered frames (see 1408a, 1408b, and / or 1408c). The reconstruction process results in a full-color frame including special image data in a linear RGB color space. In this example, white balance coefficients at 1418 and a color correction matrix at 1420 are applied before conversion to YCbCr space at 1422 for subsequent edge enhancement at 1424. After edge enhancement at 1424, the image is converted back to linear RGB at 1426 for scaling (if applicable) at 1428. Finally, a gamma transfer function is applied at 1430 to convert the data to the sRGB domain at 1432.

[0248] Fig.15 1504 and 1505. The color fusion hardware 1500 uses a memory writer 1502 to store an RGBG-dedicated video data stream in a memory 1504 and converts the video data stream into a parallel RGB+dedicated video data stream at 1505. The bit width on the input side may be, for example, 12 bits per color. The output width of this example will be at least 36 bits per pixel. Other embodiments may have different initial bit widths and a number 3 times the output width. The memory writer 1502 box takes the RGBG-dedicated video stream as its input and writes each frame to its correct frame memory 1504 (the memory writer triggers the same pulse generator that runs the laser source (see 2510)). The memory 1504 may store the exposure frame data in a pattern such as the one shown, i.e., red, green 1, blue, green 2, dedicated, and then start back again with red. The memory reader 1506 reads three frames at a time to construct an RGB pixel. Each pixel is three times the bit width of a single color component. At 2510, the memory reader 1506 also triggers the laser pulse generator. In one embodiment, the memory reader 1506 waits until the Red frame, Green 1 frame, and Blue frame have been written, and then continues to read them out in parallel while the writer continues to write Green 2, Special, and begins returning to Red. When Red is complete, the reader begins reading from Blue, Green 2, Special, and Red. This pattern continues indefinitely.

[0249] Fig.1616 is a schematic diagram of a process flow 1600 for a sensor correction process. Process flow 1600 is an exemplary implementation of a front end of an image signal process that has been developed in the context of a system incorporating a minimum area image sensor. In the exemplary process flow 1600, there are two digitizers on the sensor that convert even and odd columns, respectively, and transmit serial data on two differential ports. Process flow 1600 can be used in color and hyperspectral, fluorescence, and / or laser calibration pulse systems as discussed herein. Process flow 1600 can be used to offset non-ideal factors in CMOS image sensors, such as fixed pattern noise (FPN) and row noise. Fixed pattern noise is the dispersion in the offset of the sensing element. Typically, most of the FPN is inter-pixel dispersion, which originates from random variations in dark current from photodiode to photodiode and other sources. The system disclosed herein maintains full control of the illumination source, and this enables dark data to be collected and used to correct pixel and column offsets. In the illustrated example, a single frame buffer can be used to perform a sliding average over the entire frame in the absence of light using, for example, a simple exponential smoothing filter. During normal operation, this dark average frame can be subtracted from each illuminated frame. Row noise is the random temporal variation of pixel offsets within each row. Because row noise is temporal, corrections are calculated for each row and each frame. To do this, there are typically many optically blind (OB) pixels within each row in the pixel array. Before sampling the light-sensitive pixels, the OB pixels must first be sampled to estimate the line offset. The row offset is then subtracted during the row noise correction process.

[0250] The process flow 1600 includes performing top deserialization 1602 and bottom deserialization 1603. The process flow 1600 includes performing line reordering at the top port at 1604 and performing line reordering at the bottom port at 1605. When the line reordering is completed, the information can be stored in a separate database 1632, 1634 or other memory device. The process flow 1600 includes performing black clamp calculations on the top ADC at 1606 and performing black clamp calculations on the bottom ADC at 1607. The information exits the process flow 1600 on a first-in-first-out (FIFO) basis. The process flow 1600 includes performing row noise correction at the top ADC at 1608 and performing row noise correction at the bottom ADC at 1609. The process flow 1600 includes performing full row reordering at 1610 and performing dark frame accumulation at 1612. The information can be stored in the database 1630 or other storage device before performing fixed pattern noise (FPN) correction. The process flow includes performing fixed pattern noise (FPN) correction at 1614 and performing pixel defect correction at 1616. The process flow 1600 includes performing programmable digital gain at 1618 before the video stream leaves the process flow 1600 to be provided to a user.

[0251] Individual optical black pixels that behave abnormally can severely degrade the quality of the black shift measurement. The method disclosed herein includes calculating a median of a set of five pixels for each optical black pixel. The five pixels include the pixel in question and its four nearest neighbors. The final line shift estimate can then be calculated as the average of all the medians. In one embodiment, some provisions are made to avoid losing statistical values ​​at the beginning and end. Such provisions include buffering the entire sample of optical black and surrounding five samples. Buffering requires pipeline transfer of data and causes a delay at least equal to the total number of optical blacks for each analog-to-digital converter channel for each row.

[0252] For line skew estimation for even channels (assuming two ADCs with even-odd spread), row #r can be calculated as follows:

[0253]

[0254] Line offset N OB can be the total number of optically black pixels per row, and μ i can be the median value of the optically black pixels i and is thus calculated:

[0255]

[0256]

[0257] μ4 = median [x0, x2, x4, x6, x8]

[0258]

[0259] Likewise, for line skew estimation for odd channels (assuming two ADCs with even-odd spread), row #r can be calculated as follows:

[0260]

[0261] in

[0262]

[0263]

[0264] μ5 = median [x1, x3, x5, x7, x9]

[0265] …

[0266]

[0267] The overall frame black level can be calculated by accumulating each line offset to calculate the overall black level. In this method, simple exponential smoothing (SES) is used to determine the overall black level. Simple exponential smoothing allows the lines at the end of the frame to have a greater impact on the final black estimate. In some implementations, this may be desirable to address changes in black offset that occur on a sub-frame time scale. The simple exponential smoothing algorithm determines a running estimate that can be incrementally adjusted each time a sample is available. For convenience, the samples can be divided by a binary number (2 q ), which is then added to the previous estimate. The previous estimate can be first multiplied by (2 q -1) / 2 q to normalize the results. High values ​​of q lead to greater statistical accuracy over time in stable scenarios. Lower values ​​of q can make the correction more reactive to rapid changes. q should be available as a tunable parameter. This can be calculated as follows:

[0268] k r =L r (r=0)

[0269]

[0270] where k r is the black level estimate after row r, and L r It may be the line offset estimate for row r. The decision about what to do with the black clamp digital to analog converter may be made after the final row in the array has been added.

[0271] The black clamp algorithm at 1606, 1607 may require a target black level, and this may be provided by an adjustable parameter. The black clamp DAC on the sensor of the channel in question will be pushed up or down depending on whether the observed black estimate is above or below the target. The size of the push may be the smallest unit, i.e., one DAC count, providing that the black excursion may be closer to the target. In the case where the black level may be far from the target, a larger proportional push may be made. The algorithm will need to know a rough calibration of the correspondence between the black clamp DAC counts and the sensor ADC counts, as well as the directionality of the DAC adjustment relative to the output black level.

[0272] Line noise is corrected at 1608 and 1609. Line noise refers to random temporal variations in the offset of horizontal pixel rows. There may be multiple sources, but it may be considered reset noise due to analog components being reset each time a pixel row is read out. It may be temporal, and a new correction should be calculated for each new line of each frame. Since the amplifier stage at the analog to digital converter input may be the final analog component, there may be good reason to suspect that line noise appears to be phenomenologically independent for each analog to digital converter channel. One approach is to correct each analog to digital converter channel individually.

[0273] Completely eliminating line noise can be challenging because the samples of optically black pixels used for line offset estimation may be separated from the samples to which the correction can be applied. The sample statistics are finite. Assuming the noise is Gaussian, the post-correction line noise can be approximated to be equal to the uncertainty in the line offset estimate caused by the pixel temporal noise present in the optically black pixels:

[0274]

[0275] where σ L，post is the post-corrected timeline noise, σ P is the optical black pixel temporal noise, and N OB is the number of optically black pixels. Line noise correction also introduces a spatial line noise component, which is mainly the result of pixel fixed pattern noise present within optically black pixels:

[0276]

[0277] This artifact can be removed by fixed pattern noise correction later in the chain.Simulations have indicated that in order for the timeline noise to be invisible, the magnitude should be less than about 1 / 10 of the pixel temporal noise.

[0278] Line noise correction is applied to optically viewed (transparent) pixels:

[0279] x′ i =x i -L+B

[0280] Where L is the line offset estimate for the current line, transmitted from the "black clamp" module, and B is the black clamp target level.

[0281] Full line reassembly at 1610 includes combining two data channels, namely the top data channel and the bottom data channel, into a full line. The top data channel and the bottom data channel need to be interleaved in such a way that the final transparent pixel order reflects the correct order in the array.

[0282] Fixed pattern noise correction at 1614 includes adjusting the running offset estimate on a per physical pixel basis. CMOS image sensors have multiple noise sources. The magnitude and appearance of the noise depends on a range of physical conditions. Pure Poisson or Gaussian temporal noise without coherent components (e.g., photon scattering noise or source follower 1 / f read noise) looks natural, just as image noise can look natural. All other perceptible noise types can degrade image quality to a greater extent of the same magnitude. Spatial noise (fixed pattern noise) can be particularly destructive, and CMOS sensors inherently have at least two sources. CMOS image sensors experience pixel fixed pattern noise and column fixed pattern noise. Pixel fixed pattern noise can be primarily due to variations in photodiode leakage current (dark signal) on a pixel-by-pixel (DSNU) basis. This source can be exponentially dependent on junction temperature (T J ) and is linearly dependent on exposure time. Column fixed pattern noise can be a result of a readout architecture where pixels from within the same column are directed through a common analog readout element.

[0283] Typically, on-chip digital fixed pattern noise correction involves processing only the column fixed pattern noise component and requires an offset correction register for each column. The accuracy of this correction can typically be around 20 bits per column, which translates to about 5kB of RAM for a 1920×1080 array. One of the benefits of migrating the digital sensor correction to the image signal processor can be the ready availability of RAM. This opens up the possibility of comprehensive fixed pattern noise correction, canceling any row, column or pixel-by-pixel component. This can be achieved with the help of simple exponential smoothing (SES), where each new dark frame sample can be used to adjust the running offset estimate on a per physical pixel basis.

[0284] Programmable digital gain 1618 may be performed by a programmable digital amplifier. CMOS image sensors are typically equipped with digitally programmable gain stages that have very fine increments. This can facilitate an automatic exposure process that typically adjusts gain and exposure time. A programmable digital amplifier can be used to align the range of the sensor analog-to-digital converter with the range of the image signal processor (e.g., ×2 for an 11-bit analog-to-digital converter to a 12-bit image signal processor). A small amount of digital gain can also be used to trim the footprint of digital line noise and fixed pattern noise correction, which becomes apparent over the full range of the analog-to-digital converter.

[0285] A more space conservative approach involves combining a large amount of control RAM into a single long register. In the extreme case, all parameters can be placed in a single register without the need for an address ROM. This solution may not be very practical because writing to the control registers takes time and typical video applications involve changing a small number of operating parameters on a frame-by-frame basis. The most practical solution can be provided by connecting sets of functionally related parameters into a small number of long registers. The spatial difference implied by having ten registers (requiring 4 address bits) relative to one register is negligible. Specifically, it means that all parameters that are written periodically at a high rate are grouped together in an exclusive register (frame register) in order to keep the time required to write them to a minimum value. Such parameters include exposure time, gain, incremental offset adjustment, and any other parameters necessary to maintain continuous high quality video. If the digital data path logic has been migrated from the chip as described above, the black clamp voltage adjustment data is also grouped in such a register because it should also be modified every frame. In a specific implementation, the register can be written during this configuration phase, and therefore the timing of the frame register write relative to the overall frame timing should be carefully controlled by the camera.

[0286] Other examples of parameter register groups may include: analog current, analog voltage, pixel timing, vertical timing, sensor command (reset, etc.), etc. Fig.15 , the register arrangement can be shown for a specific minimum area sensor design. The "command" register can be used for the highest level of event-oriented 1-bit commands, such as chip reset and the load of other registers shown below it. The 2-wire protocol address decoder determines which shift register leads the incoming 2-wire protocol data forward. In order to load the "format" register, for example, the external controller sends a command with an address associated with the format register. This places the data stream in the format register shift register. Then, in order to latch the data, a follow-up command can be sent to the command register with a specific "load format" bit set. It should be understood that multiple control registers can be used. The control register can be a digital latch that can be loaded via a shift register. The length of the shift register can be arbitrary. In one embodiment, most of the multiple control registers can be loaded using a shift register including tens of bits. In one embodiment, most of the multiple control registers can be loaded using a shift register including hundreds of bits. In one embodiment, most of the multiple control registers can be loaded using a shift register including thousands of bits. In one embodiment, a serial, 2-wire protocol can be used to load the shift register. In one implementation, one of the shift registers may be dedicated to frame-by-frame parameter changes, such as integration time and black clamp offset adjustment.

[0287] Fig.17 It is a schematic diagram of the pattern reconstruction process. Fig.17The exemplary pattern shown includes red pulses, green pulses, blue pulses, and special pulses of light each lasting a duration of T1. The special pulses may include one or more pulses for generating one or more special exposure frames, such as hyperspectral exposure frames, fluorescence exposure frames, and / or topological or laser mapping exposure frames. In various embodiments, the light pulses may have the same duration or different durations. The red exposure frame, green exposure frame, blue, and special exposure frames are combined to generate an RGB image with laser mapping data stacked thereon. A single image frame including a red exposure frame, a green exposure frame, a blue exposure frame, and a special exposure frame requires a time period of 4*T1 to generate. Fig.17 The durations shown are exemplary only and may vary for different specific implementations. In other embodiments, different pulse schemes may be used. For example, an embodiment may be based on the timing of each color component or frame (T1), and the reconstructed frame has a period that is twice the period of the input color frame (2×T1). Different frames within a sequence may have different frame periods, and the average capture rate may be any multiple of the final frame rate.

[0288] In one embodiment, the dynamic range of the system is increased by changing the pixel sensitivity of pixels within the pixel array of the image sensor. Some pixels may sense reflected electromagnetic radiation at a first sensitivity level, other pixels may sense reflected electromagnetic radiation at a second sensitivity level, and so on. Different pixel sensitivities may be combined to increase the dynamic range provided by the pixel configuration of the image sensor. In one embodiment, adjacent pixels are set with different sensitivities so that each cycle includes data generated by pixels that are more sensitive and less sensitive relative to each other. When multiple sensitivities are recorded in a single cycle of the pixel array, the dynamic range increases. In one embodiment, a wide dynamic range can be achieved by having multiple global TXs, each TX firing only on different groups of pixels. For example, in global mode, the global TX1 signal is firing pixel group 1, the global TX2 signal is exciting pixel group 2, the global TXn signal is exciting pixel group n, and so on.

[0289] 18A to 18CEach light source 1800 is shown having multiple emitters. The emitters include a first emitter 1802, a second reflector 1804, and a third emitter 1806. Additional emitters may be included, as discussed further below. The emitters 1802, 1804, and 1806 may include one or more laser generators that emit light having different wavelengths. For example, the first emitter 1802 may emit a wavelength consistent with a blue laser, the second emitter 1804 may emit a wavelength consistent with a green laser, and the third emitter 1806 may emit a wavelength consistent with a red laser. For example, the first emitter 1802 may include one or more blue lasers, the second emitter 1804 may include one or more green lasers, and the third emitter 1806 may include one or more red lasers. The lasers 1802, 1804, 1806 emit laser beams toward a collection area 1808, which may be a waveguide, a lens, or a device for collecting light and / or transmitting light to a waveguide (such as Figure 2 The jumper waveguide 206 or the intracavity waveguide 210) provides a location for other optical components of the light.

[0290] In one specific implementation, the emitters 1802, 1804, and 1806 emit electromagnetic radiation at hyperspectral wavelengths. Certain hyperspectral wavelengths can penetrate tissue and enable a medical practitioner to "see through" tissue in the foreground to identify chemical processes, structures, compounds, biological processes, etc., behind the tissue in the foreground. Hyperspectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc., that are known to have certain spectral responses.

[0291] In a specific implementation where a reagent or dye has been administered to a patient that aids in identifying certain tissues, structures, chemical reactions, biological processes, etc., the emitters 1802, 1804, and 1806 may emit wavelengths for causing the reagent or dye to fluoresce. Such wavelengths may be determined based on the reagent or dye administered to the patient. In such embodiments, the emitters may need to be highly accurate in order to emit the desired wavelengths to cause certain reagents or dyes to fluoresce or activate.

[0292] In one specific implementation, the transmitters 1802, 1804, and 1806 emit laser mapping patterns for mapping the topology of the scene and / or for calculating the size and distance between objects in the scene. In one embodiment, the endoscopic imaging system is used in conjunction with multiple tools (such as scalpels, retractors, clamps, etc.). In such embodiments, each of the transmitters 1802, 1804, and 1806 can emit a laser mapping pattern so that the laser mapping pattern is projected onto each tool individually. In such embodiments, the laser mapping data for each of the tools can be analyzed to identify the distance between the tool and other objects in the scene.

[0293] exist Fig.18BIn an embodiment, emitters 1802, 1804, 1806 each deliver laser light to collection area 1808 at different angles. Changes in angle can result in changes in the location of the electromagnetic energy in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at collection area 1808, the changing angles can cause different amounts of light to enter different fibers. For example, the angles can result in intensity variations across collection area 1808. In addition, light from different emitters may not be mixed evenly, so some fibers may receive different amounts of different colors of light. Variations in the color or intensity of light in different fibers can result in non-optimal illumination of the scene. For example, variations in the delivered light or light intensity can result in both the scene and the captured image.

[0294] In one embodiment, an intermediate optical element may be placed between the fiber bundle and the transmitters 1802, 1804, 1806 to mix light of different colors (wavelengths) before entering the fiber or other waveguide. Exemplary intermediate optical elements include diffusers, mixing rods, one or more lenses, or other optical components for mixing light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. This mixing may result in the same color in each fiber, but, in some embodiments, may still result in different total brightness delivered to different fibers. In one embodiment, the intermediate optical element may also spread or evenly spread the light over the collection area so that each fiber carries the same total amount of light (e.g., the light may be spread out in a top hat profile). Diffusers or mixing rods may cause light loss.

[0295] Although the collection area 1808 Fig.18A 1808 as a physical component, but collection region 1808 may simply be the region where light from emitters 1802, 1804, and 1806 is delivered. In some cases, collection region 1808 may include optical components such as diffusers, mixing rods, lenses, or any other intervening optical components between emitters 1802, 1804, 1806 and the output waveguides.

[0296] Fig. 18CAn embodiment of a light source 1800 is shown having emitters 1802, 1804, 1806 that provide light to a collection area 1808 at the same or substantially the same angle. The light is provided at an angle that is approximately perpendicular to the collection area 1808. The light source 1800 includes a plurality of dichroic mirrors, including a first dichroic mirror 1810, a second dichroic mirror 1812, and a third dichroic mirror 1814. The dichroic mirrors 1810, 1812, 1814 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 1814 may reflect blue laser light provided by a third emitter, while being transparent to red and green light provided by each of the first and second emitters 1802, 1804. The second dichroic mirror 1812 may be transparent to light from the first emitter 1802, but reflective to light from the second emitter 1804. If other colors or wavelengths are included, the dichroic mirrors may be selected to reflect light corresponding to at least one emitter and be transparent to other emitters. For example, the third dichroic mirror 1814 reflects light from the third emitter 1806, but is transparent to emitters "behind" it, such as the first emitter 1802 and the second emitter 1804. In embodiments where there are dozens or hundreds of emitters, each dichroic mirror may reflect the corresponding emitter and the emitter in front of it, while being transparent to the emitter behind it. This may allow dozens or hundreds of emitters to emit electromagnetic energy to the collection area 1808 at substantially the same angle.

[0297] Because these dichroic mirrors allow other wavelengths to be transmitted or passed through, each of these wavelengths can arrive at the collection area 1808 from the same angle and / or with the same center point or focus. Providing light from the same angle and / or the same focus / center point can significantly improve reception and color mixing at the collection area 1808. For example, a particular fiber can receive different colors in the same proportions as they are transmitted / reflected by the emitters 1802, 1804, 1806 and mirrors 1810, 1812, 1814. Fig.18B Compared to the embodiment of the present invention, the light mixing can be significantly improved at the collection area. In one embodiment, any optical component discussed herein can be used at the collection area 1808 to collect light before providing it to the fiber or fiber bundle.

[0298] Fig. 18CAn embodiment of a light source 1800 having emitters 1802, 1804, 1806 that also provide light to a collection area 1808 at the same or substantially the same angle is shown. For example, the light incident on the collection area 1808 is offset from the vertical. Angle 1816 indicates the angle of offset from the vertical. In one embodiment, the laser emitters 1802, 1804, 1806 may have a Gaussian cross-sectional intensity profile. As previously described, the improved distribution of light energy between fibers can be achieved by forming a flatter or top-hat-shaped intensity profile. In one embodiment, as the angle 1816 increases, the intensity across the collection area 1808 approaches a top-hat profile. For example, by increasing the angle 1816 until the profile is sufficiently flat, the top-hat profile can even approximate a non-flat output beam. The top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intermediate optical components between the emitters 1802, 1804, 1806 and the output waveguide, fiber or fiber bundle.

[0299] Fig.19 1 is a schematic diagram showing a single optical fiber 1902 outputted via a diffuser 1904 at the output. In one embodiment, the optical fiber 1902 has a diameter of 500 microns, a numerical aperture of 0.65, and emits a light cone 1906 of about 70 or 80 degrees without the diffuser 1904. With the diffuser 1904, the light cone 1906 may have an angle from about 110 or 120 degrees. The light cone 1906 may be a large portion of where all the light reaches and is evenly distributed. The diffuser 1904 may allow for a more even distribution of the electromagnetic energy of the scene observed by the image sensor.

[0300] In one embodiment, the intracavity waveguide 210 comprises a single plastic or glass optical fiber of about 500 microns. Plastic fibers are relatively low in cost and their width may allow the fiber to carry a sufficient amount of light to the scene, but with coupling, diffusion or other losses. For example, a smaller fiber may not carry as much light or power as a larger fiber. The intracavity waveguide 210 may comprise a single or multiple optical fibers. The intracavity waveguide 210 may receive light directly from a light source or via a jumper waveguide. A diffuser may be used to widen the light output 206 to obtain a desired field of view for an image sensor 214 or other optical component.

[0301] Although in 18A to 18CThree emitters are shown, but in some embodiments, a number ranging from one to hundreds or more emitters may be used. The emitters may have different wavelengths or spectra of light that they emit, and these lights may be used to continuously cover the desired portion of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra). The emitters may be configured to be able to emit visible light such as red, green, and blue light, and may also be configured to be able to emit hyperspectral emissions of electromagnetic radiation, fluorescence excitation wavelengths for causing reagents to fluoresce, and / or laser mapping modes for calculating parameters and distances between objects in the scene.

[0302] Fig. 20 A portion of the electromagnetic spectrum 2000 is shown divided into twenty different sub-spectra. The number of sub-spectra is exemplary only. In at least one embodiment, the spectrum 2000 can be divided into hundreds of sub-spectra, each with a small band. The spectrum can extend from the infrared spectrum 2002, through the visible spectrum 2004, and into the ultraviolet spectrum 2006. The sub-spectra each have a band 2008 covering a portion of the spectrum 2000. Each band can be defined by an upper wavelength and a lower wavelength.

[0303] Hyperspectral imaging includes imaging information from across the electromagnetic spectrum 2000. A hyperspectral pulse of electromagnetic radiation may include multiple sub-pulses spanning one or more portions of the electromagnetic spectrum 2000 or the entire electromagnetic spectrum 2000. A hyperspectral pulse of electromagnetic radiation may include a single wavelength partition of the electromagnetic radiation. The resulting hyperspectral exposure frame includes information sensed by the pixel array after the hyperspectral pulse of electromagnetic radiation. Thus, the hyperspectral exposure frame may include data for any suitable partition of the electromagnetic spectrum 2000, and may include multiple exposure frames for multiple partitions of the electromagnetic spectrum 2000. In one embodiment, the hyperspectral exposure frame includes multiple hyperspectral exposure frames, such that the combined hyperspectral exposure frame includes data for the entire electromagnetic spectrum 2000.

[0304] In one embodiment, for each sub-spectrum, at least one emitter (such as a laser emitter) is included in a light source (such as light source 202, 1800) to provide complete and continuous coverage of the entire spectrum 2000. For example, a light source for providing coverage of the sub-spectrums shown may include at least 20 different emitters, at least one emitter for each sub-spectrum. In one embodiment, each emitter covers a spectrum covering 40 nanometers. For example, one emitter may emit light in a band of 500nm to 540nm, while another emitter may emit light in a band of 540nm to 580nm. In another embodiment, the emitters may cover bands of other sizes, depending on the type of emitters available or the imaging needs. For example, a plurality of emitters may include a first emitter covering a band of 500nm to 540nm, a second emitter covering a band of 540nm to 640nm, and a third emitter covering a band of 640nm to 650nm. Each emitter may cover a different segment of the electromagnetic spectrum ranging from far infrared, mid-infrared, near infrared, visible light, near ultraviolet, and / or far ultraviolet. In some cases, multiple emitters of the same type or wavelength may be included to provide sufficient output power for imaging. The number of emitters required for a particular wavelength band may depend on the sensitivity of the monochromatic sensor to the wavelength band and / or the power output capability of the emitters in that wavelength band.

[0305] The bandwidth and coverage provided by the emitter can be selected to provide any desired combination of spectra. For example, continuous coverage of spectra using very small bandwidths (e.g., 10 nm or less) can allow highly selective hyperspectral imaging and / or fluorescence imaging. The bandwidth can allow selective emission of excitation wavelengths for one or more specific fluorescent agents. In addition, the bandwidth can allow selective emission of certain partitions of hyperspectral electromagnetic radiation for identification of specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths come from selectively activated emitters, extreme flexibility in causing one or more specific fluorescent agents to fluoresce during an inspection can be achieved. In addition, extreme flexibility in identifying one or more objects or processes through hyperspectral imaging can be achieved. Therefore, more fluorescence information and / or hyperspectral information can be achieved in less time and within a single inspection, which would otherwise require multiple inspections, delays due to the application of dyes or stains, etc.

[0306] Fig.212100 is a schematic diagram showing a timing diagram 2100 for emission and readout for generating an image. The solid line represents the readout (peak 2102) and blanking period (valley) for capturing a series of exposure frames 2104 to 2114. A series of exposure frames 2104 to 2114 may include a series of repeated exposure frames that can be used to generate laser mapping, hyperspectral and / or fluorescence data that can be stacked on an RGB video stream. In one embodiment, a single image frame includes information from multiple exposure frames, wherein one exposure frame includes red image data, another exposure frame includes green image data, and another exposure frame includes blue image data. In addition, a single image frame may include one or more of hyperspectral image data, fluorescence image data, and laser mapping data. Multiple exposure frames are combined to produce a single image frame. A single image frame is an RGB image with hyperspectral imaging data. The exposure frame series includes a first exposure frame 2104, a second exposure frame 2106, a third exposure frame 2108, a fourth exposure frame 2110, a fifth exposure frame 2112, and an Nth exposure frame 2126.

[0307] In addition, hyperspectral image data, fluorescent image data, and laser mapping data can be used in combination to identify key tissues or structures and further measure the size of those key tissues or structures. For example, hyperspectral image data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, cancerous tissues, etc. The location and identification of key structures can be received from a corresponding system and can also be used to generate the topology of key structures using laser mapping data. For example, a corresponding system determines the location of a cancerous tumor based on hyperspectral imaging data. Since the location of the cancerous tumor is known based on the hyperspectral imaging data, the topology and distance of the cancerous tumor can then be calculated based on the laser mapping data. This example is also applicable when a cancerous tumor or other structure is identified based on fluorescent imaging data.

[0308] In one embodiment, each exposure frame is generated based on at least one pulse of electromagnetic energy. The electromagnetic energy pulses are reflected and detected by the image sensor and then read out in a subsequent readout (2102). Thus, each blanking period and readout results in an image frame for a particular spectrum of electromagnetic energy. For example, a first exposure frame 2104 may be generated based on the spectrum of the first one or more pulses 2116, a second exposure frame 2106 may be generated based on the spectrum of the second one or more pulses 2118, a third exposure frame 2108 may be generated based on the spectrum of the third one or more pulses 2120, a fourth exposure frame 2110 may be generated based on the spectrum of the fourth one or more pulses 2122, a fifth exposure frame 2112 may be generated based on the spectrum of the fifth one or more pulses 2124, and an Nth exposure frame 2126 may be generated based on the spectrum of the Nth one or more pulses 2126.

[0309] Pulses 2116 to 2126 may include energy from a single emitter or from a combination of two or more emitters. For example, a spectrum included in a single readout cycle or within multiple exposure frames 2104 to 2114 may be selected for the desired inspection or detection of a specific tissue or condition. According to one embodiment, one or more pulses may include visible spectrum light for generating RGB or black and white images, while one or more additional pulses are emitted to sense the spectral response of electromagnetic radiation for fluorescent wavelengths. For example, pulse 2116 may include red light, pulse 2118 may include blue light, and pulse 2120 may include green light, while the remaining pulses 2122 to 2126 may include wavelengths and spectra for detecting specific tissue types, making reagents fluoresce, and / or marking the topology of the scene. For another example, the pulses of a single readout cycle include spectra (e.g., different fragments of the electromagnetic spectrum) generated by multiple different emitters that can be used to detect specific tissue types. For example, if a combination of wavelengths causes a pixel to have a value above or below a threshold, the pixel can be classified as corresponding to a specific type of tissue. Each frame can be used to further narrow down the type of tissue present at that pixel (e.g., and every pixel in the image) to provide a very specific classification of the tissue and / or the state of the tissue (disease / healthy) based on the spectral response of the tissue and / or the presence or absence of fluorescent agents in the tissue.

[0310] Multiple frames 2104 to 2114 are shown with readout periods of varying lengths and with pulses of varying lengths or intensities. Blanking periods, pulse lengths or intensities, etc. may be selected based on the sensitivity of the monochromatic sensor to a particular wavelength, the power output capability of the transmitter, and / or the carrying capacity of the waveguide.

[0311] In one embodiment, dual image sensors may be used to obtain a three-dimensional image or video feed.Three-dimensional examination may allow for an improved understanding of the three-dimensional structure of the examination region and mapping of different tissue or material types within the region.

[0312] In an exemplary implementation, a fluorescent agent is provided to a patient, and the fluorescent agent is configured to be able to attach to cancer cells. It is known that the fluorescent agent fluoresces when irradiated by electromagnetic radiation of a specific partition. The relaxation wavelength of the fluorescent agent is also known. In this exemplary implementation, an endoscopic imaging system is used to image the patient as described herein. The endoscopic imaging system pulses the partitions of the red wavelength, green wavelength, and blue wavelength of light to generate an RGB video stream inside the patient's body. In addition, the endoscopic imaging system pulses electromagnetic radiation of the excitation wavelength for the fluorescent agent administered to the patient. In the example, the patient has cancer cells and the fluorescent agent has adhered to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength for the fluorescent agent, the fluorescent agent will fluoresce and emit the relaxation wavelength. If cancer cells are present in the scene imaged by the endoscopic imaging system, the fluorescent agent will also be present in the scene and emit its relaxation wavelength after fluorescing due to the emission of the excitation wavelength. The endoscopic imaging system senses the relaxation wavelength of the fluorescent agent, and thereby senses the presence of the fluorescent agent in the scene. Because it is known that the fluorescent agent is attached to cancer cells, the presence of the fluorescent agent further indicates the presence of cancer cells in the scene. The endoscopic imaging system thereby identifies the location of cancer cells within the scene. The endoscopic imaging system may also emit a laser mapping pulse scheme for generating a topology of the scene and calculating the size of objects within the scene. The location of the cancer cells (as identified by the fluorescence imaging data) may be combined with the topological and size information calculated based on the laser mapping data. Thus, the precise location, size, dimensions, and topology of the cancer cells may be identified. This information may be provided to a medical practitioner to aid in the removal of the cancer cells. Additionally, this information may be provided to a robotic surgical system to enable the surgical system to remove the cancer cells.

[0313] In another exemplary implementation, an endoscopic imaging system is used to image a patient to identify quantitative diagnostic information about the patient's tissue pathology. In an example, the patient is suspected or known to have a disease that can be tracked using hyperspectral imaging to observe the progression of the disease in the patient's tissue. The endoscopic imaging system pulses red, green, and blue wavelengths of light to generate an RGB video stream inside the patient's body. In addition, the endoscopic imaging system pulses one or more hyperspectral wavelengths of light that allow the system to "see through" some tissues and generate imaging of tissues affected by the disease. The endoscopic imaging system senses reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of diseased tissues, and thereby identifies the location of the diseased tissue in the patient's body. The endoscopic imaging system can also emit a laser mapping pulse scheme for generating a topology of a scene and calculating the size of an object within the scene. The location of the diseased tissue (as identified by the hyperspectral imaging data) can be combined with the topological and dimensional information calculated using the laser mapping data. Therefore, the precise location, size, size, and topology of the diseased tissue can be identified. This information can be provided to a practicing physician to help remove, image, or study the diseased tissue. Additionally, this information may be provided to a robotic surgical system to enable the surgical system to resect diseased tissue.

[0314] Fig. 22 2200. System 2200 includes an endoscope 2206 or other suitable imaging device with a light source 2208 for use in a light-deficient environment. Endoscope 2206 includes an image sensor 2204 and a filter 2202 for filtering out unwanted wavelengths of light or other electromagnetic radiation before reaching image sensor 2204. Light source 2208 transmits light that can illuminate surface 2212 in a light-deficient environment, such as a body cavity. Light 2210 reflects from surface 2212 and passes through filter 2202 before striking image sensor 2204.

[0315] Filter 2202 can be used in specific implementations where a fluorescent agent or dye has been applied. In such embodiments, light source 2208 emits an excitation wavelength for causing the fluorescent agent or dye to fluoresce. Typically, the relaxation wavelength emitted by the fluorescent agent or dye will have a different wavelength than the excitation wavelength. Filter 2202 can be selected to filter out the excitation wavelength and only allow the relaxation wavelength to pass through the filter and be sensed by image sensor 2204.

[0316] In one embodiment, the filter 2202 is configured to filter out the excitation wavelength of electromagnetic radiation that causes the reagent or dye to fluoresce, so that only the expected relaxation wavelength of the fluorescing reagent or dye is allowed to pass through the filter 2202 and reach the image sensor 2204. In one embodiment, the filter 2202 filters out the fluorescent reagent excitation wavelength at least between 770nm and 790nm. In one embodiment, the filter 2202 filters out the fluorescent reagent excitation wavelength at least between 795nm and 815nm. In one embodiment, the filter 2202 filters out the fluorescent reagent excitation wavelength at least between 770nm and 790nm and between 795nm and 815nm. In these embodiments, the filter 2202 filters out the excitation wavelength of the reagent and allows only the relaxation wavelength of the fluorescent reagent to be read by the image sensor 2204. The image sensor 2204 can be a wavelength-indeterminate image sensor, and the filter 2202 can be configured to allow the image sensor 2204 to receive only the relaxation wavelength of the fluorescent reagent and not the emission excitation wavelength of the reagent. The data determined by the image sensor 2204 may then indicate the presence of key body structures, tissues, biological processes, or chemical processes as determined by the location of the agent or dye.

[0317] The filter 2202 may also be used in implementations where fluorescent agents or dyes have not been applied. The filter 2202 may be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read by the image sensor 2204. The image sensor 2204 may be a monochrome image sensor, so that pixels of the captured image that are above a threshold or below a threshold may be characterized as corresponding to certain spectral responses or fluorescent emissions. The spectral response or fluorescent emission determined by the pixels captured by the image sensor 2204 may indicate the presence of certain body tissues or structures, certain conditions, certain chemical processes, etc.

[0318] Fig.23 2300 is a schematic diagram of an imaging system 2300 having multiple cut-off filters. System 2300 includes an endoscope 2306 or other suitable imaging device having a light source 2308 for a light-deficient environment. Endoscope 2306 includes an image sensor 2304 and two filters 2302a, 2302b. It should be understood that in alternative embodiments, system 2300 may include any number of filters, and the number of filters and the type of filters may be selected for certain purposes, for example, for collecting imaging information of specific body tissues, body conditions, chemical processes, etc. Filters 2302a, 2302b are configured to prevent image sensor 2304 from sensing unwanted wavelengths of light or other electromagnetic radiation. Filters 2302a, 2302b may be configured to filter out unwanted wavelengths from white light or other electromagnetic radiation that may be emitted by light source 2308.

[0319] Relative to Fig. 22 The present disclosure is also described, and filters 2302a, 2302b can be used in specific implementations where fluorescent reagents or dyes have been applied. Filters 2302a, 2302b can be configured to block the emission excitation wavelengths of the reagents or dyes and allow the image sensor 2304 to read only the relaxation wavelengths of the reagents or dyes. In addition, filters 2302a, 2302b can be used in specific implementations where fluorescent reagents or dyes have not been applied. In such specific implementations, filters 2302a, 2302b can be selected to allow wavelengths corresponding to the desired spectral response to pass through the image sensor 2304 and be read by the image sensor.

[0320] Multiple filters 2302a, 2302b can each be configured to filter out different ranges of wavelengths of the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than a desired wavelength range, and an additional filter can be configured to filter out wavelengths shorter than a desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 2304.

[0321] In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 513nm and 545nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 565nm and 585nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 900nm and 1000nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 425nm and 475nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 520nm and 545nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 625nm and 645nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 760nm and 795nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 795nm and 815nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 370nm and 420nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are tailored to allow electromagnetic radiation between 600nm and 670nm to contact the image sensor 2304. In one embodiment, the filters 2302a, 2302b are configured to allow only certain fluorescent relaxation emissions to pass through the filters 2302a, 2302b and contact the image sensor 2304.

[0322] In one embodiment, system 2300 includes multiple image sensors 2304, and may specifically include two image sensors for generating a three-dimensional image. Image sensors 2304 may be color / wavelength agnostic and configured to read electromagnetic radiation of any wavelength reflected from surface 2312. In one embodiment, image sensors 2304 are each color-dependent or wavelength-dependent and configured to read electromagnetic radiation of a specific wavelength reflected from surface 2312 and returned to image sensor 2304. Alternatively, image sensor 2304 may include a single image sensor having multiple different pixel sensors configured to read light of different wavelengths or colors, such as a Bayer filter color filter array. Alternatively, image sensor 2304 may include one or more color-agnostic image sensors that may be configured to read electromagnetic radiation of any wavelength reflected from surface 2312 according to a pulse schedule such as Figures 5 to 7E Those shown in ) read electromagnetic radiation of different wavelengths.

[0323] Fig.24 24 is a schematic diagram showing a system 2400 for mapping a surface and / or tracking an object by laser mapping imaging in an environment lacking light. In one embodiment, an endoscope 2406 pulses a grid array 2406 (which may be referred to as a laser mapping pattern) onto a surface 2404 in an environment lacking light. Fig.24 In one embodiment shown in FIG. 2 , the grid array 2406 includes vertical hashes 2408 and horizontal hashes 2410. It should be understood that the grid array 2406 may include any suitable array for mapping the surface 2404, including, for example, a raster grid of discrete points, an occupancy grid map, a dot array, etc. In addition, the endoscope 2406 may pulse multiple grid arrays 2406, and may pulse one or more separate grid arrays on each of multiple objects or structures, for example, in an environment lacking light.

[0324] In one embodiment, the system 2400 pulses a grid array 2406 that can be used to map the three-dimensional topology of a surface and / or track the position of an object such as a tool or another device in a light-deficient environment. In one embodiment, the system 2400 provides data to a third-party system or computer algorithm for determining surface size and configuration by way of light detection and ranging (LIDAR) mapping. The system 2400 can pulse any suitable wavelength of light or electromagnetic radiation in the grid array 2406, including, for example, ultraviolet light, visible light, light, and / or infrared light or near infrared light. Surfaces 2404 and / or objects within an environment can be mapped and tracked at very high resolution and with very high accuracy and precision.

[0325] In one embodiment, system 2400 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. System 2400 may include a light engine having an emitter and an inner cavity, the emitter generating one or more electromagnetic radiation pulses, the inner cavity transmitting the one or more electromagnetic radiation pulses to a distal end of an endoscope in a light-deficient environment such as a body cavity. In one embodiment, at least a portion of the one or more electromagnetic radiation pulses includes a laser mapping pattern emitted onto a surface in the light-deficient environment (such as, the surface of body tissue in the body cavity and / or the surface of a tool or other device). Endoscope 2406 may include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations in the light-deficient environment.

[0326] In one embodiment, the system 2400 includes a processor for determining the distance between the endoscope or tool and an object, such as a surface 2404. The processor may also determine the angle between the endoscope or tool and the object. The processor may also determine surface area information about the object, including, for example, the size of the surgical tool, the size of the structure, the size of the anatomical structure, position information, and other position data and measurements. The system 2400 may include one or more image sensors that provide image data output to a control system for determining the distance between the endoscope or tool and an object, such as a surface 2404. The image sensor may output information to the control system for determining the angle between the endoscope or tool and the object. In addition, the image sensor may output information to the control system for determining surface area information about the object, the size of the surgical tool, the size of the structure, the size of the anatomical structure, position information, and other position data and measurements.

[0327] In one embodiment, the transmitter of endoscope 2406 pulses grid array 2406 at a sufficient speed so that grid array 2406 is not visible to the user. In various specific implementations, seeing grid array 2406 during endoscopic imaging procedures and / or endoscopic surgery may distract the user. Grid array 2406 may be pulsed at a sufficiently short period so that grid array 2406 cannot be detected by the human eye. In an alternative embodiment, endoscope 2406 pulses grid array 2406 at a sufficient repetition frequency so that grid array 2406 can be viewed by the user. In such embodiments, grid array 2406 may be superimposed on the image of surface 2404 on the display. Grid array 2406 may be superimposed on a black and white or RGB image of surface 2404 so that grid array 2406 may be visible to the user during use of system 2400. A user of system 2400 may indicate whether grid array 2406 should be superimposed on the image of surface 2404 and / or whether grid array 2406 should be visible to the user. System 2400 may include a display that provides real-time measurement of the distance from endoscope 2406 to surface 2404 or another object in the light-deficient environment. The display may also provide real-time surface area information about surface 2404 and / or any object, structure, or tool in the light-deficient environment. The accuracy of the measurement may be accurate to less than one millimeter.

[0328] In one embodiment, the system 2400 pulses a plurality of grid arrays 2406. In one embodiment, each of the plurality of grid arrays 2406 corresponds to a tool or other device present within the light-deficient environment. The precise position and parameters of each of the tools and other devices can be tracked by pulsing and sensing the plurality of grid arrays 2406. The information generated by sensing the reflective grid arrays 2406 can be evaluated to identify the relative positions of the tools and other devices within the light-deficient environment.

[0329] The endoscope 2406 may pulse electromagnetic radiation according to a pulse schedule such as those shown herein, which may also include pulsing the grid array 2406 and pulsing red, green, and blue light for generating an RGB image and further generating a grid array 2406 that may be overlaid on the RGB image and / or used to map and track surfaces 2404 and objects in an aphotic environment. The grid array 2406 may additionally be pulsed in conjunction with electromagnetic radiation at hyperspectral or fluorescence excitation wavelengths. Data from each of the RGB imaging, laser mapping imaging, hyperspectral imaging, and fluorescence imaging may be combined to identify the location, size, and surface topology of key structures within the body.

[0330] In one embodiment, the endoscope 2406 includes one or more color-agnostic image sensors. In one embodiment, the endoscope 2406 includes two color-agnostic image sensors for generating a three-dimensional image or map of an aphotic environment. The image sensor can generate an RGB image of the aphotic environment according to a pulse schedule as disclosed herein. In addition, the image sensor can determine data for mapping the aphotic environment and tracking one or more objects within the aphotic environment based on data determined when pulsing the grid array 2406. In addition, the image sensor can determine spectral or hyperspectral data and fluorescence imaging data according to the pulse schedule, which can be modified by the user to adapt to the specific needs of the imaging procedure. In one embodiment, the pulse schedule includes red, green and blue pulses and pulses of the grid array 2406 and / or pulses for generating hyperspectral image data and / or fluorescence image data. In various specific implementations, the pulse schedule may include any suitable combination of electromagnetic radiation pulses according to the needs of the user. The recurrence frequency of electromagnetic radiation of different wavelengths can be determined based on, for example, the energy of certain pulses, the needs of the user, whether certain data (e.g., hyperspectral data and / or fluorescence imaging data) need to be continuously updated or can be updated less frequently, etc.

[0331] The pulse schedule may be modified in any suitable manner, and certain electromagnetic radiation pulses may be repeated at any suitable frequency, as desired by a user or a computer-implemented program for certain imaging procedures. For example, in embodiments where surface tracking data generated based on grid array 2406 is provided to a computer-implemented program for use, for example, in robotic surgery, grid array 2406 may be pulsed more frequently than where surface tracking data is provided to a user visualizing a scene during an imaging procedure. In such embodiments where surface tracking data is used for robotic surgery, the surface tracking data may need to be updated more frequently or may need to be extremely accurate so that the computer-implemented program can perform the robotic surgery with precision and accuracy.

[0332] In one embodiment, the system 2400 is configured to generate an occupancy grid map including an array of cells divided into a grid. The system 2400 is configured to store height values ​​of each of the corresponding grid cells to determine a surface map of a three-dimensional environment in an illuminant environment.

[0333] Fig.25A and Fig.25BA perspective view and a side view, respectively, of an implementation of a monolithic sensor 2500 having multiple pixel arrays for producing a three-dimensional image in accordance with the teachings and principles of the present disclosure are shown. Such an implementation may be desirable for three-dimensional image capture, where the two pixel arrays 2502 and 2504 may be offset during use. In another implementation, the first pixel array 2502 and the second pixel array 2504 may be dedicated to receiving electromagnetic radiation of a predetermined wavelength range, where the first pixel array is dedicated to electromagnetic radiation of a different wavelength range than the second pixel array.

[0334] Fig.26A and Fig.26B A perspective view and a side view of a specific implementation of an imaging sensor 2600 built on multiple substrates are shown respectively. As shown, multiple pixel columns 2604 forming the pixel array are located on a first substrate 2602 and multiple circuit columns 2608 are located on a second substrate 2606. The figure also shows the electrical connection and communication between a pixel column and its associated or corresponding circuit column. In a specific implementation, the image sensor may have a pixel array separated from all or most of the supporting circuits, while it may be otherwise manufactured so that its pixel array and supporting circuits are on a single, monolithic substrate / chip. The present disclosure may use at least two substrates / chips, which will be stacked together using three-dimensional stacking technology. The first 2602 of the two substrates / chips may be processed using an image CMOS process. The first substrate / chip 2602 may consist of only a pixel array, or may consist of a pixel array surrounded by a limited circuit. The second or subsequent substrate / chip 2606 may be processed using any process, rather than necessarily from an image CMOS process. The second substrate / chip 2606 may be, but is not limited to, a high density digital process for integrating various and multiple functions into a very limited space or area on the substrate / chip, or a mixed mode or analog process for integrating, for example, precise analog functions, or an RF process for implementing wireless capabilities, or a MEMS (micro-electro-mechanical system) for integrating MEMS devices. The image CMOS substrate / chip 2602 may be stacked with the second or subsequent substrate / chip 2606 using any three-dimensional technology. The second substrate / chip 2606 may support a substantial portion or a large portion of the circuitry that would otherwise be implemented in the first image CMOS chip 2602 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increase the overall system area while keeping the pixel array size constant and optimized as much as possible. The electrical connection between the two substrates / chips may be accomplished through interconnects, which may be bonding wires, bumps, and / or TSVs (through silicon vias).

[0335] Fig.27A and Fig.27BA perspective view and a side view of a specific implementation of an imaging sensor 2700 having multiple pixel arrays for generating a three-dimensional image are shown, respectively. The three-dimensional image sensor can be constructed on multiple substrates and can include multiple pixel arrays and other associated circuit systems, wherein multiple pixel columns 2704a forming a first pixel array and multiple pixel columns 2704b forming a second pixel array are located on respective substrates 2702a and 2702b, respectively, and multiple circuit columns 2408a and 2408b are located on a separate substrate 2406. Electrical connections and communications between pixel columns and associated or corresponding circuit columns are also shown.

[0336] Multiple pixel arrays can sense information simultaneously, and information from multiple pixel arrays can be combined to generate a three-dimensional image. In one embodiment, an endoscopic imaging system includes two or more pixel arrays that can be deployed to generate three-dimensional imaging. The endoscopic imaging system may include a transmitter for emitting electromagnetic radiation pulses during a blanking period of the pixel array. The pixel arrays may be synchronized so that optical black pixels are read simultaneously for two or more pixel arrays (i.e., a blanking period occurs). The transmitter may emit electromagnetic radiation pulses for charging each pixel array in two or more pixel arrays. Two or more pixel arrays can read their corresponding charged pixels simultaneously, so that the readout cycles of two or more pixel arrays occur simultaneously or approximately simultaneously. In one embodiment, the endoscopic imaging system includes multiple transmitters, each transmitter is individually synchronized with one or more pixel arrays in a plurality of pixel arrays. Information from multiple pixel arrays can be combined to generate three-dimensional image frames and video streams.

[0337] It should be understood that the teachings and principles of the present disclosure can be used for reusable device platforms, limited-use device platforms, reconfigurable device platforms, or single-use / disposable device platforms without departing from the scope of the present disclosure. It should be understood that in a reusable device platform, the end user is responsible for the cleaning and disinfection of the device. In a limited-use device platform, the device can be used a specified number of times before becoming inoperable. A typical new device has been sterilized before delivery, and if it is to be used for other purposes, it should be cleaned and disinfected by the end user before other uses. In a reconfigurable device platform, a third party can reprocess the device (e.g., clean, package and disinfect) a single-use device for additional use at a lower cost than a new unit. In a single-use / disposable device platform, the device is provided to the operating room in a sterile manner and can only be used once before being discarded.

[0338] Example

[0339] The following examples relate to preferred features of further embodiments:

[0340] Embodiment 1 is a method. The method includes actuating an emitter to emit a plurality of electromagnetic radiation pulses; and sensing reflected electromagnetic radiation generated by the plurality of electromagnetic radiation pulses through a pixel array of an image sensor to generate a plurality of exposure frames. The method includes generating a reference frame for removing fixed pattern noise from the plurality of exposure frames, wherein the reference frame is based on dark frame data captured when the emitter does not emit electromagnetic radiation. The method includes reducing fixed pattern noise in an exposure frame in the plurality of exposure frames by subtracting the reference frame from the exposure frame. The method causes at least a portion of the electromagnetic radiation pulses emitted by the emitter to include one or more of the following: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm; electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm; an electromagnetic radiation excitation wavelength that causes a reagent to fluoresce; or a laser marking mode.

[0341] Embodiment 2 is a method according to embodiment 1, further comprising: stopping the emitter from pulsating electromagnetic radiation; and sensing the dark frame data through the pixel array when the emitter does not emit electromagnetic radiation.

[0342] Embodiment 3 is a method according to any one of embodiments 1 to 2, wherein the multiple electromagnetic radiation pulses are emitted in a pattern of electromagnetic radiation of different wavelengths, and wherein the emitter repeats the pattern of electromagnetic radiation of different wavelengths.

[0343] Embodiment 4 is a method according to any one of embodiments 1 to 3, wherein the pattern of electromagnetic radiation of different wavelengths includes: a zero emission period in which the transmitter does not pulse emit electromagnetic radiation, and wherein the method further includes: sensing the dark frame data through the pixel array during the zero emission period.

[0344] Embodiment 5 is a method according to any one of embodiments 1 to 4, further comprising sensing a plurality of dark exposure frames through the pixel array when the emitter does not emit electromagnetic radiation, wherein each of the plurality of dark exposure frames comprises dark frame data.

[0345] Embodiment 6 is the method according to any one of embodiments 1 to 5, wherein the plurality of dark exposure frames are interspersed within the plurality of exposure frames.

[0346] Embodiment 7 is a method according to any one of Embodiments 1 to 6, wherein generating the reference frame includes: generating multiple reference frames based on the multiple dark exposure frames, and wherein the method also includes: enhancing the accuracy of at least one reference frame among the multiple reference frames by sampling subsequent dark frame data from subsequent dark exposure frames.

[0347] Embodiment 8 is a method according to any one of embodiments 1 to 7, wherein enhancing the accuracy of the at least one reference frame includes: using exponential smoothing to consider the subsequent dark frame data from the subsequent dark exposure frame.

[0348] Embodiment 9 is a method according to any one of embodiments 1 to 8, wherein generating the reference frame comprises: interpolating two or more instances of dark frame data from two or more dark exposure frames captured at different integration times.

[0349] Embodiment 10 is a method according to any one of embodiments 1 to 9, wherein sensing the reflected electromagnetic radiation includes: sensing during a readout period of the pixel array, wherein the readout period is the duration when valid pixels in the pixel array are read.

[0350] Embodiment 11 is a method according to any one of embodiments 1 to 10, wherein actuating the transmitter comprises: actuating the transmitter to emit multiple sub-pulses of electromagnetic radiation during a pulse duration, the multiple sub-pulses having a sub-duration shorter than the pulse duration.

[0351] Embodiment 12 is a method according to any one of embodiments 1 to 11, wherein actuating the emitter includes: actuating the emitter to simultaneously emit two or more wavelengths as a single pulse or a single sub-pulse.

[0352] Embodiment 13 is a method according to any one of embodiments 1 to 12, wherein sensing the reflected electromagnetic radiation includes: generating a fluorescence exposure frame, and wherein the method further includes: providing the fluorescence exposure frame to a corresponding fluorescence system, and the corresponding fluorescence system determines the position of key tissue structures within the scene based on the fluorescence exposure frame.

[0353] Embodiment 14 is a method according to any one of Embodiments 1 to 13, further comprising: receiving the position of the critical tissue structure from the corresponding fluorescence system; generating a stacked frame including the position of the critical tissue structure; and combining the stacked frame with a color image frame depicting the scene to indicate the position of the critical tissue structure within the scene.

[0354] Embodiment 15 is a method according to any one of embodiments 1 to 14, wherein sensing reflected electromagnetic radiation through the pixel array includes: sensing reflected electromagnetic radiation generated by the laser marking mode to generate a laser marking exposure frame, and wherein the method also includes: providing the laser marking exposure frame to a corresponding laser marking system, the corresponding laser marking system determines the topology of the scene and / or the size of one or more objects in the scene; providing the position of the key tissue structure to the corresponding laser marking system; and receiving the topology and / or size of the key tissue structure from the corresponding laser marking system.

[0355] Embodiment 16 is a method according to any one of embodiments 1 to 15, wherein the critical tissue structure comprises one or more of nerves, ureters, blood vessels, arteries, blood flow or tumors.

[0356] Embodiment 17 is a method according to any one of Embodiments 1 to 16, further comprising: synchronizing the timing of the multiple electromagnetic radiation pulses to be emitted during a blanking period of the image sensor, wherein the blanking period corresponds to the time between the readout of the last row of valid pixels in the pixel array and the start of the next subsequent readout of the valid pixels in the pixel array.

[0357] Embodiment 18 is a method according to any one of Embodiments 1 to 17, wherein sensing the reflected electromagnetic radiation includes: sensing through a first pixel array and a second pixel array, so that a three-dimensional image can be generated based on the sensed reflected electromagnetic radiation.

[0358] Embodiment 19 is a method according to any one of embodiments 1 to 18, wherein actuating the transmitter comprises: actuating the transmitter to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to an electromagnetic radiation pulse.

[0359] Embodiment 20 is a method according to any one of embodiments 1 to 19, wherein at least a portion of the multiple electromagnetic radiation pulses emitted by the emitter is an excitation wavelength for causing the reagent to fluoresce, and wherein at least a portion of the reflected electromagnetic radiation sensed by the pixel array is a relaxation wavelength of the reagent.

[0360] Embodiment 21 is a method according to any one of embodiments 1 to 20, wherein sensing reflected electromagnetic radiation by the pixel array includes: generating a laser marking exposure frame by sensing reflected electromagnetic radiation generated by the transmitter pulse emitting the laser marking mode, wherein the laser marking exposure frame includes information for determining real-time measurement, and the information includes one or more of the following: the distance from the endoscope to the object; the angle between the endoscope and the object; or surface topology information about the object.

[0361] Embodiment 22 is a method according to any one of embodiments 1 to 21, wherein the laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 centimeters.

[0362] Embodiment 23 is a method according to any one of embodiments 1 to 22, wherein the laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

[0363] Embodiment 24 is a method according to any one of embodiments 1 to 23, wherein actuating the transmitter to emit the plurality of electromagnetic radiation pulses includes actuating the transmitter to emit a plurality of tool-specific laser marking patterns for each of a plurality of tools within a scene.

[0364] Embodiment 25 is a method according to any one of embodiments 1 to 24, wherein the laser marking mode emitted by the transmitter includes: a first output and a second output independent of each other, wherein the first output is used for light illumination and the second output is used for tool tracking.

[0365] Embodiment 26 is a method according to any one of embodiments 1 to 25, wherein at least a portion of the multiple electromagnetic radiation pulses include hyperspectral emissions, the hyperspectral emissions including one of the following: electromagnetic radiation having a wavelength from about 513nm to about 545nm and electromagnetic radiation having a wavelength from about 900nm to about 1000nm; or electromagnetic radiation having a wavelength from about 565nm to about 585nm and electromagnetic radiation having a wavelength from about 900nm to about 1000nm; wherein sensing the reflected electromagnetic radiation by the pixel array includes: generating a hyperspectral exposure frame based on the hyperspectral emissions.

[0366] Embodiment 27 is a method according to any one of Embodiments 1 to 26, further comprising: providing the hyperspectral exposure frame to a corresponding hyperspectral system, the corresponding hyperspectral system determining the position of the key tissue structure based on the hyperspectral exposure frame; receiving the position of the key tissue structure from the corresponding hyperspectral system; generating a stacked frame including the position of the key tissue structure; and combining the stacked frame with a color image frame depicting the scene to indicate the position of the key tissue structure within the scene.

[0367] Embodiment 28 is a method according to any one of Embodiments 1 to 27, wherein sensing reflected electromagnetic radiation through the pixel array includes: generating a laser marking exposure frame based on the emission of the laser marking mode, and wherein the method further includes: providing the laser marking exposure frame to a corresponding laser marking system, the corresponding laser marking system determines the topology of the scene and / or the size of one or more objects within the scene; providing the position of the key tissue structure to the corresponding laser marking system; and receiving the topology and / or size of the key tissue structure from the corresponding laser marking system.

[0368] Embodiment 29 is a method according to any one of embodiments 1 to 28, wherein the critical tissue structure is one or more of a nerve, a ureter, a blood vessel, an artery, blood flow, cancer tissue or a tumor.

[0369] Embodiment 30 is a system. The system includes: an emitter for emitting a plurality of electromagnetic radiation pulses; and an image sensor including a pixel array for sensing reflected electromagnetic radiation to generate a plurality of exposure frames. The system includes one or more processors that can be configured to execute instructions stored in a non-transitory computer-readable storage medium. The instructions include generating a reference frame for removing fixed pattern noise from the plurality of exposure frames, wherein the reference frame is based on dark frame data. The instructions include reducing fixed pattern noise in an exposure frame in the plurality of exposure frames by subtracting the reference frame from the exposure frame. The system causes at least a portion of the electromagnetic radiation pulses emitted by the emitter to include one or more of: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm; electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm; an electromagnetic radiation excitation wavelength that causes the reagent to fluoresce; or a laser marking mode.

[0370] Embodiment 31 is a system according to embodiment 30, wherein the instructions further include: stopping the emitter from pulsing electromagnetic radiation; and sensing the dark frame data through the pixel array when the emitter is not emitting electromagnetic radiation.

[0371] Embodiment 32 is a system according to any one of embodiments 30 to 31, wherein the multiple electromagnetic radiation pulses are emitted in a pattern of electromagnetic radiation of different wavelengths, and wherein the emitter repeats the pattern of electromagnetic radiation of different wavelengths.

[0372] Embodiment 33 is a system according to any one of Embodiments 30 to 32, wherein the pattern of electromagnetic radiation of different wavelengths includes: a zero emission period in which the transmitter does not pulse emit electromagnetic radiation, and wherein the method further includes: sensing the dark frame data through the pixel array during the zero emission period.

[0373] Embodiment 34 is a system according to any one of embodiments 30 to 33, wherein, when the emitter is not emitting electromagnetic radiation, the pixel array senses multiple dark exposure frames over multiple instances, wherein each of the multiple dark exposure frames includes dark frame data.

[0374] Embodiment 35 is a system according to any one of embodiments 30 to 34, wherein at least a portion of the electromagnetic radiation pulse emitted by the emitter causes a fluorescence exposure frame to be generated by the image sensor, and wherein the instructions also include providing the fluorescence exposure frame to a corresponding fluorescence system, which determines the location of key tissue structures within the scene based on the fluorescence exposure frame.

[0375] Embodiment 36 is a system according to any one of Embodiments 30 to 35, wherein the instructions further include: receiving the position of the critical tissue structure from the corresponding fluorescence system; generating an overlay frame including the position of the critical tissue structure; and combining the overlay frame with a color image frame depicting the scene to indicate the position of the critical tissue structure within the scene.

[0376] Embodiment 37 is a system according to any one of embodiments 30 to 36, wherein sensing reflected electromagnetic radiation through the pixel array includes: sensing reflected electromagnetic radiation generated by the laser marking mode to generate a laser marking exposure frame, and wherein the instruction also includes: providing the laser marking exposure frame to a corresponding laser marking system, which determines the topology of the scene and / or the size of one or more objects in the scene; providing the position of the key tissue structure to the corresponding laser marking system; and receiving the topology and / or size of the key tissue structure from the corresponding laser marking system.

[0377] Embodiment 38 is a device for performing any of the methods according to any of embodiments 1 to 29.

[0378] It should be appreciated that the various features disclosed herein provide significant advantages and advances in the art. The following claims are examples of some of those features.

[0379] In the above-mentioned detailed description of the present disclosure, for the purpose of simplifying the present disclosure, the various features of the present disclosure are concentrated in a single embodiment. The method of the present disclosure should not be understood as embodying such an intention: the disclosure protected by the claims requires more features than those explicitly listed in each claim. On the contrary, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0380] It should be understood that any features of the above-described arrangements, examples and embodiments may be combined in a single embodiment comprising a combination of features obtained from any of the disclosed arrangements, examples and embodiments.

[0381] It should be understood that the above configuration is only an exemplary application of the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, those skilled in the art may design many modifications and alternative configurations, and the appended claims are intended to cover these modifications and configurations.

[0382] Thus, while the present disclosure is shown in the drawings and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications may be made without departing from the principles and concepts described herein, including but not limited to changes in size, material, shape, form, function and mode of operation, assembly and use.

[0383] In addition, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components, where appropriate. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays may be programmed to perform one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. It will be appreciated by those skilled in the art that components may have different names. It is not intended herein to distinguish between components that are different in name but not in function.

[0384] For the purpose of illustration and description, the above-mentioned specific embodiments have been provided. These specific embodiments are not intended to be exhaustive or to limit the present disclosure to the disclosed specific forms. According to the above-mentioned teachings, many modifications and changes can be made to the present disclosure. In addition, it should be noted that any or all of the aforementioned alternative specific implementations can be used in any desired combination to form other mixed specific implementations of the present disclosure.

[0385] In addition, although specific implementations of the present disclosure have been described and illustrated, the present disclosure is not limited to the specific forms or arrangements of parts as described and illustrated. The scope of the present disclosure will be defined by the claims appended hereto, any future claims filed here and in different applications, and their equivalents.

Claims

1. A method for endoscopic imaging, the method comprising: actuating the transmitter to emit a plurality of pulses of electromagnetic radiation; sensing, by a pixel array of an image sensor, reflected electromagnetic radiation produced by the plurality of electromagnetic radiation pulses to generate a plurality of exposure frames; generating a reference frame for removing fixed pattern noise from the plurality of exposure frames, wherein the reference frame is based on dark frame data captured when the emitter is not emitting electromagnetic radiation; as well as reducing fixed pattern noise in an exposure frame of the plurality of exposure frames by subtracting the reference frame from the exposure frame; wherein at least a portion of the plurality of electromagnetic radiation pulses emitted by the emitter comprises a laser marking mode, and further comprising one or more of the following: Hyperspectral pulses of electromagnetic radiation that induce a spectral response; or an electromagnetic radiation excitation wavelength that causes the agent to fluoresce, wherein the plurality of exposure frames includes laser mapping exposure frames sensed in response to the emitter pulsing a laser mapping pattern, wherein the laser mapping exposure frames include information for determining a real-time measurement, the information including one or more of the following: distance from the endoscope to the subject; the angle between the endoscope and the object; or Surface topological information about the object.

2. The method according to claim 1, further comprising: stopping the pulsing of electromagnetic radiation by the transmitter; as well as The dark frame data is sensed by the pixel array when the emitter is not emitting electromagnetic radiation.

3. The method according to claim 1, wherein: The plurality of electromagnetic radiation pulses are emitted in a pattern of electromagnetic radiation at different wavelengths, and wherein the emitter repeats the pattern of electromagnetic radiation at different wavelengths.

4. The method according to claim 3, wherein: The pattern of electromagnetic radiation of different wavelengths includes a zero emission period during which the emitter does not pulse electromagnetic radiation, and wherein the method further includes sensing the dark frame data via the pixel array during the zero emission period.

5. The method according to claim 1, further comprising: A plurality of dark exposure frames are sensed by the pixel array when the emitter is not emitting electromagnetic radiation, wherein each dark exposure frame of the plurality of dark exposure frames includes dark frame data.

6. The method according to claim 5, wherein: The plurality of dark exposure frames are interspersed within the plurality of exposure frames.

7. The method according to claim 5, wherein: Generating the reference frame includes generating a plurality of reference frames based on the plurality of dark exposure frames, and wherein the method further includes enhancing the accuracy of at least one reference frame of the plurality of reference frames by sampling subsequent dark frame data from subsequent dark exposure frames.

8. The method according to claim 7, wherein: Enhancing the accuracy of the at least one reference frame includes taking into account the subsequent dark frame data from the subsequent dark exposure frame using exponential smoothing.

9. The method according to claim 1, wherein: Generating the reference frame includes interpolating two or more instances of dark frame data from two or more dark exposure frames captured at different integration times.

10. The method according to claim 1, wherein: Sensing the reflected electromagnetic radiation includes sensing during a readout period of the pixel array, wherein the readout period is a duration when active pixels in the pixel array are read.

11. The method according to claim 1, wherein: Actuating the emitter includes actuating the emitter to emit a plurality of sub-pulses of electromagnetic radiation during a pulse duration, the plurality of sub-pulses having sub-durations shorter than the pulse duration.

12. The method according to claim 1, wherein: Actuating the emitter includes actuating the emitter to simultaneously emit two or more wavelengths as a single pulse or a single sub-pulse.

13. The method according to claim 1, wherein: Sensing the reflected electromagnetic radiation includes generating a fluorescence exposure frame, and wherein the method further includes providing the fluorescence exposure frame to a corresponding fluorescence system, the corresponding fluorescence system determining locations of key tissue structures within a scene based on the fluorescence exposure frame.

14. The method according to claim 13, further comprising: receiving the location of the critical tissue structure from the corresponding fluorescence system; generating a stacked frame including the location of the key tissue structure; as well as The overlay frame is combined with a color image frame depicting the scene to indicate the location of the key tissue structures within the scene.

15. The method according to claim 14, wherein: Sensing reflected electromagnetic radiation by the pixel array includes sensing reflected electromagnetic radiation produced by the laser marking pattern to generate a laser marking exposure frame, and wherein the method further includes: providing the laser mapping exposure frame to a corresponding laser mapping system, the corresponding laser mapping system determining the topology of the scene and / or the size of one or more objects within the scene; Providing the location of the key tissue structure to the corresponding laser mapping system; and The topology and / or dimensions of the critical tissue structure are received from the corresponding laser mapping system.

16. The method according to claim 15, wherein: The critical tissue structures include one or more of nerves, ureters, blood vessels, arteries, blood flow, or tumors.

17. The method according to claim 1, further comprising: The timing of the plurality of electromagnetic radiation pulses to be emitted is synchronized during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of valid pixels in the pixel array and the start of a next subsequent readout of valid pixels in the pixel array.

18. The method according to claim 1, wherein: Sensing the reflected electromagnetic radiation includes sensing by a first pixel array and a second pixel array such that a three-dimensional image can be generated based on the sensed reflected electromagnetic radiation.

19. The method according to claim 1, wherein: Actuating the emitter includes actuating the emitter so as to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to an electromagnetic radiation pulse.

20. The method according to claim 1, wherein: At least a portion of the plurality of electromagnetic radiation pulses emitted by the emitter is an excitation wavelength for causing an agent to fluoresce, and wherein at least a portion of the reflected electromagnetic radiation sensed by the pixel array is a relaxation wavelength of the agent.

21. The method according to claim 1, wherein: The laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 centimeters.

22. The method according to claim 1, wherein: The laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

23. The method according to claim 1, wherein: Actuating the transmitter to emit the plurality of electromagnetic radiation pulses includes actuating the transmitter to emit a plurality of tool-specific laser marking patterns for each tool of a plurality of tools within a scene.

24. The method according to claim 1, wherein: The laser marking pattern emitted by the emitter includes a first output and a second output that are independent of each other, wherein the first output is used for light illumination and the second output is used for tool tracking.

25. The method of claim 1, wherein: At least a portion of the plurality of electromagnetic radiation pulses comprises hyper-spectral emissions, the hyper-spectral emissions comprising one of: Electromagnetic radiation having a wavelength from about 513 nm to about 545 nm and electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm; or electromagnetic radiation having a wavelength from about 565 nm to about 585 nm and electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm; Wherein sensing reflected electromagnetic radiation by the pixel array includes generating a hyperspectral exposure frame based on the hyperspectral emission.

26. The method according to claim 25, further comprising: providing the hyperspectral exposure frame to a corresponding hyperspectral system, the corresponding hyperspectral system determining a location of a key tissue structure based on the hyperspectral exposure frame; receiving the location of the critical tissue structure from the corresponding hyperspectral system; generating a stacked frame including the location of the key tissue structure; as well as The overlay frame is combined with a color image frame depicting a scene to indicate the location of the key tissue structures within the scene.

27. The method according to claim 26, wherein: Sensing reflected electromagnetic radiation by the pixel array includes generating a laser mapping exposure frame based on the emission of the laser mapping pattern, and wherein the method further includes: providing the laser mapping exposure frame to a corresponding laser mapping system, the corresponding laser mapping system determining the topology of the scene and / or the size of one or more objects within the scene; Providing the location of the key tissue structure to the corresponding laser mapping system; and The topology and / or dimensions of the critical tissue structure are received from the corresponding laser mapping system.

28. The method according to claim 27, wherein: The critical tissue structure is one or more of a nerve, a ureter, a blood vessel, an artery, blood flow, cancerous tissue, or a tumor.

29. An endoscope system comprising: a transmitter for transmitting a plurality of electromagnetic radiation pulses; an image sensor comprising an array of pixels for sensing reflected electromagnetic radiation to generate a plurality of exposure frames; and One or more processors capable of being configured to execute instructions stored in a non-transitory computer-readable storage medium, the instructions comprising: generating a reference frame for removing fixed pattern noise from the plurality of exposure frames, wherein the reference frame is based on dark frame data captured when the emitter is not emitting electromagnetic radiation; and reducing fixed pattern noise in an exposure frame of the plurality of exposure frames by subtracting the reference frame from the exposure frame; wherein at least a portion of the plurality of electromagnetic radiation pulses emitted by the emitter comprises a laser marking mode, and further comprising one or more of the following: Hyperspectral pulses of electromagnetic radiation that induce a spectral response; an electromagnetic radiation excitation wavelength that causes the agent to fluoresce, wherein the plurality of exposure frames includes laser mapping exposure frames sensed in response to the emitter pulsing the laser mapping pattern, wherein the laser mapping exposure frames include information for determining a real-time measurement, the information including one or more of the following: distance from the endoscope to the subject; the angle between the endoscope and the object; or Surface topological information about the object.

Images