Scene offset illumination using multiple emitters in a hyperspectral imaging system

Abstract

Offset illumination using multiple emitters in a hyperspectral imaging system is described. The system includes an emitter to emit pulses of electromagnetic radiation and an image sensor including an array of pixels to sense reflected electromagnetic radiation. The emitter includes a first emitter and a second emitter to emit electromagnetic radiation of different wavelengths. The system is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter includes electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, about 565 nm to about 585 nm, and / or about 900 nm to about 1000 nm.

Description

Technical Field

[0001] This disclosure relates to digital imaging, and more particularly to hyperspectral imaging in low-light environments. Background Technology

[0002] Advances in technology have enabled advancements in medical imaging capabilities. Endoscopes can be used to observe the inside of the body and examine the interior of organs or cavities. Endoscopes are used to investigate patient symptoms, confirm diagnoses, or provide medical treatment. Medical endoscopes can be used to observe a variety of body systems and parts, such as the gastrointestinal tract, respiratory tract, urethra, and abdominal cavity. Endoscopes are also used in surgical procedures, such as orthopedic surgery, surgery on joints or bones, surgery on the nervous system, and surgery within the abdominal cavity.

[0003] In some cases of endoscopic imaging, observing the color space may be advantageous or necessary. A digital color image comprises at least three layers, or "color channels," that accumulate to form an image with a range of hues. Each color channel measures the intensity and chromaticity of light in a spectral band. Typically, a digital color image includes color channels with red, green, and blue spectral bands (this may be referred to as a red-green-blue or RGB image). Each of the red, green, and blue channels contains luminance information for the red, green, or blue spectral bands. The luminance information from the individual red, green, and blue layers is combined to generate a color image. Because a color image is composed of individual layers, a conventional digital camera image sensor includes an array of color filters that allow red, green, and blue visible light wavelengths to strike selected pixel sensors. Each individual pixel sensor element is sensitive to a red, green, or blue wavelength and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image. The at least three different types of pixel sensors consume a lot of physical space, making it impossible to fit the complete pixel array into the small distal end of the endoscope.

[0004] Because conventional image sensors cannot be fitted into the distal end of an endoscope, they are traditionally located in the endoscope's handheld unit, which is held by the operator and not placed within the body cavity. In such endoscopes, light travels along the length of the endoscope from the handheld unit to the distal end. This configuration has significant limitations. Endoscopes with this configuration are delicate and can easily become misaligned or damaged by impacts or shocks during normal use. This can significantly degrade image quality, and the endoscope requires frequent repair or replacement.

[0005] Traditional endoscopes with image sensors housed in a handheld unit are limited to capturing only color images. However, in some implementations, it may be desirable to capture images in addition to color image data, using hyperspectral image data. Color images reflect what the human eye detects when observing the environment. However, the human eye is limited to seeing visible light and cannot detect other wavelengths of the electromagnetic spectrum. Additional information about the environment can be obtained at other wavelengths of the electromagnetic spectrum beyond the "visible light" wavelength. One means of obtaining image data beyond the visible spectrum is to apply hyperspectral imaging.

[0006] Hyperspectral imaging is used to identify different materials or objects and to identify different processes by providing information beyond what the human eye can see. Unlike normal camera images, which provide limited information to the human eye, hyperspectral imaging identifies specific compounds and biological processes based on their unique spectral characteristics. Hyperspectral imaging is complex and may require rapid computing power, sensitive detectors, and large data storage capacity.

[0007] Hyperspectral imaging traditionally requires dedicated image sensors that consume significant physical space and cannot be fitted into the distal end of an endoscope. Furthermore, if hyperspectral images are superimposed on black-and-white or color images to provide context for a physician, the camera (or multiple cameras) capable of generating the superimposed image can have many different types of pixel sensors sensitive to varying ranges of electromagnetic radiation. This would include three separate types of pixel sensors for generating RGB color images, and additional pixel sensors for generating hyperspectral image data at different wavelengths of the electromagnetic spectrum. This consumes significant physical space and requires a large pixel array to ensure satisfactory image resolution. In the case of endoscopic imaging, one or more cameras would be too large to fit into the distal end of the endoscope and would therefore be placed within the endoscope hand unit or robotic unit. This introduces the same drawbacks as described above and can result in endoscopes being extremely delicate, causing a significant deterioration in image quality if the endoscope is bumped or impacted during use.

[0008] Based on the foregoing, this paper describes systems, methods, and apparatuses for improved endoscopic imaging in low-light environments. The systems, methods, and apparatuses disclosed herein provide means for color and hyperspectral imaging using endoscopic devices. Attached Figure Description

[0009] Non-limiting and incomplete embodiments of this disclosure are described with reference to the following accompanying drawings, wherein, unless otherwise specified, similar reference numerals in the various views indicate similar parts. The advantages of this disclosure will be better understood with reference to the following description and drawings, wherein:

[0010] Figure 1This is a schematic diagram of a system with paired emitters and pixel arrays for digital imaging in low-light environments;

[0011] Figure 2 It is a system used to provide illumination to light-deficient environments used for endoscopic imaging;

[0012] Figure 2A This is a schematic diagram of the hardware of the complementary system;

[0013] Figures 3A to 3D This is a diagram illustrating the operating cycle of the sensor used to construct the exposure frame;

[0014] Figure 4A It is a graphical representation of the operation of the implementation scheme of the electromagnetic transmitter;

[0015] Figure 4B It is a graphical representation of changing the duration and magnitude of the emitted electromagnetic pulse to provide exposure control;

[0016] Figure 5 It is Figures 3A to 4B The illustration of an embodiment of the present disclosure is shown in the form of a sensor operating cycle, an electromagnetic transmitter, and a combination of emitted electromagnetic pulses, illustrating an imaging system during operation.

[0017] Figure 6A This is a schematic diagram of a method for recording video using full-spectrum light during the time period from t(0) to t(1);

[0018] Figure 6B This is a schematic diagram of the process of recording video by pulsating partial spectral light during the time period from t(0) to t(1);

[0019] Figures 7A to 7E A schematic diagram illustrates a method for recording video frames of both full-spectrum light and partitioned-spectrum light within a time interval;

[0020] Figure 8 It is a graphical display of the delay or jitter between the control signal and the electromagnetic radiation emitted by the transmitter;

[0021] Figure 9 It is a cross-sectional view of an optical fiber bundle including a central fiber and multiple surrounding fibers;

[0022] Figure 10 It is a graphical display of the top-hat profile and Gaussian profile used to transmit electromagnetic radiation to an optical fiber bundle.

[0023] Figure 11 It is a side view showing the output of electromagnetic radiation (light) from the fiber optic bundle compared to the camera's field of view;

[0024] Figure 12This is a side view showing the output of electromagnetic radiation from a bundle of optical fibers, where the ends of individual fibers are designed to provide a more uniform distribution of electromagnetic radiation.

[0025] Figure 13 This is a side view showing the output of electromagnetic radiation from an optical fiber bundle, wherein the optical fiber bundle includes plastic fibers and glass fibers coupled near the output;

[0026] Figure 14 This is a side view showing the output of electromagnetic radiation from an optical fiber bundle, which includes a diffuser located near the output.

[0027] Figure 15 It is a schematic flowchart of a method for driving a transmitter to illuminate a scene according to jitter specifications;

[0028] Figure 16 It is a schematic flowchart of a method for providing electromagnetic radiation to imaging scenes in low-light environments;

[0029] Figures 17A to 17C A light source with multiple emitters is shown;

[0030] Figure 18 A single optical fiber is shown, which illuminates a scene in a dark environment by outputting via a diffuser at the output point;

[0031] Figure 19 This illustrates a portion of an electromagnetic spectrum that, according to the principles and teachings of this disclosure, is divided into multiple distinct sub-spectrums that can be emitted by an emitter of a light source;

[0032] Figure 20 This is a schematic diagram illustrating the emission and readout timing for generating an image frame that includes multiple exposure frames produced by different partitions of pulsed light;

[0033] Figure 21A and Figure 21B A specific implementation of a plurality of pixel arrays for generating a three-dimensional image is shown, based on the principles and teachings of this disclosure;

[0034] Figure 22A and Figure 22B Perspective and side views of a specific implementation of an imaging sensor constructed on multiple substrates are shown, respectively, wherein multiple pixel columns forming a pixel array are located on a first substrate, and multiple circuit columns are located on a second substrate. The figures illustrate the electrical connections and communication between a pixel column and its associated or corresponding circuit column; and

[0035] Figure 23A and Figure 23BPerspective and side views of a specific implementation of an imaging sensor having a plurality of pixel arrays for generating three-dimensional images are shown, wherein the plurality of pixel arrays and the image sensor are constructed on a plurality of substrates. Detailed Implementation

[0036] This document discloses systems, methods, and apparatuses for digital imaging that are primarily applicable to medical applications such as medical endoscopic imaging. One embodiment of this disclosure is an endoscopic system for hyperspectral and color imaging in low-light environments. The methods, systems, and computer-based products disclosed herein provide imaging or diagnostic capabilities for use in medical robotic applications such as robots for performing imaging procedures, surgical procedures, etc.

[0037] One embodiment of this disclosure is an endoscopic imaging system including a transmitter for emitting pulses of electromagnetic radiation to illuminate a scene. The transmitter includes multiple laser beams (which may alternatively be referred to herein as separate “emitters” constituting the overall transmitter), which can operate independently of each other and emit electromagnetic radiation of different wavelengths. The multiple laser beams constituting the transmitter can each be configured to pulse electromagnetic radiation in different zones or wavelengths of the electromagnetic spectrum. The electromagnetic radiation pulses can be pulsed to an optical fiber bundle, which can then carry the pulsed electromagnetic radiation to the distal end of the endoscope to illuminate the scene. This specific implementation using multiple laser beams introduces problems when the electromagnetic radiation reaches the optical fiber bundle. For example, individual fibers within the fiber bundle may receive light of different wavelengths, different power levels, or more or less light than other individual fibers. This results in non-uniform illumination of the scene.

[0038] Based on the foregoing, embodiments of this disclosure include an intervening optical element and / or one or more dichroic mirrors. The intervening optical element and the one or more dichroic mirrors can be used in combination to provide uniform light to the fiber bundle. The intervening optical element can be positioned between the transmitter and the fiber bundle. The intervening optical element may include, for example, a diffuser, a mixing rod, a lens, or some other optical component for promoting a uniform light mixture. In one embodiment, a dichroic mirror is present for each laser beam of the transmitter. A first dichroic mirror for a first laser beam may be configured to reflect electromagnetic radiation pulsed by the first laser beam at a certain wavelength. A second dichroic mirror for a second laser beam may be configured to reflect electromagnetic radiation pulsed by the second laser beam at a certain wavelength, etc. Thus, the dichroic mirror can be configured to reflect electromagnetic radiation pulsed by a certain laser beam and be transparent to electromagnetic radiation of other wavelengths that may be pulsed by other laser beams of the transmitter. In one embodiment, the dichroic mirror is angled and positioned within an endoscope system such that electromagnetic radiation pulsed by the transmitter strikes the dichroic mirror and then changes the direction pulsed into the fiber bundle. The fiber optic bundle can then carry light through the endoscope system to illuminate the scene. This system for offset illumination makes it possible to use multiple different lasers, laser beams, or transmitters without the risk of illuminating the scene or non-uniform light transmitted through the fiber optic bundle.

[0039] Conventional endoscopes are designed such that the image sensor is placed at the proximal end of the device within the handheld unit. This configuration requires incident light to travel the length of the endoscope through precisely coupled optics. During normal use, precise alignment of these optics can be difficult, leading to image distortion or loss. The embodiments disclosed herein place the image sensor in a space-constrained environment at the distal end of the endoscope itself. This offers greater optical simplicity compared to specific implementations known in the art. However, the acceptable solution of this method is by no means simple and presents its own set of engineering challenges.

[0040] When the overall size of an image sensor is minimized so that it can be fitted into the distal end of an endoscope, a significant loss of image quality can occur. The area of ​​the pixel array of an image sensor can be reduced by decreasing the number of pixels and / or reducing the sensing area of ​​each individual pixel. Each of these modifications affects the resolution, sensitivity, and dynamic range of the resulting image. Conventional endoscopic imaging systems aim to sense stable broadband illumination and provide color information using segmented pixel arrays, such as Bayer pattern arrays. Given the drawbacks associated with segmented pixel arrays, this paper discloses alternative systems and methods using monochromatic (which may be referred to as “color-indeterminate”) pixel arrays that do not include individual pixel filters. In the embodiments disclosed herein, color information is provided by emitting electromagnetic radiation pulses of different wavelengths. The pulse imaging system disclosed herein can generate a color image overlaid with hyperspectral imaging data.

[0041] In one embodiment, color information is determined by capturing independent exposure frames in response to pulses of electromagnetic radiation of different wavelengths. Alternative pulses may include red, green, and blue wavelengths to generate an RGB image frame consisting of red, green, and blue exposure frames. In another embodiment, alternative pulses may include luminance (“Y”) pulses, chroma (“Cr”) pulses, and chroma (“Cb”) pulses to generate a YCbCr image frame consisting of luminance, chroma, and chroma data. The color image frame may also include data from hyperspectral exposure frames superimposed on the RGB or YCbCr image frame. The hyperspectral pulse may include one or more electromagnetic radiation pulses for inducing a spectral response. In one embodiment, the hyperspectral emission includes one or more of electromagnetic radiation having wavelengths of about 513 nm to about 545 nm; about 565 nm to about 585 nm; or about 900 nm to about 1000 nm. The alternation of wavelengths of pulsed electromagnetic radiation allows the use of full-pixel arrays and avoids artifacts induced by Bayer-mode pixel arrays.

[0042] In some cases, it is desirable to generate endoscopic images with multiple data types or multiple images stacked on top of each other. For example, it is desirable to generate color (RGB or YCbCr) images that also include hyperspectral imaging data stacked on top of color images. Overlapping images with this property allow physicians or computer programs to identify key body structures based on hyperspectral imaging data. Historically, this would have required the use of multiple sensor systems, including an image sensor for color imaging and one or more additional image sensors for hyperspectral imaging. In such systems, the multiple image sensors would have multiple types of pixel sensors, each sensitive to different ranges of electromagnetic radiation. In systems known in the art, this includes three separate types of pixel sensors for generating color images, and additional pixel sensors for generating hyperspectral image data at different wavelengths of the electromagnetic spectrum. These multiple different pixel sensors occupy too much physical space and cannot be located at the distal end of the endoscope. In systems known in the art, one or more cameras are not placed at the distal end of the endoscope, but rather within the endoscope handpiece or robotic unit. This introduces many disadvantages and makes the endoscope very fragile. When a fragile endoscope is subjected to impact or shock during use, it may be damaged and image quality may be degraded. In view of the above, this document discloses systems, methods, and apparatuses for endoscopic imaging in low-light environments. The systems, methods, and apparatuses disclosed herein provide a way to employ multiple imaging techniques in a single imaging session while simultaneously allowing one or more image sensors to be positioned at the distal end of the endoscope.

[0043] Hyperspectral imaging

[0044] In one embodiment, the systems, methods, and apparatuses disclosed herein provide means for generating hyperspectral imaging data in low-light environments. Spectral imaging utilizes multiple frequency bands across the electromagnetic spectrum. This differs from conventional cameras that capture only light across three wavelengths (including red, green, and blue light wavelengths) of the visible spectrum, which are distinguishable by the human eye, to generate RGB images. Spectral imaging can use any wavelength band of the electromagnetic spectrum, including infrared wavelengths, the visible spectrum, the ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands.

[0045] Hyperspectral imaging was initially developed for applications in mining and geology. Unlike normal camera images, which provide limited information to the human eye, hyperspectral imaging can identify specific minerals based on the spectral characteristics of different minerals. Hyperspectral imaging is useful even when captured in aerial imagery and can provide information about, for example, oil or gas leaks from pipes or natural wells and their impact on nearby vegetation. This information is gathered based on the spectral characteristics of certain materials, objects, or processes that can be identified through hyperspectral imaging. Hyperspectral imaging can also be used in medical imaging applications, where certain tissues, chemical processes, biological processes, and diseases can be identified based on unique spectral features.

[0046] In one implementation of hyperspectral imaging, the complete spectrum or some spectral information is collected at each pixel in the image plane. A hyperspectral camera can use specialized hardware to capture any suitable number of wavelength bands for each pixel, which can be interpreted as the complete spectrum. The objectives of hyperspectral imaging vary depending on the application. In one application, the objective is to obtain imaging data of the entire electromagnetic spectrum for each pixel in the image scene. In another application, the objective is to obtain imaging data of certain regions of the electromagnetic spectrum for each pixel in the image scene. Certain regions of the electromagnetic spectrum can be selected based on what can be identified in the image scene. These applications enable the precise identification of materials, tissues, chemical processes, biological processes, and diseases when certain materials or tissues cannot be identified in the visible light wavelength band. In some medical applications, hyperspectral imaging includes one or more specific regions of the electromagnetic spectrum that have been selected to identify certain tissues, diseases, chemical processes, etc. Some exemplary regions of the electromagnetic spectrum that can be pulsed for hyperspectral imaging in medical applications include the emission of electromagnetic radiation with wavelengths of about 513 nm to about 545 nm; about 565 nm to about 585 nm; and / or about 900 nm to about 1000 nm.

[0047] Hyperspectral imaging offers numerous advantages over conventional imaging and provides specific benefits in medical applications. Endoscopic hyperspectral imaging allows healthcare practitioners or computer-based programs to identify neural tissue, muscle tissue, blood vessels, cancer cells, typical non-cancerous cells, blood flow direction, and more. Hyperspectral imaging enables precise differentiation of atypical cancerous tissue from typical healthy tissue, thus allowing practitioners or computer-based programs to distinguish the boundaries of cancerous tumors during surgery or research imaging. Information obtained through hyperspectral imaging allows for the precise identification of certain tissues or conditions that might be undiagnosable or poorly diagnosed using conventional imaging. Furthermore, hyperspectral imaging can be used during medical procedures to provide image-guided surgery, allowing practitioners to, for example, view tissues behind certain tissues or fluids, identify atypical cancer cells in contrast to typical healthy cells, identify certain tissues or conditions, and identify key structures. Hyperspectral imaging provides specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated using conventional imaging.

[0048] In one embodiment of this disclosure, the endoscope system illuminates a source and pulses electromagnetic radiation for spectral or hyperspectral imaging. Pulsed hyperspectral imaging discussed herein includes one or more bands of pulsed electromagnetic spectrum and may include infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. In one embodiment, hyperspectral imaging includes the emission of electromagnetic radiation having wavelengths of about 513 nm to about 545 nm; about 565 nm to about 585 nm; and / or about 900 nm to about 1000 nm.

[0049] Pulse imaging

[0050] Some specific embodiments of this disclosure include aspects of a sensor and system combination design capable of generating high-resolution images with a reduced number of pixels in confined lighting environments. This is achieved by pulsed monochromatic wavelengths frame by frame, and by using a controlled light source combined with a high frame capture rate and a specially designed corresponding monochromatic sensor to switch or alternate between single different color wavelengths each frame. Additionally, electromagnetic radiation outside the visible spectrum can be pulsed to enable the generation of hyperspectral images. Pixels can be color-indeterminate, such that each pixel generates data for each electromagnetic radiation pulse, each pulse including pulses of red, green, and blue visible light wavelengths as well as pulses of other wavelengths suitable for hyperspectral imaging.

[0051] The system disclosed herein is an endoscope system for use in low-light environments. The system includes an endoscope comprising an image sensor configured to sense reflected electromagnetic radiation for generating multiple exposure frames, which can be combined to generate an RGB image frame overlaid with hyperspectral data. The system includes a transmitter for emitting pulses of electromagnetic radiation. The system includes a controller (optionally referred to as "control circuitry") in electrical communication with the image sensor and the transmitter. The controller controls the duty cycle of the transmitter in response to a signal corresponding to the duty cycle of the transmitter. The image sensor includes bidirectional pads for transmitting and receiving information. The bidirectional pads of the image sensor operate in a frame period divided into three defined states: a rolling readout state, a service line state, and a configuration state. The system includes an oscillator disposed in the controller and a frequency detector connected to the controller. The frequency detector controls the clock frequency of the image sensor in response to a signal from the controller corresponding to the oscillator frequency. The system enables clock signal data to be transmitted from the bidirectional pads of the image sensor to the controller during the service line and configuration phases. The system enables the exposure frames to be synchronized without using an input clock or data transmission clock.

[0052] For the purpose of facilitating an understanding of the principles of this disclosure, reference will now be made to embodiments shown in the accompanying drawings, and these embodiments will be described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any changes and further modifications to the features of the invention shown herein, as well as any additional applications of the principles of this disclosure as shown herein (which will generally occur to those skilled in the art and those familiar with the contents of this disclosure), will be considered within the scope of the disclosure protected by the claims.

[0053] Before disclosing and describing the structures, systems, and methods for generating images in low-light environments, it should be understood that this disclosure is not limited to the specific structures, configurations, process steps, and materials disclosed herein, as such structures, configurations, process steps, and materials can vary to some extent. Furthermore, it should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the invention will be defined only by the appended claims and their equivalents.

[0054] In describing and claiming the subject matter of this disclosure, the following terms will be used in accordance with the following definitions.

[0055] It should be noted that the singular forms “a,” “an,” and “the” used in this specification and the appended claims include multiple referents unless the context clearly indicates otherwise.

[0056] As used herein, the terms “comprising,” “including,” “characterized by,” and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unmentioned elements or method steps.

[0057] As used herein, the phrase “consisting of” and its grammatical equivalents exclude any element or step not included in the claims.

[0058] As used herein, the phrase “consistent with…” and its grammatical equivalents limit the scope of the claims to the specified materials or steps and to materials or steps that do not substantially affect one or more of the essential and novel features of the claimed disclosure.

[0059] As used in this article, the term "proximal" broadly refers to the concept of the part closer to the starting point.

[0060] As used in this article, the term "distal" generally refers to the opposite of the proximal, and therefore, depending on the context, it refers to the part that is farther from the starting point or the farthest part.

[0061] As used herein, color sensors or multispectral sensors are those known to have a color filter array (CFA) on which incident electromagnetic radiation is filtered to its single component. In the visible range of the electromagnetic spectrum, such CFAs can be based on a Bayer template or a modified form thereof to separate the green, red, and blue spectral components of light.

[0062] As used in this article, a monochrome sensor refers to an imaging sensor without filtering capabilities. Because pixels are color-invariant, their effective spatial resolution is significantly higher than that of pixel color equivalents in traditional single-sensor cameras (which typically employ Bayer mode filtering). Monochrome sensors also offer higher quantum efficiency because fewer incident photons are wasted between individual pixels.

[0063] As used herein, a transmitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of a transmitter can be configured to emit pulses having very specific frequencies or frequency ranges from the entire electromagnetic spectrum. Pulses may include wavelengths in both the visible and invisible ranges. The transmitter may cycle on and off to generate pulses, or may generate pulses with a shutter mechanism. The transmitter may have a variable power output level, or may be controlled by auxiliary devices such as apertures or filters. The transmitter may emit broad-spectrum or full-spectrum electromagnetic radiation that can generate pulses through color filtering or shutter opening and closing. The transmitter may include multiple electromagnetic sources, acting individually or in concert.

[0064] It should be noted that, as used herein, the term "light" refers to both a particle and a wavelength, and is intended to represent electromagnetic radiation detectable by the pixel array 122, and may include wavelengths from both the visible and invisible spectra of electromagnetic radiation. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is smaller than the entire spectrum, or in other words, wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a controllable light source with respect to a portion of the emitted electromagnetic spectrum, or a light source with physical properties of its components, emission intensity, or emission duration, or all of the above. An emitter may emit light in any dithered, diffuse, or collimated emission, and may be controlled digitally or through analog methods or systems. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and includes light sources such as lasers, LEDs, incandescent light, or any digitally controllable light source.

[0065] Now refer to the attached diagram, Figure 1 A schematic diagram of a system 100 for sequential pulse imaging in a low-light environment is shown. The system 100 can be deployed to generate an RGB image, wherein hyperspectral data is superimposed on the RGB image. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a partition of electromagnetic radiation in the low-light environment 112, and the pixel array 122 senses instances of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 operate sequentially such that one or more pulses of the partition of electromagnetic radiation produce an exposure frame comprising image data sensed by the pixel array 122.

[0066] The pixel array 122 of the image sensor can be electronically paired with the transmitter 102, such that the transmitter 102 and the pixel array 122 are synchronized during operation for both receiving emissions and adjustments performed within the system. The transmitter 102 can be tuned to emit electromagnetic radiation in the form of a laser, which can be pulsed to illuminate a darkened environment 112. The transmitter 102 can correspond to interval pulses for the operation and function of the pixel array 122. The transmitter 102 can pulse light in multiple electromagnetic zones, causing the pixel array to receive electromagnetic energy and generate a dataset corresponding in time to each specific electromagnetic zone. For example, Figure 1 A specific implementation is illustrated, wherein transmitter 102 emits electromagnetic radiation in four distinct zones, including red light at wavelength 104, green light at wavelength 106, blue light at wavelength 108, and hyperspectral emission 110. Hyperspectral emission 110 may include wavelength bands in the electromagnetic spectrum that elicit a spectral response. Hyperspectral emission 110 may comprise multiple separate and independent emissions.

[0067] exist Figure 1In an alternative embodiment not shown, the pulsed emission of light includes luminance (“Y”) emission, red hue (“Cr”) emission, and blue hue (“Cb”) emission, instead of pulsed red light 104 emission, pulsed green light 106 emission, and pulsed blue light 108 emission. In one embodiment, the controller or transmitter 102 modulates the electromagnetic radiation pulses according to color conversion coefficients to provide luminance and / or chromaticity information, which convert light energy from red, green, and blue light energy spaces to luminance, red hue, and blue hue light energy spaces. The pulsed emission of light may also include modulating blue hue (“λY+Cb”) pulses and / or modulating red hue (“δY+Cr”) pulses.

[0068] The low-light environment 112 includes structures, structures, and other elements that reflect a combination of red light 114, green light 116, and / or blue light 118. Structures perceived as red light 114 reflect pulsed red light 104. Reflection from the red structures causes pixel array 122 to sense red light 105 after the emission of pulsed red light 104. Data sensed by pixel array 122 generates a red exposure frame. Structures perceived as green light 116 reflect pulsed green light 106. Reflection from the green structures causes pixel array 122 to sense green light 107 after the emission of pulsed green light 106. Data sensed by pixel array 122 generates a green exposure frame. Structures perceived as blue light 118 reflect pulsed blue light 108. Reflection from the blue structures causes pixel array 122 to sense blue light 109 after the emission of pulsed blue light 108. Data sensed by pixel array 122 generates a blue exposure frame.

[0069] When the structure is a combination of colors, it will reflect a combination of pulsed red light 104, pulsed green light 106, and / or pulsed blue light 108. For example, a structure perceived as purple will reflect light from both pulsed red light 104 and pulsed blue light 108. The resulting data sensed by the pixel array 122 will indicate that light is reflected in the same area after the emission of pulsed red light 104 and pulsed blue light 108. When the resulting red and blue exposure frames are combined to form an RGB image frame, the RGB image frame will indicate that the structure is purple.

[0070] In embodiments where the light-deficient environment 112 includes a fluorescent reagent or dye, or includes one or more fluorescent structures, tissues, or other elements, the pulsed scheme may include the emission of certain fluorescent excitation wavelengths. Certain fluorescent excitation wavelengths may be selected to cause a known fluorescent reagent, fluorescent dye, or other structure to fluoresce. The fluorescent structure will be sensitive to the fluorescent excitation wavelength and will emit a fluorescent relaxation wavelength. After the emission of the fluorescent excitation wavelength, the fluorescence relaxation wavelength will be sensed by the pixel array 122. The data sensed by the pixel array 122 generates a fluorescent exposure frame. The fluorescent exposure frame may be combined with multiple other exposure frames to form an image frame. The data in the fluorescent exposure frame may be overlaid on an RGB image frame including data from red, green, and blue exposure frames.

[0071] In embodiments where the light-deficient environment 112 includes structures, tissues, or other materials that emit spectral responses to certain regions of the electromagnetic spectrum, the pulsed scheme may include emission of hyperspectral regions of electromagnetic radiation to elicit a spectral response from the structure, tissue, or other material present in the light-deficient environment 112. The spectral response includes the emission or reflection of electromagnetic radiation at certain wavelengths. The spectral response may be sensed by the pixel array 122 and generate a hyperspectral exposure frame. The hyperspectral exposure frame may be combined with multiple other exposure frames to form an image frame. Data in the hyperspectral exposure frame may be overlaid on an RGB image frame that includes data from red, green, and blue exposure frames.

[0072] In one implementation, the pulse scheme includes the emission of a laser mapping mode or an tool tracking mode. Following the emission of the laser mapping mode or tool tracking mode, reflected electromagnetic radiation sensed by the pixel array 122 generates a laser mapping exposure frame. Data from the laser mapping exposure frame can be provided to the corresponding system to identify, for example, the distance between tools present in the dark environment 112, the three-dimensional surface topology of the scene in the dark environment 112, the distance, size, or position of structures or objects within the scene. This data can be overlaid on an RGB image frame or otherwise provided to the system user.

[0073] Emitter 102 may be a laser emitter capable of emitting pulsed red light 104 to generate sensing red light 105 data, thereby identifying red light 114 elements within the dark environment 112. Emitter 102 may also emit pulsed green light 106 to generate sensing green light 107 data, thereby identifying green light 116 elements within the dark environment. Emitter 102 may also emit pulsed blue light 108 to generate sensing blue light 109 data, thereby identifying blue light 118 elements within the dark environment. Emitter 102 may further emit hyperspectral 110 emission to identify elements sensitive to hyperspectral 120 radiation. Emitter 102 may emit pulsed red light 104, pulsed green light 106, pulsed blue light 108, and pulsed hyperspectral 110 emission in any desired order.

[0074] Pixel array 122 senses reflected electromagnetic radiation. Each of the data sensed (red light 105, green light 107, blue light 109, and hyperspectral 111) can be referred to as an "exposure frame." Sensing hyperspectral 111 can result in multiple separate and independent exposure frames. For example, sensing hyperspectral 111 can generate a first hyperspectral exposure frame at a first partition of electromagnetic radiation, a second hyperspectral exposure frame at a second partition of electromagnetic radiation, and so on. A specific color or wavelength partition is assigned to each exposure frame, where the assignment is based on the timing of pulsed color or wavelength partitions from emitter 102. The combination of exposure frames and assigned specific color or wavelength partitions can be referred to as a dataset. Even if pixel 122 is not a dedicated color, colors can be assigned to any given dataset based on prior information about the emitter.

[0075] For example, during operation, after the pulsed red light 104 is pulsed in the dark environment 112, the pixel array 122 senses the reflected electromagnetic radiation. The reflected electromagnetic radiation generates an exposure frame, and this exposure frame is classified as the sensed red light 105 data because it corresponds temporally to the pulsed red light 104. The exposure frame, along with its temporal correspondence to the pulsed red light 104, constitutes a “dataset.” This process is repeated for each partition of electromagnetic radiation emitted by the transmitter 102. The data created by the pixel array 122 includes the sensed red light 105 exposure frame, which identifies the red light 114 component in the dark environment and corresponds temporally to the pulsed red light 104. The data also includes the sensed green light 107 exposure frame, which identifies the green light 116 component in the dark environment and corresponds temporally to the pulsed green light 106. The data also includes the sensed blue light 109 exposure frame, which identifies the blue light 118 component in the dark environment and corresponds temporally to the pulsed blue light 108. The data also includes sensed hyperspectral 111 exposure frames, which identify elements that are sensitive to hyperspectral 120 radiation and correspond in time to hyperspectral 110 emission.

[0076] In one implementation, three datasets representing red, green, and blue electromagnetic pulses are combined to form a single image frame. Thus, information from red, green, and blue exposure frames is combined to form a single RGB image frame. One or more additional datasets representing other wavelength partitions may be overlaid on the single RGB image frame. These additional datasets may represent, for example, laser mapping data, fluorescence imaging data, and / or hyperspectral imaging data.

[0077] It should be understood that, without departing from the scope of this disclosure, this disclosure is not limited to any particular color combination or any particular electromagnetic partition, and any color combination or any electromagnetic partition can be used in place of RED, GREEN, and BLUE, such as cyan, magenta, and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible and invisible wavelengths. In the figure, the dark environment 112 to be imaged includes a red light portion 114, a green light portion 116, and a blue light portion 118, and also includes elements sensitive to hyperspectral radiation 120, which can be sensed and mapped into the 3D rendering. As shown, the reflected light from the electromagnetic pulse contains only data of the portion of the object having a specific color corresponding to the pulse's color partition. These individual color (or color interval) datasets can then be used to reconstruct the image by combining the datasets at 126. Information in each of the multiple exposure frames (i.e., multiple datasets) can be combined by a controller 124, control circuitry, a camera controller, an image sensor, an image signal processing pipeline, or some other computational resource configured to process the multiple exposure frames and combine the datasets at 126. As discussed herein, controller 124 may include the structure and functionality of control circuitry, a camera controller, and / or an image signal processing pipeline. This dataset can be combined to generate a single image frame either within the endoscope unit itself or off-site by some other processing resources.

[0078] Figure 2This is a system 200 for providing illumination to light-deficient environments, such as those used for endoscopic imaging. System 200 can be used in conjunction with any of the systems, methods, or apparatuses disclosed herein. System 200 includes a transmitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an internal cavity waveguide 210, an internal cavity 212, and an image sensor 214 with accompanying optical components, such as lenses. The transmitter 202 (generally referred to as a “light source”) generates light that passes through the jumper waveguide 206 and the internal cavity waveguide 210 to illuminate the scene at the distal end of the internal cavity 212. The transmitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, hyperspectral, fluorescence excitation, laser mapping pulse schemes, or other wavelengths. The internal cavity 212 can be inserted into a patient for imaging, such as during surgery or examination. The output light is shown as dashed line 216. The image sensor 214 can be used to capture the scene illuminated by the light and display the scene to a physician or other medical personnel. Controller 204 can provide control signals to transmitter 202 to control when illumination is provided to a scene. In one embodiment, transmitter 202 and controller 204 are located within a camera controller (CCU) or external console to which the endoscope is attached. If image sensor 214 includes a CMOS sensor, light can be periodically provided to the scene in a series of illumination pulses between readout cycles of image sensor 214 during a so-called blanking period. Therefore, the light can be pulsed in a controlled manner to avoid overlapping with the readout cycles of image pixels in the pixel array of image sensor 214.

[0079] In one embodiment, the cavity waveguide 210 includes one or more optical fibers. These optical fibers may be made of low-cost materials such as plastic to allow for the handling of the cavity waveguide 210 and / or other parts of the endoscope. In one embodiment, the cavity waveguide 210 is a single glass fiber with a diameter of 500 micrometers. A jumper waveguide 206 may be permanently attached to the transmitter 202. For example, the jumper waveguide 206 may receive light from a transmitter within the transmitter 202 and provide light to the cavity waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 206 includes one or more glass optical fibers. The jumper waveguide may include any other type of waveguide for guiding light to the cavity waveguide 210. The connector 208 may selectively couple the jumper waveguide 206 to the cavity waveguide 210 and allow light within the jumper waveguide 206 to pass through the cavity waveguide 210. In one implementation, the cavity waveguide 210 is directly coupled to the light source without any intervening jumper waveguide 206.

[0080] Image sensor 214 includes a pixel array. In one embodiment, image sensor 214 includes two or more pixel arrays for generating a three-dimensional image. Image sensor 214 may constitute two additional image sensors, each with an independent pixel array and capable of operating independently of each other. The pixel array of image sensor 214 includes active pixels and optical black (“OB”) pixels or light-blind pixels. Active pixels may be transparent “color-indeterminate” pixels capable of sensing imaging data of electromagnetic radiation at any wavelength. Optical black pixels are read during the blanking period of the pixel array when the pixel array is “reset” or calibrated. In one embodiment, light pulses during the blanking period of the pixel array when an optical black pixel is read. After the optical black pixel has been read, the active pixel is read during the readout period of the pixel array. The active pixel may be charged by electromagnetic radiation pulsed during the blanking period, such that the active pixel is ready to be read by the image sensor during the readout period of the pixel array.

[0081] Figure 2A This is a schematic diagram of complementary system hardware such as a dedicated or general-purpose computer. Embodiments within the scope of this disclosure may also include physical and other non-transitory computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible through a general-purpose or dedicated computer system. A computer-readable medium storing computer-executable instructions is a computer storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, specific embodiments of this disclosure may include at least two distinctly different types of computer-readable media: computer storage media (devices) and transmission media.

[0082] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to store program code tools in the form of desired computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0083] A “network” refers to one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. In one implementation, sensors and camera controllers may be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transmitted or provided to a computer via a network or other communication connection (hard-wired, wireless, or a combination of hard-wired and wireless), the computer reasonably considers that connection as a transmission medium. The transmission medium may include networks and / or data links that can be used to carry program code tools in the form of desired computer-executable instructions or data structures and are accessible via general-purpose or special-purpose computers. The above combinations should also be covered within the scope of computer-readable media.

[0084] Furthermore, upon arrival at various computer system components, program code tools in the form of computer-executable instructions or data structures can be automatically transferred by a transmission medium to computer storage media (devices) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or the computer system's non-volatile computer storage media (devices). RAM may also include solid-state drives (SSDs) or PCIx-based real-time memory tiered storage devices, such as FusionIO. Therefore, it should be understood that computer storage media (devices) may be included in computer system components that also (or even primarily) utilize transmission media.

[0085] Computer-executable instructions include, for example, instructions and data that, when executed by one or more processors, cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform certain functions or groups of functions. Computer-executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter of the invention has been set forth in terms of language with respect to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or steps described above. Rather, the features and steps described above are disclosed as examples of implementing the claims.

[0086] Those skilled in the art will understand that this disclosure can be implemented in a network computing environment with various types of computer system configurations, including personal computers, desktop computers, laptop computers, information processors, controllers, camera controllers, handheld devices, handheld components, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, various storage devices, etc. It should be noted that any of the aforementioned computing devices can be provided by or located within an entity. This disclosure can also be implemented in a distributed system environment, where local and remote computer systems are connected via a network (through a hard-wired data link, a wireless data link, or a combination of hard-wired and wireless data links), and both can perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0087] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to execute one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. Those skilled in the art will understand that components may have different names. This document is not intended to distinguish between components that are different in name rather than function.

[0088] Figure 2A This is a block diagram illustrating an exemplary computing device 250. The computing device 250 can be used to execute various programs, such as those discussed herein. The computing device 250 can be used as a server, client, or any other computing entity. The computing device 250 can perform various monitoring functions discussed herein and can execute one or more applications, such as those described herein. The computing device 250 can be any of a variety of computing devices, such as a desktop computer, laptop computer, server computer, handheld computer, camera controller, tablet computer, etc.

[0089] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all of which are coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory devices 254 and / or mass storage devices 258. The processor 252 may also include various types of computer-readable media, such as cache memory.

[0090] The memory device 254 includes various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 264) and / or non-volatile memory (e.g., read-only memory (ROM) 266). The memory device 254 may also include rewritable ROM, such as flash memory.

[0091] Mass storage devices 258 include various computer-readable media, such as magnetic tape, magnetic disks, optical disks, solid-state storage (e.g., flash memory), etc. Figure 2 As shown, a specific mass storage device is hard disk drive 274. Various drives may also be included in mass storage device 258 to enable reading and / or writing from various computer-readable media. Mass storage device 258 includes removable media 276 and / or non-removable media.

[0092] I / O device 260 includes various means capable of inputting data and / or retrieving data and / or other information to or from computing device 250. Exemplary I / O device 260 includes digital imaging device, electromagnetic sensor and transmitter, cursor control device, keyboard, keypad, microphone, monitor or other display device, speaker, printer, network interface card, modem, lens, CCD or other image capture device, etc.

[0093] Display device 280 includes any type of device capable of displaying information to one or more users of computing device 250. Examples of display device 280 include monitors, display terminals, video projection devices, etc.

[0094] Interface 256 includes various interfaces enabling computing device 250 to interact with other systems, devices, or computing environments. Exemplary interface 256 may include any number of different network interfaces 270, such as interfaces for connecting to a local area network (LAN), wide area network (WAN), wireless network, and the Internet. Other interfaces include user interface 268 and peripheral device interface 272. Interface 256 may also include one or more user interface elements 268. Interface 256 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touchpad, etc.), keyboards, etc.

[0095] Bus 262 enables processor 252, memory device 254, interface 256, mass storage device 258, and I / O device 260 to communicate with each other and with other devices or components coupled to bus 262. Bus 262 represents one or more of several types of bus architectures (such as system bus, PCI bus, IEEE 1394 bus, USB bus, etc.).

[0096] For illustrative purposes, the programs and other executable devices shown herein are discrete entities; however, it should be understood that such programs and devices may reside at various times in different memory devices of computing device 250 and be executed by processor 252. Alternatively, the systems and programs described herein may be implemented in hardware, or by a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays may be programmed to execute one or more systems and programs described herein.

[0097] Figure 3A The image sensor's operating cycle is illustrated in scroll readout mode or during sensor readout 300. Frame readout may begin at and be represented by a vertical line 310. The readout cycle is represented by a diagonal or slanted line 302. The sensor can be read out line by line, with the top of the downward-sloping edge designated as the top row 312 and the bottom of the downward-sloping edge designated as the bottom row 314. The time between the last row readout and the next readout cycle may be referred to as the blanking cycle 316. It should be noted that some of the sensor pixel rows may be covered with light-shielding material (e.g., a metallic coating or any other generally black layer of another material type). These covered pixel rows may be referred to as optical black rows 318 and 320. Optical black rows 318 and 320 can be used as input to correction algorithms. Figure 3A As shown, these optical black rows 318 and 320 can be located on top of the pixel array, at the bottom of the pixel array, or at both the top and bottom of the pixel array.

[0098] Figure 3B A method is illustrated for controlling the amount of electromagnetic radiation (e.g., light) exposed to pixels so that it is integrated or accumulated by the pixels. It should be understood that photons are the fundamental particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into electrical charge or current. An electronic shutter or rolling shutter (shown as dashed line 322) can be used to begin the integration time by resetting the pixels. Light will then be integrated until the next readout stage. The position of the electronic shutter 322 can be moved between two readout cycles 302 to control the pixel saturation of a given amount of light. It should be noted that this technique allows for a constant integration time between two different rows, but introduces a delay when moving from the top row to the bottom row.

[0099] Figure 3C The diagram shows the case where the electronic shutter 322 has been removed. In this configuration, the integration of incident light can begin during readout 302 and end at the next readout cycle 302, which also defines the start of the next integration.

[0100] Figure 3DThis illustrates a configuration without an electronic shutter 322 but with controlled and pulsed light 210 during the blanking period 316. This ensures that all lines see the same light emitted from the same light pulse 210. In other words, each line will begin its integration in a dark environment, which may be located after the optically black line 320 of the readout frame (m) to obtain the maximum light pulse width, and will then receive the light pass and end its integration in a dark environment, which may be located before the optically black line 318 of the next subsequent readout frame (m+1) to obtain the maximum light pulse width. For example... Figure 3D In this process, the image generated by the light pulse will only be available during the readout of frame (m+1) without interfering with frames (m) and (m+2). It should be noted that the condition for the light pulse to be read out only in one frame and not interfere with adjacent frames is that a given light pulse is fired during the blanking period 316. Because the optical black lines 318 and 320 are not sensitive to light, the optical black line following time 320 of frame (m) and the optical black line preceding time 318 of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 210.

[0101] like Figure 3A As shown, the sensor can cycle multiple times to receive data for each pulse color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum). Each cycle can be timed. In one embodiment, the cycle can be timed to operate at intervals of 16.67 ms. In another embodiment, the cycle can be timed to operate at intervals of 8.3 ms. It should be understood that other timing intervals are contemplated and are intended to fall within the scope of this disclosure.

[0102] Figure 4A The operation of an embodiment of the electromagnetic transmitter is illustrated graphically. The transmitter can be timed to correspond to the sensor's operating cycle, such that electromagnetic radiation is emitted during and / or a portion of the sensor's operating cycle. Figure 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the transmitter may pulse during readout period 302 of a sensor operating cycle. In one embodiment, the transmitter may pulse during blanking period 316 of a sensor operating cycle. In one embodiment, the transmitter may pulse for a duration that spans a portion of two or more sensor operating cycles. In one embodiment, the transmitter may begin pulsed during blanking period 316 or during optical black portion 320 of readout period 302, and end pulsed during readout period 302 or during optical black portion 318 of the readout period 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of this disclosure, provided that the transmitter's pulses correspond to the sensor's cycles.

[0103] Figure 4B The control of exposure is graphically represented by varying the duration and magnitude of the emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, and pulse 3 at 416). A transmitter with a fixed output magnitude can be combined with the above... Figure 3D and Figure 4A During any given cycle, pulses are applied at regular intervals to provide the required electromagnetic energy to the pixel array. A transmitter with a fixed output value can pulse over longer time intervals, thus providing more electromagnetic energy to the pixels, or it can pulse over shorter time intervals, thus providing less electromagnetic energy. Whether longer or shorter time intervals are required depends on the operating conditions.

[0104] Compared to adjusting the time interval of a fixed output pulse value from the transmitter, increasing the pulse value itself can provide more electromagnetic energy to the pixel. Similarly, decreasing the pulse value provides less electromagnetic energy to the pixel. It should be noted that, if needed, the system implementation can have the ability to adjust both the pulse value and duration simultaneously. Furthermore, the sensor can be adjusted to increase its sensitivity and duration according to the requirements of optimal image quality. Figure 4B The diagram illustrates variations in the magnitude and duration of pulses. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. Additionally, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416, such that the electromagnetic energy provided by this pulse is represented by the area under the pulse shown in the diagram. In the illustration, pulse 2 at 414 has a relatively lower magnitude or intensity and a longer duration compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration compared to pulse 1 at 412 and pulse 2 at 414.

[0105] Figure 5 For the combination of the principles and teachings of this disclosure Figures 3A to 3D and Figure 4A The operating cycle, electromagnetic transmitter, and emitted electromagnetic pulses are graphically represented to indicate an embodiment of the imaging system disclosed herein during operation. As can be seen in the figures, the electromagnetic transmitter pulses primarily during the blanking cycle 316 of the image sensor, causing the pixels to be charged and ready to be read during the readout cycle 302 of the image sensor cycle. Figure 5 The dotted line in the image represents an electromagnetic radiation pulse (from...). Figure 4A The electromagnetic radiation pulses are primarily emitted during the blanking period 316 of the image sensor, but can be superimposed with the readout period 302 of the image sensor.

[0106] An exposure frame includes data read from the pixel array of the image sensor during readout cycle 302. The exposure frame can be combined with an indication of the type of pulse emitted by the transmitter prior to readout cycle 302. This combination of exposure frames and pulse type indication can be referred to as a dataset. Multiple exposure frames can be combined to generate a black-and-white or RGB color image. Additionally, hyperspectral, fluorescence, and / or laser mapping imaging data can be overlaid on the black-and-white or RGB image.

[0107] In one embodiment, the RGB image frame is generated based on three exposure frames, including a red exposure frame generated by the image sensor after red emission, a green exposure frame generated by the image sensor after green emission, and a blue exposure frame generated by the image sensor after blue emission. Hyperspectral imaging data may be superimposed on the RGB image frame. Hyperspectral imaging data may be extracted from one or more hyperspectral exposure frames. The hyperspectral exposure frame includes data generated by the image sensor during readout period 302 following hyperspectral emission of electromagnetic radiation. Hyperspectral emission includes any suitable emission in the electromagnetic spectrum and may include multiple light emissions spanning the entire electromagnetic spectrum. In one embodiment, hyperspectral emission includes emission of electromagnetic radiation having wavelengths of about 513 nm to about 545 nm; about 565 nm to about 585 nm; and / or about 900 nm to about 1000 nm. Hyperspectral exposure frames may include multiple hyperspectral exposure frames, each generated by the image sensor following a different type of hyperspectral emission. In one embodiment, the hyperspectral exposure frame includes a plurality of hyperspectral exposure frames, including a first hyperspectral exposure frame generated by an image sensor after the emission of electromagnetic radiation with wavelengths of about 513 nm to about 545 nm, a second hyperspectral exposure frame generated by an image sensor after the emission of electromagnetic radiation with wavelengths of about 565 nm to about 585 nm, and a third hyperspectral exposure frame generated by an image sensor after the emission of electromagnetic radiation with wavelengths of about 900 nm to about 1000 nm. The hyperspectral exposure frame may include additional hyperspectral exposure frames generated by the image sensor after other hyperspectral emissions of light required for the imaging application.

[0108] Hyperspectral exposure frames can be generated by an image sensor after emission of electromagnetic radiation in multiple different zones. For example, a single hyperspectral exposure frame can be sensed by a pixel array after emission of electromagnetic radiation with wavelengths of approximately 513 nm to approximately 545 nm; approximately 565 nm to approximately 585 nm; and approximately 900 nm to approximately 1000 nm. The emission of electromagnetic radiation may include a single pulse in which each of the multiple wavelengths is emitted simultaneously; multiple sub-pulses in which each sub-pulse is electromagnetic radiation of a different wavelength; or some combination thereof. The emission of electromagnetic radiation with one or more pulses may occur during a blanking period 316, which occurs before a readout period 302, in which the pixel array senses the exposure frame.

[0109] In one embodiment, the exposure frame is data sensed by the pixel array during readout period 302, which occurs after blanking period 316. Electromagnetic radiation is emitted during blanking period 316. In one embodiment, a portion of the electromagnetic radiation emission overlaps with readout period 316. Blanking period 316 occurs while the optical black pixels of the pixel array are being read, and readout period 302 occurs while the active pixels of the pixel array are being read. Blanking period 316 may overlap with readout period 302.

[0110] Figure 6A and Figure 6B The process for recording image frames is illustrated. Multiple image frames can be chained together to generate a video stream. A single image frame can include data from multiple exposure frames, where the exposure frames are data sensed by the pixel array after electromagnetic radiation is emitted. Figure 6A The diagram illustrates a typical process that is usually implemented using a color image sensor with a color filter array (CFA) to filter out certain wavelengths of light for each pixel. Figure 6B This is the process disclosed in this paper, and it can be achieved using a monochrome "color-indeterminate" image sensor that receives electromagnetic radiation of all wavelengths.

[0111] Figure 6A The process shown occurs from time t(0) to time t(1). The process begins with the emission of white light at 602 and the sensing of white light at 604. At 606, an image is processed and displayed based on the sensing at 604.

[0112] Figure 6BThe process shown occurs from time t(0) to time t(1). The process begins with the emission of green light 612, and after the emission of green light 612, reflected electromagnetic radiation 614 is sensed. The process continues with the emission of red light 616, and after the emission of red light 616, reflected electromagnetic radiation 618 is sensed. The process continues with the emission of blue light 620, and after the emission of blue light 620, reflected electromagnetic radiation 622 is sensed. The process continues with one or more emissions of hyperspectral light 624, and after each of these one or more emissions of hyperspectral light 624, reflected electromagnetic energy 626 is sensed.

[0113] Figure 6B The method shown provides higher resolution images and provides apparatus for generating RGB images that also include hyperspectral imaging data. When using partitioned spectroscopy (such as...) Figure 6B As shown), this allows the sensor to be sensitive to electromagnetic energy across all wavelengths. Figure 6B In the process illustrated, the monochromatic pixel array is instructed to sense electromagnetic energy from predetermined partitions of the full electromagnetic energy spectrum in each cycle. Therefore, to form an image, the sensor only needs to cycle through multiple different partitions within the full spectrum. The final image is assembled based on these multiple cycles. Because the image from each color partition frame cycle has a higher resolution (compared to a CFA pixel array), the resulting image created when the partitioned light frames are combined also has a higher resolution. In other words, because each pixel within the array (rather than at most every other pixel in a CFA sensor) senses the amplitude of the energy for a given pulse and a given scene only at intervals, a higher resolution image is produced for each scene.

[0114] As in Figure 6A and Figure 6B The implementation shown can be graphically illustrated between times t(0) and t(1), for Figure 6B Sensors in the partitioned spectral system Figure 6A Each cycle in the full-spectrum system is performed at least four times. In one implementation, the display device (LCD panel) operates at a rate of 50 to 60 frames per second. In such implementations, Figure 6B The localized optical system can operate at 200 to 240 frames per second to maintain the continuity and smoothness of the displayed video. In other implementations, different capture and display frame rates may exist. Furthermore, the average capture rate can be any multiple of the display rate.

[0115] In one implementation, it may be desirable that not all zones are represented equally within the system frame rate. In other words, not all light sources must pulse with the same regularity in order to emphasize and de-emphasize various aspects of the recorded scene as needed by the user. It should also be understood that invisible and visible zones of the electromagnetic spectrum can be pulsed together within the system, with their corresponding data values ​​stitched into the video output for display to the user.

[0116] The implementation scheme may include the following pulse cycle modes:

[0117] i. Green pulse;

[0118] ii. Red pulse;

[0119] iii. Blue pulse;

[0120] iv. Green pulse;

[0121] v. Red pulse;

[0122] vi. Blue pulse;

[0123] vii. Hyperspectral pulse;

[0124] viii. (repeated)

[0125] The implementation scheme may include the following pulse cycle modes:

[0126] i. Brightness pulse;

[0127] ii. Redness pulse;

[0128] iii. Luminance pulse;

[0129] iv. Blue light pulse;

[0130] v. Hyperspectral pulse;

[0131] vi. (to repeat)

[0132] The implementation scheme may include the following pulse cycle modes:

[0133] i. Brightness pulse;

[0134] ii. Redness pulse;

[0135] iii. Luminance pulse;

[0136] iv. Blue light pulse;

[0137] v. Brightness pulse;

[0138] vi. Redness pulse;

[0139] vii. Brightness pulse;

[0140] viii. Blue light pulse;

[0141] ix. Hyperspectral pulse;

[0142] x. (repeated)

[0143] As can be seen in this example, hyperspectral partitions can be pulsed at rates different from those of other partitions. This emphasizes a particular aspect of the scene, where hyperspectral data is simply superimposed on other data in the video output to make the desired emphasis. It should be noted that adding hyperspectral partitions above the red, green, and blue partitions does not necessarily require the serialized system to operate at four times the rate of a full-spectrum non-serialized system, as each partition does not need to be represented equally in the pulse pattern. As seen in this implementation, adding fewer hyperspectral partition pulses represented in the pulse pattern results in a sensor loop speed increase of less than 20% to accommodate irregular partition sampling.

[0144] In various implementations, the pulse cycling mode may also include any of the following wavelengths in any suitable order. Such wavelengths are particularly suitable for exciting fluorescent reagents to generate fluorescence imaging data by sensing the relaxation emission of the fluorescent reagent based on its relaxation emission:

[0145] i.770±20nm;

[0146] ii. 770±10nm;

[0147] iii. 770±5nm;

[0148] iv. 790±20nm;

[0149] v.790±10nm;

[0150] vi. 790±5nm;

[0151] vii.795±20nm;

[0152] viii. 795±10nm;

[0153] ix.795±5nm;

[0154] x.815±20nm;

[0155] xi.815±10nm;

[0156] xii.815±5nm;

[0157] xiii. 770nm to 790nm; and / or

[0158] xiv. 795nm to 815nm.

[0159] In various implementations, the pulse cycle may also include any of the following wavelengths in any suitable order. Such wavelengths are particularly suitable for generating hyperspectral imaging data:

[0160] i.513±545nm;

[0161] ii. 565nm to 585nm;

[0162] iii. 900nm to 1000nm;

[0163] iv. 513±5nm;

[0164] v.513±10nm;

[0165] vi. 513±20nm;

[0166] vii.513±30nm;

[0167] viii.513±35nm;

[0168] ix.545±5nm;

[0169] x.545±10nm;

[0170] xi.545±20nm;

[0171] xii.545±30nm;

[0172] xiii.545±35nm;

[0173] xiv.565±5nm;

[0174] xv.565±10nm;

[0175] xvi.565±20nm;

[0176] xvii.565±30nm;

[0177] xviii.565±35nm;

[0178] xix.585±5nm;

[0179] xx.585±10nm;

[0180] xxi.585±20nm;

[0181] xxii.585±30nm;

[0182] xxiii.585±35nm;

[0183] xxiv.900±5nm;

[0184] xxv.900±10nm;

[0185] xxvi.900±20nm;

[0186] xxvii.900±30nm;

[0187] xxviii.900±35nm;

[0188] xxix.1000±5nm;

[0189] xxx.1000±10nm;

[0190] xxxi.1000±20nm;

[0191] xxxii. 1000±30nm; or

[0192] xxxiii.1000±35nm.

[0193] Partition loops can be divided to adapt to or approximate various imaging and video standards. In one implementation, a partition loop includes the following: Figures 7A to 7D The pulses of electromagnetic energy in the red, green, and blue spectra are best illustrated in the image. The timing relationship between the emission of electromagnetic radiation pulses by the transmitter and the readout of the pixel array is shown in... Figures 7A to 7D This is further illustrated in the text.

[0194] exist Figure 7A Different light intensities have been achieved by modulating the width or duration of the light pulse within the working range indicated by the vertical gray dashed line. Figure 7A The diagram illustrates the general timing relationship between the mixing of three wavelength pulses within a four-frame cycle and the readout cycle of the pixel array of the image sensor. In one embodiment, three monochromatic pulse light sources are present under the control of a controller. For example, a periodic sequence of monochromatic red, monochromatic green, and monochromatic blue exposure frames is captured using an RGBG pulse mode and combined into an sRGB image frame by an image signal processor pipeline.

[0195] exist Figure 7B In this process, different light intensities are achieved by modulating the power of the optical power or the power of the electromagnetic transmitter (which can be a laser or an LED transmitter), while keeping the pulse width or duration constant.

[0196] Figure 7CThis illustrates a scenario where both optical power and pulse width are modulated for greater flexibility. Partition cycling can utilize cyan, magenta, yellow (CMY), infrared, ultraviolet, hyperspectral, and fluorescence, employing invisible pulse sources mixed with visible pulse sources, as well as any other color space required to generate the image or approximating currently known or yet-to-be-developed desired video standards. It should also be understood that the system is capable of switching between operating color spaces to provide the desired image output quality.

[0197] Using the color space green-blue-green-red (e.g.) Figure 7D In the implementation shown, it may be desirable to pulse the luminance component more frequently than the chromaticity component, because users are generally more sensitive to differences in light intensity than to differences in light color. Examples of such implementations include... Figure 7D The monochrome image sensor shown utilizes this principle. Figure 7D In this configuration, green, which contains the most luminance information, can pulse more frequently or have greater intensity in the (GBGRGBGR…) scheme to obtain luminance data. This configuration will create a video stream with noticeably more detail, without creating and transmitting imperceptible data.

[0198] In one implementation, all three light sources are pulsed synchronously with light energy modulated to provide pure luminance information within the same exposure frame. The light energy can be modulated according to color conversion coefficients from the RGB color space to the YCbCr color space. It should be understood that the color conversion can be implemented according to any suitable standard such as ITU-R BT.709HD, ITU-R BT.601, ITU-R BT.2020, or any other suitable standard or formula. This conversion can be performed according to the ITU-R BT.709HD standard as follows:

[0199]

[0200] In addition to modulating luminance information, full-color images also require red and blue chrominance components. However, algorithms applied to the luminance component cannot be directly applied to the chrominance component because the algorithm is signed, as reflected in the fact that some RGB coefficients are negative. In one implementation, the luminance is increased to such that all final pulse energies are positive. As long as the color fusion processes in the image signal processor know the composition of the chrominance exposure frames, they can decode them by subtracting an appropriate amount of luminance from adjacent frames. The pulse energy ratio is given by the following formula:

[0201] Y=0.183·R+0.614·G+0.062·B

[0202] Cb=λ·Y-0.101·R-0.339·G+0.439·B

[0203] Cr=δ·Y+0.439·R-0.399·G-0.040·B

[0204] in

[0205]

[0206]

[0207] If the λ factor equals 0.552, the red and green components are canceled out. In this case, the blue hue information can be provided as pure blue light. Similarly, if the δ factor equals 0.650, the blue and green components are canceled out, and the red hue information can be provided as pure red light. This implementation is a convenient approximation for digital frame reconstruction.

[0208] In an implementation of white balance in the illumination domain, modulation is applied in addition to white balance modulation.

[0209] In one implementation, pulses that replicate weaker regions can be used to generate an output already adjusted for the weaker pulses. For example, blue lasers are considered less sensitive relative to silicon-based pixels and harder to generate than red or green light; therefore, they can be pulsed more frequently during frame cycles to compensate for the weakness of the light. These additional pulses can be performed continuously over time or by pulsed simultaneously using multiple lasers to produce the desired compensation effect. It should be noted that by pulses during the blanking period (the time during which the sensor does not read out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same type and simply focuses the light for the desired output. In another implementation, the maximum range of light pulses may vary from frame to frame. This is in... Figure 7E As shown, the light pulses are different from frame to frame. The sensor can be configured to be programmed with different blanking periods in repeating patterns of two, three, four, or n frames.

[0210] exist Figure 7E The diagram illustrates four distinct optical pulses, with pulse 1 repeating, for example, after pulse 4, and a pattern of four frames with different blanking periods. This technique can be used to place the most powerful partition on the smallest blanking period, thus allowing the weakest partition to have a wider pulse on a subsequent frame without increasing readout speed. The reconstructed frame can still have a regular frame-to-frame pattern because it consists of many pulse frames.

[0211] Figure 8A graphical representation of the delay or jitter between control signal 802 and the emission of electromagnetic radiation 804 is shown. In one embodiment, control signal 802 represents a signal provided to a driver of the transmitter. The driver is configured to cause transmitter 202 to emit electromagnetic radiation pulses. In one embodiment, the driver is a component of controller 204, or may be independent of and communicate with controller 204. In one embodiment, the driver is controller 204. In one embodiment, the driver is a component of transmitter 202 or communicates with transmitter 202. As shown, there is a duration delay t1 between the control signal 802 reaching its peak (i.e., turning on) and transmitter 202 emitting electromagnetic radiation 804. There is a duration delay t2 between the control signal 802 decreasing (i.e., turning off) and the end of electromagnetic radiation emission 804.

[0212] For example, delays t1 and t2 may include some constant delays as well as some non-constant variations caused by jitter in the transmitter's driver. For instance, a constant delay may exist when control signal 802 is transmitted to the driver, and when transmitter 202 actually emits electromagnetic radiation at emission 804. This delay can be very short and can be based on the time required for electrical communication to occur between the driver and the transmitter. Non-constant variations in delay can be a result of jitter in the transmitter's driver, in the controller 204, and / or in the transmitter itself.

[0213] The jitter experienced by a system or its components (such as the transmitter's driver) can be described by a value known as a jitter specification. A jitter specification is a numerical value that describes the amount or duration of jitter experienced by the system. Figure 8 In the example shown, delay t1 has a shorter duration than delay t2. In this example, delay t1 can represent the constant delay experienced after the start control signal 802 and the transmitter emits electromagnetic radiation at emission 804. The difference between t2 and t1 can represent the jitter experienced by the system. This value can be referred to as the jitter specification.

[0214] In one implementation, the jitter specification is a numerical value representing the amount of variation in a constant or predictable delay used to initiate or deactivate an electromagnetic radiation pulse (EMIP). In such implementations, the system experiences a constant, predictable delay between when the driver signals the transmitter to emit the EMIP and when the transmitter actually initiates the EMIP. Similarly, a constant, predictable delay may exist between when the transmitter should deactivate the EMIP and when the transmitter actually deactivates the EMIP. This constant, predictable delay does not represent the jitter specification. Rather, the jitter specification is the variation of this constant, predictable delay. Figure 8 In the example shown, the difference between times t2 and t1 represents a change in constant, predictable delay.

[0215] Jitter is beyond the user's control over the system. Jitter specification represents a quantity of unpredictable and non-constant time-varying variations present in the system. If the jitter specification is too large relative to an electromagnetic radiation pulse, a significant degrade in image quality or variation in image brightness will occur in the resulting exposure frames. For example, in a video endoscope system as discussed herein, a long jitter specification can cause exposure frames in different rows within the video stream to have different brightness. This results in flickering and overall quality degradation in the video stream. A long jitter specification can cause light to be emitted during the readout cycle 302 of the image sensor. If pulsed electromagnetic radiation occurs during readout cycle 302, significant variations will occur between pixels and pixel rows in the pixel readings, and this degrades the image quality in the resulting video stream.

[0216] In an exemplary implementation, controller 204 has a jitter specification of 10% of the duration of the electromagnetic radiation pulse. In this example, the pulse can vary from 90% to 110% of its desired duration. This can result in brightness variations of up to one-third between exposure frames or rows within the image frame of the video.

[0217] In one implementation, if the jitter specification has a duration longer than a threshold amount, the electromagnetic radiation pulse is limited for that duration to avoid overlapping into the readout period 302. Limiting the pulse duration may necessitate reducing the frame rate by increasing the time between captured exposure frames and / or increasing the duration of the blanking period 316. This can result in a reduction in image brightness, and this can further reduce the image sensor's ability to capture detailed images.

[0218] In one embodiment, if the jitter specification has a duration shorter than a threshold amount, the pulse sequence of transmitter 202 and the readout sequence 204 of the image sensor remain unchanged. In one embodiment, the threshold indicates that the jitter specification must be 1 microsecond or less. In one embodiment, the threshold indicates that the jitter specification must be 50 nanoseconds or less. In one embodiment, the threshold indicates that the jitter specification must be less than the time it takes for the image sensor to read out one row of the pixel array. In one embodiment, the threshold indicates that the jitter specification must be less than the time it takes for the image sensor to read out a single pixel of the pixel array. In one embodiment, the threshold indicates that the jitter specification may be less than or equal to 10% to 25% of the readout period 302 of the image sensor, or the time required for the image sensor to read out all valid pixels in the pixel array. For example, in such an embodiment, if the pixel array comprises 400 rows, the jitter specification must be less than or equal to the time required to read out 40 to 100 rows of the 400 rows in the pixel array. Therefore, the amount of variation in the light captured by the pixel array can be low enough to reduce image flicker and / or provide as much light as possible between readout periods 302.

[0219] In one implementation, jitter specifications are reduced (shortened) by implementing a higher clock rate or a more accurate clock in the driver of controller 204 or transmitter 202. Reduced jitter specifications and tolerances in the driver of transmitter 202 address issues of driver intolerance that lead to artifacts in the resulting video stream.

[0220] In one embodiment, the camera control unit (CCU) signals the controller 204 or transmitter 202 to avoid overlapping electromagnetic radiation pulses with the readout period 302 of the image sensor. The CCU can determine the signals to be sent to the controller 204 and / or transmitter 202 to avoid overlap with the readout of valid (i.e., not optically black) pixels in the pixel array. The CCU can maximize the duration of the temporal electromagnetic radiation emitted by the transmitter 202 without overlapping the readout period 302 of the image sensor.

[0221] Figure 9 A cross-section of an optical fiber bundle 900 used to carry electromagnetic radiation from transmitter 202 into a dark environment to illuminate the scene is shown. Figure 9 In the exemplary embodiment shown, the fiber bundle 900 includes seven fibers; however, it should be understood that the number of fibers is merely illustrative and any suitable number of fibers may be used outside the scope of this disclosure. The fiber bundle includes a central fiber 902 and a plurality of surrounding fibers 904.

[0222] In one embodiment, the total number of fibers is limited to reduce the cross-sectional area of ​​the fiber bundle 900. The fiber bundle 900 may include a suitable number of fibers to provide sufficient light dispersion while allowing a small cross-sectional area. This may be desirable because the cross-sectional area of ​​the endoscope lumen is critical in some applications that require a small endoscope. In one embodiment, the fiber bundle 900 may include 2 to 150 fibers. A smaller number of fibers reduces the cost and cross-sectional area required to carry the fiber bundle 900. However, a larger number of fibers improves redundancy. In one embodiment, the fiber bundle 900 includes 5 to 100 fibers, 5 to 50 fibers, or 7 to 15 fibers. In one embodiment, the fiber bundle includes seven fibers, such as... Figure 9 As shown.

[0223] When the fiber bundle 900 has a small number of fibers, it may be desirable for each fiber to receive the same amount of electromagnetic radiation and the same amount of electromagnetic radiation of a specific wavelength. For example, if the electromagnetic radiation is primarily transmitted through the central fiber 902, the central fiber 902 will receive most of the electromagnetic radiation, and the scene will be illuminated in a colored or unevenly bright manner. Furthermore, if more light enters one fiber than another, the total amount (power) of electromagnetic radiation that can be carried in the fiber bundle 900 will be reduced. For example, if electromagnetic radiation above a certain energy level or intensity is supplied to a fiber, the fiber may have melting limits that can cause it to melt or otherwise fail. Therefore, if the electromagnetic radiation is distributed more evenly across the fibers, it is possible to increase the power and illumination of the scene.

[0224] In one embodiment, transmitter 202 mixes electromagnetic radiation of two or more wavelengths before providing electromagnetic radiation to fiber bundle 900. This can be achieved when transmitter 202 includes two or more independent laser beams for emitting electromagnetic radiation of different wavelengths. Transmitter 202 may include, for example, a first laser beam for emitting a first wavelength and a second laser beam for emitting a second wavelength. Transmitter 202 may mix the electromagnetic radiation such that light from the first laser beam and light from the second laser beam enter the jumper waveguide (or another waveguide) at the same or substantially the same angle. The same or substantially the same angle can be achieved by positioning the laser beams at the same angle relative to each other. In one embodiment, a dichroic mirror allows the same or substantially the same angle by reflecting electromagnetic radiation of one wavelength while being transparent to another wavelength. In one embodiment, transmitter 202 includes a diffuser, mixing rod, lens, or other optical element to mix the light before it enters fiber bundle 900.

[0225] In one embodiment, emitter 202 provides a uniformly distributed light intensity to the waveguide. The peak intensity of the light in the region where it collects light for the waveguide may be substantially equal to or close to the average intensity of the light in that region. The light provided to the collection region may have a top-hat profile, such that each fiber collects and / or receives light of the same or similar intensity. Emitter 202 can provide or approximate a top-hat profile by providing a laser beam at an angle to the surface of the collection region. For example, emitter 202 may include a Gaussian or other non-constant intensity profile. By angulating the laser beam relative to the collection region, the Gaussian profile is flattened into a more constant profile or a top-hat profile. Lenses, diffusers, mixing rods, or the like can be used to generate the top-hat profile.

[0226] Figure 10The top-hat profile 1002 and the Gaussian profile 1004 are illustrated graphically. The horizontal axis represents the horizontal distance, and the vertical axis represents the light intensity. The row labeled 1006 in the figure represents the boundary or width of the collection area 1006 of the fiber bundle 900. The row labeled 1008 in the figure represents the fusion level 1008 for the fiber or other waveguide.

[0227] Using a Gaussian profile 1004, most of the electromagnetic radiation is directed to the central fiber 902. With most of the energy located in the central fiber 902, the remaining surrounding fibers 904 can remain well below the fusing level 1008. For example, using the Gaussian profile 1004, an increase in the total energy can cause the central fiber 902 to significantly exceed the fusing level 1008, while multiple surrounding fibers 904 remain well below the fusing level 1008.

[0228] Using the top-hat profile 1002, all fibers carry the same energy level. This energy level can be close to or below the fusion break level 1008. For example, using the top-hat profile 1002, the total energy carried by the fiber bundle 900 can be significantly increased because the fiber bundle 900 can be concentrated near the fusion break level 1008 without the risk of fusing any individual fiber.

[0229] Figure 10 This illustrates how implementing a top-hat profile 1002 can provide more energy before any single fiber reaches the fusing level 1008. For example, Gaussian profile 1004 and top-hat profile 1002 can provide the same amount of watts to the fiber bundle 900, while top-hat profile 1002 can still significantly increase the amount of watts before reaching the fusing level 1008. Therefore, a significant improvement in the total amount of energy delivered can be achieved using plastic fibers. In some cases, an increase of 50% or more in the watts carried by the fiber bundle 900 can be achieved by implementing top-hat profile 1002. In one embodiment, the plastic fibers may have a fusing energy level for optical / electromagnetic energy emitted by one or more transmitters, above which would damage the plastic fibers, wherein the optical energy propagates across multiple plastic fibers to allow the fiber bundle 900, including the plastic fibers, to carry a greater amount of energy without reaching the fusing level 1008 of any single fiber.

[0230] In one embodiment, the top-hat profile 1002 and the Gaussian profile 1004 are combined by the transmitter 202 for use with the plastic fiber bundle 900. The transmitter 202 and / or the jumper waveguide may not include a plastic waveguide. However, the transmitter 202 may blend the Gaussian profile 1004 with the top-hat profile 1002 to allow use with the plastic fiber bundle 900 at the cavity waveguide. In one embodiment, the blended top-hat profile 1002 allows for greater power delivery, considering the losses that can occur when electromagnetic radiation is moved between different materials (e.g., from diffuser to glass fiber, to plastic fiber, and / or back to glass fiber or diffuser). This greater power delivery can offset losses in previous or subsequent transitions, ensuring that sufficient light is still delivered to illuminate the scene.

[0231] Figure 11 This is a side view showing the output from the fiber bundle 1102 compared to the camera's field of view. In one embodiment, the plastic fiber has a numerical aperture of 0.63, with a field of view of 100 degrees, as shown by dashed line 1106. The glass fiber has a numerical aperture of 0.87, with a field of view of 120 degrees, as shown by solid line 1104. However, light emitted within the field of view has an approximately Gaussian profile within a light cone smaller than that field of view. For example, almost all the light used for the plastic fiber can be within an 80-degree cone, as shown by dashed line 1108. Therefore, the central region of the exposure frame may be too bright and the edges too dark.

[0232] Figure 12 It shows relative to Figure 11 The output shown is a side view of the output from fiber bundle 1202, exhibiting a more uniform light distribution. Figure 12 In the illustrated embodiment, uniform light distribution is achieved by aiming the light off the ends of the fibers in the fiber bundle 1202. Aiming the fibers away from the center widens the cones in the field of view and eliminates light loss at the output ends. One end of each fiber can be held in a desired position to distribute light, where the combination of light cones from the fibers provides uniform illumination. The fiber bundle 1202 includes multiple fibers and rows 1204 indicating the orientation of the cones output from individual fibers. In one embodiment, a fixing device such as a physical mold or a sheet with holes holds the ends of the fibers in a desired orientation. The fibers can be oriented to the optimal orientation for uniform illumination of the scene. The ends of the fibers in the fiber bundle 1202 can be located near the distal end of an endoscope and can be directed to propagate light around an area centered on the focal point or camera lens axis.

[0233] Figure 13This is a side view showing the output of the fiber bundle 1302, which transitions from plastic fiber 1304 to glass fiber 1306 at connector 1308. In this embodiment, the cavity waveguide includes plastic fiber 1304, which then transitions to glass fiber 1306 at or near the output. Glass fiber 1306 typically has a higher numerical aperture and a wider field of view than plastic fiber 1304. Therefore, as shown by light cone 1310, a wider and more uniform light energy distribution is achieved. Light passing through plastic fiber 1304 is guided to glass fiber 1306 via connector 1308. This coupling can occur within the handpiece unit of the endoscope's cavity. Connector 1308 can be positioned in the handpiece or cavity to limit the amount of glass fiber 1306 used. Moving from plastic fiber 1304 to glass fiber 1306 via a tapered member in the handpiece or cavity produces the same field of view as conventional endoscopes. However, compared to the targeting implementation (which does not experience optical loss at the output), optical loss can be significant, such as about 25%.

[0234] Figure 14 This is a side view showing the light output from the fiber bundle 1402 using a diffuser 1408. In this embodiment, the cavity waveguide includes plastic fibers 1404, which then transition to the diffuser 1408 at or near the output. The diffuser 1408 can include any suitable optical diffuser, such as a hybrid rod. Exemplary diffusers include holographic diffusers. The diffuser 1408 at the output can produce a larger field of view compared to glass fibers. However, the diffuser 1408 is less efficient, such as compared to... Figure 12 The aiming implementation shown is approximately 40% to 60% more efficient.

[0235] Plastic fibers are generally cheaper than glass fibers. This lower price can lead to significant savings in manufacturing lighting systems. Significant cost savings can be achieved because glass is only used for short distances near the output, or not at all.

[0236] In one implementation, a single fiber replaces the fiber bundle. The single fiber can be larger than the typical fibers constituting the fiber bundle, allowing it to handle greater power than a smaller fiber bundle for the same cross-sectional area. The single fiber can extend from the console and through the cavity to provide light to the interior of the body or other light-deficient environments. The single fiber can operate as an intracavity waveguide extending from transmitter 202 or a jumper waveguide and through the cavity. Electromagnetic radiation can be directly supplied by transmitter 202 to the single fiber, which has a top-cap profile.

[0237] Because plastic fibers can have numerical apertures of only 0.63 or 0.65, most electromagnetic radiation can be emitted at angles of only 70 or 80 degrees. At the output of a single fiber, a diffuser can be positioned to propagate the output light and create more uniform illumination within the camera's field of view. In one implementation, the type of diffuser or its presence can be based on the field of view used by the camera. For example, laparoscopic surgery may allow a narrower field of view, such as 70 degrees, while arthroscopic surgery may use a wider field of view, such as 110 degrees. Therefore, a diffuser may be used for arthroscopy, while it may not be necessary for laparoscopy.

[0238] It should be understood that the implementation scheme for outputting electromagnetic radiation (light) may include Figures 11 to 14 The embodiments shown are combinations of one or more of the embodiments. For example, plastic fibers can be converted to glass fibers, and glass fibers can be designed to provide more uniform and improved lighting.

[0239] Figure 15 This is a schematic flowchart illustrating an exemplary method 1500 for providing light to an imaging scene in a low-light environment. Method 1500 can be provided by a lighting system (such as...) Figure 1 (System 100) executes.

[0240] Method 1500 begins, and at 1502, the image sensor generates and reads out pixel data for an image from the image sensor based on light received by the image sensor, wherein the time length for reading out a row of pixel data includes a row readout length. At 1504, the transmitter emits light to illuminate the scene observed by the image sensor. At 1506, the driver drives the transmitter to emit, wherein the driver includes a jitter specification less than or equal to the row readout length. At 1508, the controller controls the driver to drive the transmitter to generate light pulses between readout cycles of the image sensor.

[0241] Figure 16 This is a schematic flowchart illustrating an exemplary method 1600 for providing light to an imaging scene in a low-light environment. Method 1600 can be provided by a lighting system (such as...) Figure 1 (System 100) executes.

[0242] Method 1600 begins, and the first and second transmitters emit light comprising a first wavelength and a second wavelength at 1602. Multiple optical fibers at 1604 guide the light generated by the first and second transmitters into the scene within the endoscopic environment. At 1606a, the multiple optical fibers receive substantially equal amounts of light (mixed light) from the first and second transmitters at each of the multiple optical fibers.

[0243] Figures 17A to 17CLight sources 1700 with multiple emitters are shown. These emitters may alternatively be referred to as “laser beams,” wherein each emitter / laser beam may operate independently of the other emitters / laser beams and / or pulse different zones or wavelengths of the electromagnetic spectrum. Light sources 1700 may be collectively referred to herein as “emitters.” The multiple emitters include a first emitter 1702, a second emitter 1704, and a third emitter 1706. Additional emitters may be included, as discussed further below. Emitters 1702, 1704, and 1706 may include one or more laser emitters that emit light with different wavelengths. For example, the first emitter 1702 may emit a wavelength consistent with blue laser light, the second emitter 1704 may emit a wavelength consistent with green laser light, and the third emitter 1706 may emit a wavelength consistent with red laser light. For example, the first emitter 1702 may include one or more blue lasers, the second emitter 1704 may include one or more green lasers, and the third emitter 1706 may include one or more red lasers. Lasers 1702, 1704, and 1706 emit laser beams toward a collection region 1708, which may be a waveguide, a lens, or a device for collecting light and / or directing it toward the waveguide (such as...). Figure 2 The jumper waveguide 206 or cavity waveguide 210 provides the location of other optical components for light.

[0244] In one specific implementation, transmitters 1702, 1704, and 1706 emit electromagnetic radiation at hyperspectral wavelengths. Certain hyperspectral wavelengths can penetrate tissue and allow physicians to "see through" foreground tissue to identify chemical processes, structures, compounds, biological processes, etc., located behind the foreground tissue. Hyperspectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc., known to have specific spectral responses.

[0245] In specific embodiments where reagents or dyes that aid in the identification of certain tissues, structures, chemical reactions, biological processes, etc., have been administered to a patient, emitters 1702, 1704, and 1706 may emit wavelengths intended to fluoresce the reagents or dyes. Such wavelengths may be determined based on the reagents or dyes administered to the patient. In such embodiments, the emitters may need to be highly precise in order to emit the desired wavelengths to fluoresce or activate certain reagents or dyes.

[0246] In one implementation, transmitters 1702, 1704, and 1706 emit laser mapping patterns for mapping the topology of a scene and / or for calculating the dimensions and distances between objects in the scene. In one implementation, the endoscopic imaging system is used in conjunction with multiple tools such as scalpels, retractors, clamps, etc. In such implementations, each of transmitters 1702, 1704, and 1706 can emit a laser mapping pattern such that the laser mapping pattern is projected individually onto each tool. In such implementations, the laser mapping data of each tool can be analyzed to identify the distances between the tool and other objects in the scene.

[0247] exist Figure 17B In this implementation, transmitters 1702, 1704, and 1706 each deliver laser light to collection region 1708 at different angles. The change in angle can cause a change in the position of electromagnetic energy within the output waveguide. For example, if light enters the fiber bundle (glass or plastic) immediately at collection region 1708, the changing angle can cause different amounts of light to enter different fibers. For example, the angle can cause an intensity variation across collection region 1708. Furthermore, light from different transmitters may not be uniformly mixed, so some fibers may receive different amounts of different colors of light. Variations in the color or intensity of light in different fibers can lead to suboptimal lighting of the scene. For example, variations in delivered light or light intensity can cause this in both the scene and the captured image.

[0248] In one embodiment, an intervening optics element may be placed between the fiber bundle and emitters 1702, 1704, 1706 to mix light of different colors (wavelengths) before it enters the fiber or other waveguide. Exemplary intervening optics elements include diffusers, mixing rods, one or more lenses, or other optical components for mixing light such that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. This mixing may result in the same color in each fiber; however, in some embodiments, it may still result in different total brightness delivered to different fibers. In one embodiment, the intervening optics element may also propagate or uniformly distribute light over the collection area such that each fiber carries the same total amount of light (e.g., the light may be diffused in a top-hat profile). Diffusers or mixing rods may result in light loss.

[0249] Although the collection area 1708 is in Figure 17A The term 1708 is used to refer to physical components, but the collection region 1708 may simply be the region that delivers light from transmitters 1702, 1704, and 1706. In some cases, the collection region 1708 may include optical components such as diffusers, mixing rods, lenses, or any other intermediary optical components located between transmitters 1702, 1704, 1706, and the output waveguide.

[0250] Figure 17C An embodiment of a light source 1700 is shown, having emitters 1702, 1704, 1706 that provide light to a collection region 1708 at the same or substantially the same angle. The light is provided at an angle substantially perpendicular to the collection region 1708. The light source 1700 includes a plurality of dichroic mirrors, including a first dichroic mirror 1710, a second dichroic mirror 1712, and a third dichroic mirror 1714. Dichroic mirrors 1710, 1712, and 1714 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 1714 may reflect blue laser light provided by a third emitter, while being transparent to red and green light provided by the first emitter 1702 and the second emitter 1704, respectively. The second dichroic mirror 1712 may be transparent to red light from the first emitter 1702 but reflect green light from the second emitter 1704. If other colors or wavelengths are included, dichroic mirrors can be selected to reflect light corresponding to at least one emitter and be transparent to the other emitters. For example, a third dichroic mirror 1714 reflects light from a third emitter 1706 but is transparent to emitters "behind" it, such as a first emitter 1702 and a second emitter 1704. In embodiments where dozens or hundreds of emitters exist, each dichroic mirror can reflect light from its corresponding emitter and the emitters in front of it, while being transparent to the emitters behind it. This allows dozens or hundreds of emitters to project electromagnetic energy into the collection area 1708 at substantially the same angle.

[0251] Because these dichroic mirrors allow other wavelengths to transmit or pass through, each of these wavelengths can reach the collection area 1708 from the same angle and / or at the same center point or focal point. Providing light from the same angle and / or the same focal point / center point significantly improves reception and color mixing at the collection area 1708. For example, a particular fiber can receive different colors in proportions identical to the proportions of transmission / reflection by emitters 1702, 1704, 1706 and mirrors 1710, 1712, 1714. Figure 17B Compared to the previous implementation, this significantly improves light mixing at the collection region. In one implementation, any of the optical components discussed herein may be used at the collection region 1708 to collect light before it is supplied to the fiber or fiber bundle.

[0252] Figure 17CAn embodiment of a light source 1700 is shown having emitters 1702, 1704, 1706 that also provide light to a collection region 1708 at the same or substantially the same angle. For example, the light incident on the collection region 1708 is deflected from the vertical. Angle 1716 indicates the angle of deflection from the vertical. In one embodiment, laser emitters 1702, 1704, 1706 may have a Gaussian cross-sectional intensity profile. As previously mentioned, an improved distribution of optical energy between fibers can be achieved by forming a flatter or cap-shaped intensity profile. In one embodiment, as angle 1716 increases, the intensity across the collection region 1708 approaches a cap-shaped profile. For example, by increasing angle 1716 until the profile is sufficiently flat, a cap-shaped profile can even approximate a non-flat output beam. A cap-shaped profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intermediary optical components located between emitters 1702, 1704, 1706 and the output waveguide, fiber, or fiber bundle.

[0253] Figure 18 This is a schematic diagram showing a single optical fiber 1802 output via a diffuser 1804 at the output point. In one embodiment, the optical fiber 1802 has a diameter of 500 micrometers, a numerical aperture of 0.65, and emits a light cone 1806 of approximately 70 or 80 degrees without the diffuser 1804. Using the diffuser 1804, the light cone 1806 can have an angle of approximately 110 or 120 degrees. The light cone 1806 can be the majority of the area where all light arrives and is uniformly distributed. The diffuser 1804 allows for a more uniform distribution of electromagnetic energy in the scene observed by the image sensor.

[0254] In one embodiment, the cavity waveguide 210 includes a single plastic or glass optical fiber of approximately 500 micrometers. Plastic fibers are less expensive, but their width can be reduced through coupling, diffusing, or other methods to allow the fiber to carry a sufficient amount of light into the scene. For example, a smaller fiber may not be able to carry as much light or power as a larger fiber. The cavity waveguide 210 may include a single or multiple optical fibers. The cavity waveguide 210 may receive light directly from a light source or via a jumper waveguide. A diffuser may be used to widen the light output 206 to obtain the desired field of view for the image sensor 214 or other optical components.

[0255] Although Figures 17A to 17CThree emitters are shown, but in some embodiments, the number of emitters can range from one to hundreds or more. The emitters may emit light of different wavelengths or spectra, and this light can be used to continuously cover desired portions of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra). The emitters may be configured to emit visible light such as red, green, and blue light, and may also be configured to emit hyperspectral emission of electromagnetic radiation, fluorescence excitation wavelengths for fluorescing reagents, and / or laser mapping modes for calculating parameters and distances between objects in a scene.

[0256] Figure 19 A portion of an electromagnetic spectrum 1900 divided into twenty distinct sub-spectrums is shown. The number of sub-spectrums is merely exemplary. In at least one embodiment, spectrum 1900 may be divided into hundreds of sub-spectrums, each having a wavelet band. The spectrum may extend from the infrared spectrum 1902, through the visible spectrum 1904, and into the ultraviolet spectrum 1906. Each sub-spectrum has a wavelet band 1908 covering a portion of spectrum 1900. Each wavelet may be defined by an upper wavelength and a lower wavelength.

[0257] Hyperspectral imaging includes imaging information from across the electromagnetic spectrum 1900. Hyperspectral pulses of electromagnetic radiation may include multiple sub-pulses spanning one or more portions of the electromagnetic spectrum 1900 or the entire electromagnetic spectrum 1900. A hyperspectral pulse of electromagnetic radiation may include a single wavelength partition of electromagnetic radiation. The resulting hyperspectral exposure frame includes information sensed by the pixel array following the hyperspectral pulse of electromagnetic radiation. Therefore, a hyperspectral exposure frame may include data from any suitable partition of the electromagnetic spectrum 1900, and may include multiple exposure frames from multiple partitions of the electromagnetic spectrum 1900. In one embodiment, a hyperspectral exposure frame includes multiple hyperspectral exposure frames such that a combined hyperspectral exposure frame includes data from the entire electromagnetic spectrum 1900.

[0258] In one embodiment, for each sub-spectrum, at least one emitter (such as a laser emitter) is included in a light source (such as light source 202, 1700) to provide complete and continuous coverage of the entire spectrum 1900. For example, the light source used to provide coverage of the illustrated sub-spectrum may include at least 20 different emitters, with at least one emitter for each sub-spectrum. In one embodiment, each emitter covers a 40 nm band. For example, one emitter may emit light in a band from 500 nm to 540 nm, while another emitter may emit light in a band from 540 nm to 580 nm. In another embodiment, the emitters may cover bands of other sizes, depending on the type of emitter available or the imaging requirements. For example, multiple emitters may include a first emitter covering a band from 500 nm to 540 nm, a second emitter covering a band from 540 nm to 640 nm, and a third emitter covering a band from 640 nm to 650 nm. Each transmitter can cover different segments of the electromagnetic spectrum, ranging from far-infrared, mid-infrared, near-infrared, visible, near-ultraviolet, and / or far-ultraviolet. In some cases, multiple transmitters of the same type or wavelength may be included to provide sufficient output power for imaging. The number of transmitters required for a particular waveband may depend on the monochromatic sensor's sensitivity to the waveband and / or the power output capability of the transmitters in that waveband.

[0259] The bandwidth and coverage provided by the emitter can be selected to provide any desired combination of spectra. For example, continuous coverage of the spectrum using a very small bandwidth (e.g., 10 nm or less) allows for highly selective hyperspectral and / or fluorescence imaging. This bandwidth allows for the selective emission of excitation wavelengths of one or more specific fluorescent reagents. Additionally, the bandwidth allows for the selective emission of portions of hyperspectral electromagnetic radiation for the identification of specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths originate from an emitter that can be selectively activated, extreme flexibility is achieved in fluorescing one or more specific fluorescent reagents during examination. Furthermore, extreme flexibility is achieved in identifying one or more objects or processes via hyperspectral imaging. Therefore, more fluorescence and / or hyperspectral information can be obtained in less time and in a single examination, which would otherwise require multiple examinations and be delayed due to dye application or staining.

[0260] Figure 20This is a schematic diagram illustrating the timing of emission and readout for generating an image. The solid lines represent the readout period (peak 2002) and blanking period (valley) for capturing a series of exposure frames 2004 to 2014. This series of exposure frames 2004 to 2014 may include a series of repeated exposure frames that can be used to generate laser mapping data, hyperspectral data, and / or fluorescence data that can be overlaid on an RGB video stream. In one embodiment, a single image frame includes information from multiple exposure frames, one of which includes red image data, another includes green image data, and yet another includes blue image data. Additionally, a single image frame may include one or more of hyperspectral image data, fluorescence image data, and laser mapping data. These multiple exposure frames are combined to produce a single image frame. The single image frame is an RGB image with hyperspectral imaging data. The series of exposure frames includes a first exposure frame 2004, a second exposure frame 2006, a third exposure frame 2008, a fourth exposure frame 2010, a fifth exposure frame 2012, and an Nth exposure frame 2026.

[0261] Additionally, hyperspectral image data, fluorescence image data, and laser mapping data can be used in combination to identify key tissues or structures and further measure their dimensions. For example, hyperspectral image data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, and cancerous tissue. The location and identification of key structures can be received from the corresponding system and can also be used to generate the topology of the key structures using laser mapping data. For example, the corresponding system determines the location of a cancerous tumor based on hyperspectral imaging data. Since the location of the cancerous tumor is known based on the hyperspectral imaging data, its topology and distances can then be calculated based on the laser mapping data. This example is also applicable when identifying cancerous tumors or other structures based on fluorescence imaging data.

[0262] In one implementation, each exposure frame is generated based on at least one pulse of electromagnetic energy. The electromagnetic energy pulse is reflected and detected by an image sensor and subsequently read out in a subsequent readout (2002). Thus, each blanking period and readout results in an exposure frame for a specific electromagnetic energy spectrum. For example, a first exposure frame 2004 may be generated based on the spectrum of a first one or more pulses 2016, a second exposure frame 2006 may be generated based on the spectrum of a second one or more pulses 2018, a third exposure frame 2008 may be generated based on the spectrum of a third one or more pulses 2020, a fourth exposure frame 2010 may be generated based on the spectrum of a fourth one or more pulses 2022, a fifth exposure frame 2012 may be generated based on the spectrum of a fifth one or more pulses, and an Nth exposure frame 2026 may be generated based on the spectrum of an Nth one or more pulses 2026.

[0263] Pulses 2016 to 2026 may include energy from a single emitter or a combination of two or more emitters. For example, the spectrum may be selected to be included within a single readout cycle or multiple exposure frames 2004 to 2014 for desired examination or detection of a specific tissue or condition. According to one embodiment, one or more pulses may include visible spectral light for generating RGB or black-and-white images, while one or more additional pulses are emitted to sense a spectral response to electromagnetic radiation at hyperspectral wavelengths. For example, pulse 2016 may include red light, pulse 2018 may include blue light, and pulse 2020 may include green light, while the remaining pulses 2022 to 2026 may include wavelengths and spectra for detecting specific tissue types, fluorescing reagents, and / or mapping scene topology. Again, pulses in a single readout cycle may include spectra generated by multiple different emitters (e.g., different segments of the electromagnetic spectrum) that can be used to detect a specific tissue type. For example, if a combination of wavelengths results in a pixel having a value above or below a threshold, that pixel may be classified as corresponding to a specific type of tissue. Each frame can also be used to narrow down the type of tissue present at that pixel (e.g., and every pixel in the image) to provide a very specific classification of the tissue and / or the state (disease / health) of the tissue based on the spectral response of the tissue and / or the presence of fluorescent reagents at the tissue site.

[0264] Multiple frames from 2004 to 2014 are shown as readout periods of varying lengths and pulses of varying lengths or intensities. The blanking period, pulse length, or intensity, etc., can be selected based on the monochromatic sensor's sensitivity to a specific wavelength, the transmitter's power output capability, and / or the waveguide's carrying capacity.

[0265] In one implementation, dual image sensors can be used to acquire three-dimensional images or video feeds. Three-dimensional inspection allows for a better understanding of the three-dimensional structure of the inspected area and the mapping of different tissue or material types within that area.

[0266] In one exemplary embodiment, a fluorescent reagent is provided to a patient, and the fluorescent reagent is configured to attach to cancer cells. The fluorescent reagent is known to fluoresce when irradiated by a specific zone of electromagnetic radiation. The relaxation wavelength of the fluorescent reagent is also known. In this exemplary embodiment, the patient is imaged using an endoscopic imaging system as discussed herein. The endoscopic imaging system pulses zones of light with red, green, and blue wavelengths to generate an RGB video stream of the patient's interior. Additionally, the endoscopic imaging system pulses the electromagnetic radiation excitation wavelength of the fluorescent reagent applied to the patient. In this example, the patient has cancer cells, and the fluorescent reagent attaches to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength of the fluorescent reagent, the fluorescent reagent fluoresces and emits a relaxation wavelength. If cancer cells are present in the scene imaged by the endoscopic imaging system, the fluorescent reagent will also be present in the scene and, due to the emission of the excitation wavelength, will emit its relaxation wavelength after fluorescing. The endoscopic imaging system senses the relaxation wavelength of the fluorescent reagent, thereby sensing the presence of the fluorescent reagent in the scene. Because the fluorescent reagent is known to attach to cancer cells, the presence of the fluorescent reagent also indicates the presence of cancer cells in the scene. The endoscopic imaging system identifies the location of cancer cells within a scene. It also emits laser mapping pulses to generate the scene's topology and calculate the dimensions of objects within it. The location of cancer cells (as identified by fluorescence imaging data) can be combined with topological and dimensional information calculated based on the laser mapping data. Therefore, the precise location, size, dimensions, and topological structure of cancer cells can be identified. This information can be provided to a physician to aid in cancer cell removal. Additionally, this information can be provided to a robotic surgical system, enabling the system to remove cancer cells.

[0267] In another exemplary implementation, an endoscopic imaging system is used to image a patient to identify quantitative diagnostic information about the patient's histopathology. In this example, the patient is suspected of or known to have a disease that can be tracked using hyperspectral imaging to observe the progression of the disease in the patient's tissues. The endoscopic imaging system pulses partitions of red, green, and blue wavelengths of light to generate an RGB video stream of the patient's interior. Additionally, the endoscopic imaging system pulses one or more hyperspectral wavelengths of light, allowing the system to "see through" some tissue and generate images of the tissue affected by the disease. The endoscopic imaging system senses reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue, thereby identifying the location of the diseased tissue within the patient's body. The endoscopic imaging system may also emit a laser mapping pulse scheme to generate the topology of the scene and calculate the dimensions of objects within the scene. The location of the diseased tissue (as identified by the hyperspectral imaging data) can be combined with topological and dimensional information calculated using the laser mapping data. Therefore, the precise location, size, dimensions, and topology of the diseased tissue can be identified. This information can be provided to a physician to assist in the removal, imaging, or study of the diseased tissue. In addition, this information can be provided to the robotic surgical system so that the system can remove diseased tissue.

[0268] Figure 21A and Figure 21B Perspective and side views, respectively, are shown of a specific embodiment of a monolithic sensor 2100 according to the teachings and principles of this disclosure, which has multiple pixel arrays for generating three-dimensional images. Such an embodiment may be desirable for three-dimensional image capture, wherein two pixel arrays 2102 and 2104 can be offset during use. In another embodiment, the first pixel array 2102 and the second pixel array 2104 may be dedicated to receiving electromagnetic radiation within a predetermined wavelength range, wherein the first pixel array is dedicated to electromagnetic radiation within a different wavelength range than the second pixel array.

[0269] Figure 22A and Figure 22BPerspective and side views of a specific embodiment of an imaging sensor 2200 constructed on multiple substrates are shown. As shown, multiple pixel columns 2204 forming the pixel array are located on a first substrate 2202, and multiple circuit columns 2208 are located on a second substrate 2206. The electrical connections and communications between a pixel column and its associated or corresponding circuit column are also shown. In one embodiment, the image sensor may have a pixel array separate from all or most of the supporting circuitry, and it may otherwise be manufactured such that its pixel array and supporting circuitry are on a single, monolithic substrate / chip. This disclosure may use at least two substrates / chips, which will be stacked together using a three-dimensional stacking technique. The first of the two substrates / chips 2202 may be fabricated using an image CMOS process. The first substrate / chip 2202 may consist solely of a pixel array, or it may consist of a pixel array surrounded by a limited circuitry. The second or subsequent substrate / chip 2206 may be fabricated using any process, and is not necessarily derived from an image CMOS process. The second substrate / chip 2206 can be, but is not limited to, a high-density digital process for integrating various and multiple functions into a very limited space or area on the substrate / chip, a mixed-mode or analog process for integrating, for example, precise analog functions, an RF process for enabling wireless capabilities, or a MEMS (Micro-Electro-Mechanical Systems) process for integrating MEMS devices. The image CMOS substrate / chip 2202 can be stacked with the second or subsequent substrate / chip 2206 using any three-dimensional technology. The second substrate / chip 2206 can support the majority or most of the circuitry that may be additionally implemented in the first image CMOS chip 2202 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increases the overall system area while keeping the pixel array size constant and optimized to the maximum extent possible. Electrical connections between the two substrates / chips can be accomplished via interconnects, which can be bonding leads, lugs, and / or TSVs (Through Silicon Vias).

[0270] Figure 23A and Figure 23B Perspective and side views of a specific embodiment of an imaging sensor 2300 having multiple pixel arrays for generating three-dimensional images are shown, respectively. The three-dimensional image sensor can be constructed on multiple substrates and may include multiple pixel arrays and other associated circuitry, wherein multiple pixel columns 2302a forming a first pixel array and multiple pixel columns 2302b forming a second pixel array are located on respective substrates 2308a and 2308b, and multiple circuit columns 2306a and 2306b are located on a separate substrate 2304. Electrical connections and communications between pixel columns and associated or corresponding circuit columns are also shown.

[0271] The multiple pixel arrays can simultaneously sense information and combine information from the multiple pixel arrays to generate a three-dimensional image. In one embodiment, the endoscopic imaging system includes two or more pixel arrays that can be deployed to generate a three-dimensional image. The endoscopic imaging system may include a transmitter for emitting electromagnetic radiation pulses during the blanking period of the pixel arrays. The pixel arrays can be synchronized such that optical black pixels are read out simultaneously for the two or more pixel arrays (i.e., a blanking period occurs). The transmitter can emit electromagnetic radiation pulses to charge each of the two or more pixel arrays. The two or more pixel arrays can simultaneously read out their respective charged pixels such that the readout periods of the two or more pixel arrays occur simultaneously or substantially simultaneously. In one embodiment, the endoscopic imaging system includes multiple transmitters, each transmitter being individually synchronized with one or more of the multiple pixel arrays. Information from the multiple pixel arrays can be combined to generate three-dimensional image frames and video streams.

[0272] It should be understood that the teachings and principles of this disclosure can be applied to reusable device platforms, limited-use device platforms, reconfigurable device platforms, or single-use / disposable device platforms without departing from the scope of this disclosure. It should be understood that in a reusable device platform, the end user is responsible for cleaning and sterilizing the device. In a limited-use device platform, the device can be used a predetermined number of times before becoming inoperable. Typically, new devices are sterilized before delivery and should be cleaned and sterilized by the end user before any other use if intended for other purposes. In a reconfigurable device platform, a third party can reprocess (e.g., clean, package, and sterilize) a single-use device for additional use at a lower cost than a new unit. In a single-use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0273] Example

[0274] The following embodiments relate to preferred features of other implementations:

[0275] Example 1 is a system. The system includes a transmitter for emitting electromagnetic radiation pulses. The transmitter includes a first transmitter for emitting electromagnetic radiation pulses of a first wavelength. The transmitter includes a second transmitter for emitting electromagnetic radiation pulses of a second wavelength. The system includes an image sensor comprising a pixel array for sensing reflected electromagnetic radiation. The system includes a controller in electronic communication with the transmitter and the image sensor. The system causes at least a portion of the electromagnetic radiation pulses emitted by the transmitter to include one or more of the following: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm; electromagnetic radiation having a wavelength of about 565 nm to about 585 nm; or electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0276] Example 2 is the system according to Example 1, wherein: the first transmitter emits the electromagnetic radiation pulse of the first wavelength at the first dichroic mirror, and the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength to multiple optical fibers; the second transmitter emits the electromagnetic radiation pulse of the second wavelength at the second dichroic mirror, and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength to the multiple optical fibers; and the first dichroic mirror is transparent to the electromagnetic radiation of the second wavelength.

[0277] Example 3 is a system according to any one of Examples 1 to 2, wherein the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength into the plurality of optical fibers at an angle offset relative to the perpendicularity of the plurality of optical fibers, and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength into the plurality of optical fibers at an angle offset relative to the perpendicularity of the plurality of optical fibers.

[0278] Example 4 is a system according to any one of Examples 1 to 3, wherein: the first dichroic mirror reflects the electromagnetic radiation pulse of the first wavelength into the plurality of optical fibers at an angle substantially perpendicular to the first transmitter; and the second dichroic mirror reflects the electromagnetic radiation pulse of the second wavelength into the plurality of optical fibers through the first dichroic mirror at an angle substantially perpendicular to the second transmitter.

[0279] Example 5 is a system according to any one of Examples 1 to 4, wherein: the transmitter further includes a third transmitter for emitting an electromagnetic radiation pulse of a third wavelength at a third dichroic mirror, the third dichroic mirror reflecting the electromagnetic radiation pulse of the third wavelength into the plurality of optical fibers; the third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength into the plurality of optical fibers through the first dichroic mirror; the first dichroic mirror and the second dichroic mirror are transparent to the electromagnetic radiation of the third wavelength; the third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength into the plurality of optical fibers at an angle substantially perpendicular to the third transmitter; and the third dichroic mirror reflects the electromagnetic radiation pulse of the third wavelength into the plurality of optical fibers at an angle offset from perpendicular.

[0280] Example 6 is a system according to any one of Examples 1 to 5, further comprising: an optical fiber bundle, wherein the transmitter transmits the electromagnetic radiation pulse into the optical fiber bundle; wherein the optical fiber bundle comprises plastic fibers and glass fibers, wherein the plastic fibers and the glass fibers are coupled near the output of the optical fiber bundle.

[0281] Example 7 is a system according to any one of Examples 1 to 6, further comprising an intermediary optical component, wherein the electromagnetic radiation pulse passes through the intermediary optical component before entering the fiber bundle.

[0282] Example 8 is a system according to any one of Examples 1 to 7, wherein the intervening optical component includes one or more of a diffuser or a mixing rod.

[0283] Example 9 is a system according to any one of Examples 1 to 8, further comprising: an optical fiber bundle comprising a plurality of plastic optical fibers, wherein the transmitter transmits the electromagnetic radiation pulse into the optical fiber bundle; and an intermediary optical component, wherein the electromagnetic radiation pulse passes through the intermediary optical component before entering the optical fiber bundle; wherein the intermediary optical component comprises a plurality of glass fibers.

[0284] Example 10 is a system according to any one of Examples 1 to 9, further comprising: an optical fiber bundle, wherein the transmitter emits the electromagnetic radiation pulse into the optical fiber bundle; and a diffuser disposed at the distal end of the optical fiber bundle; wherein the diffuser provides an optical cone having an angle between 110 degrees and 120 degrees or an angle between 70 degrees and 80 degrees.

[0285] Example 11 is a system according to any one of Examples 1 to 10, further comprising a third transmitter for emitting electromagnetic radiation pulses of a third wavelength and a fourth transmitter for emitting electromagnetic radiation pulses of a fourth wavelength, wherein: the electromagnetic radiation of the first wavelength emitted by the first transmitter is red light; the electromagnetic radiation of the second wavelength emitted by the second transmitter is blue light; the electromagnetic radiation of the third wavelength emitted by the third transmitter is green light; and the electromagnetic radiation of the fourth wavelength emitted by the fourth transmitter is a hyperspectral wavelength, the hyperspectral wavelength including one or more of the following: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, electromagnetic radiation having a wavelength of about 565 nm to about 585 nm, or electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; wherein the fourth transmitter for emitting the hyperspectral wavelength includes one or more independent lasers for emitting electromagnetic radiation of different hyperspectral wavelengths.

[0286] Example 12 is a system according to any one of Examples 1 to 11, wherein the pixel array of the image sensor senses reflected electromagnetic radiation during the readout period of the pixel array to generate the plurality of exposure frames, wherein the readout period includes the time period during which effective pixels in the pixel array are read.

[0287] Example 13 is a system according to any one of Examples 1 to 12, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a hyperspectral wavelength for inducing a spectral response, wherein the hyperspectral wavelength includes one or more of the following: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or electromagnetic radiation having a wavelength of about 565 nm to about 585 nm, and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0288] Example 14 is a system according to any one of Examples 1 to 13, wherein the transmitter is configured to emit a plurality of electromagnetic radiation sub-pulses during the pulse duration, the plurality of sub-pulses having a sub-duration shorter than the pulse duration.

[0289] Example 15 is a system according to any one of Examples 1 to 14, wherein one or more electromagnetic radiation pulses emitted by the transmitter include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

[0290] Example 16 is a system according to any one of Examples 1 to 15, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a hyperspectral emission that causes the image sensor to generate a hyperspectral exposure frame, and wherein the controller is configured to provide the hyperspectral exposure frame to a corresponding hyperspectral system that determines the location of a key tissue structure within the scene based on the hyperspectral exposure frame.

[0291] Example 17 is a system according to any one of Examples 1 to 16, wherein the hyperspectral emission includes: electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0292] Example 18 is a system according to any one of Examples 1 to 17, wherein the controller is further configured to: receive the location of the key tissue structure from the corresponding hyperspectral system; generate an overlay frame including the location of the key tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the key tissue structure within the scene.

[0293] Example 19 is a system according to any one of Examples 1 to 18, wherein sensing the electromagnetic radiation reflected by the pixel array includes generating a laser mapping exposure frame by sensing the reflected electromagnetic radiation generated by the transmitter pulses the laser mapping mode, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system, the corresponding laser mapping system determining the topology of the scene and / or the size of one or more objects within the scene; provide the location of the key tissue structure to the corresponding laser mapping system; and receive the topology and / or size of the key tissue structure from the corresponding laser mapping system.

[0294] Example 20 is a system according to any one of Examples 1 to 19, wherein the key structure includes one or more of a nerve, ureter, blood vessel, artery, blood flow or tumor.

[0295] Example 21 is a system according to any one of Examples 1 to 20, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a fluorescence excitation wavelength that causes the image sensor to generate a fluorescence exposure frame, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding fluorescence system, the corresponding fluorescence system determining the location of a key tissue structure within the scene based on the fluorescence exposure frame.

[0296] Example 22 is a system according to any one of Examples 1 to 21, wherein the fluorescence excitation emission includes one or more of the following: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0297] Example 23 is a system according to any one of Examples 1 to 22, wherein the controller is further configured to: receive the location of the key tissue structure from the corresponding fluorescence system; generate an overlay frame including the location of the key tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the key tissue structure within the scene.

[0298] Example 24 is a system according to any one of Examples 1 to 23, wherein sensing the electromagnetic radiation reflected by the pixel array includes generating a laser mapping exposure frame by sensing the reflected electromagnetic radiation generated by the transmitter pulses the laser mapping mode, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system, the corresponding laser mapping system determining the topology of the scene and / or the size of one or more objects within the scene; provide the location of the key tissue structure to the corresponding laser mapping system; and receive the topology and / or size of the key tissue structure from the corresponding laser mapping system.

[0299] Example 25 is a system according to any one of Examples 1 to 24, wherein the key structure includes one or more of a nerve, ureter, blood vessel, artery, blood flow or tumor.

[0300] Example 26 is a system according to any one of Examples 1 to 25, wherein the controller is configured to synchronize the timing of the electromagnetic radiation pulses during the blanking period of the image sensor, wherein the blanking period corresponds to the time between the readout of the last row of active pixels in the pixel array and the start of the next subsequent readout of the active pixels in the pixel array.

[0301] Example 27 is a system according to any one of Examples 1 to 26, wherein two or more electromagnetic radiation pulses emitted by the transmitter generate two or more instances of reflected electromagnetic radiation, the two or more instances of reflected electromagnetic radiation are sensed by the pixel array to generate two or more exposure frames, the two or more exposure frames being combined to form an image frame.

[0302] Example 28 is a system according to any one of Examples 1 to 27, wherein the image sensor includes a first image sensor and a second image sensor, such that the image sensor is capable of generating a three-dimensional image.

[0303] Example 29 is a system according to any one of Examples 1 to 28, wherein the transmitter is configured to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each exposure frame corresponds to an electromagnetic radiation pulse.

[0304] Example 30 is a system according to any one of Examples 1 to 29, wherein the electromagnetic radiation pulse is emitted in a pattern of electromagnetic radiation of different wavelengths, and wherein the transmitter repeats the pattern of electromagnetic radiation of different wavelengths.

[0305] Example 31 is a system according to any one of Examples 1 to 30, wherein at least a portion of the electromagnetic radiation pulse includes a red light wavelength, a green light wavelength, a blue light wavelength, and a hyperspectral wavelength, such that electromagnetic radiation reflected by the pixel array corresponding to each of the red light wavelength, the green light wavelength, the blue light wavelength, and the hyperspectral wavelength can be processed to generate a red-green-blue (RGB) image frame including superimposed hyperspectral imaging data, wherein the hyperspectral wavelength of the electromagnetic radiation includes: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0306] Example 32 is a system according to any one of Examples 1 to 31, wherein at least a portion of the electromagnetic radiation pulse includes luminance emission, red hue emission, blue hue emission, and hyperspectral emission, such that electromagnetic radiation sensed by the pixel array corresponding to reflections of each of the luminance emission, the red hue emission, the blue hue emission, and the hyperspectral emission can be processed to generate a YCbCr image frame including superimposed hyperspectral imaging data, wherein the hyperspectral emission of the electromagnetic radiation includes: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0307] Example 33 is a system according to any one of Examples 1 to 32, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a fluorescence excitation wavelength for causing the reagent to fluoresce, wherein the fluorescence excitation wavelength includes one or more of the following: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0308] Example 34 is a system according to any one of Examples 1 to 33, wherein at least a portion of the electromagnetic radiation pulse includes a red light wavelength, a green light wavelength, a blue light wavelength, and a fluorescence excitation wavelength, such that electromagnetic radiation reflected by the pixel array corresponding to each of the red light wavelength, the green light wavelength, the blue light wavelength, and the fluorescence excitation wavelength can be processed to generate a red-green-blue (RGB) image frame comprising a stack of fluorescence imaging data, wherein the electromagnetic radiation of the fluorescence wavelength includes electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0309] Example 35 is a system according to any one of Examples 1 to 34, wherein at least a portion of the electromagnetic radiation pulse includes luminance emission, red hue emission, blue hue emission, and fluorescence excitation emission, such that electromagnetic radiation sensed by the pixel array corresponding to reflections of each of the luminance emission, the red hue emission, the blue hue emission, and the fluorescence excitation emission can be processed to generate a YCbCr image frame comprising a stack of fluorescence imaging data, wherein the electromagnetic radiation of the fluorescence wavelength includes electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0310] Example 36 is a system according to any one of Examples 1 to 35, further comprising a single optical fiber, wherein the transmitter transmits the electromagnetic radiation pulse into the single optical fiber.

[0311] Example 37 is a system according to any one of Examples 1 to 36, wherein the pixel array is a two-dimensional array of independent pixels, each of which is capable of detecting electromagnetic radiation of any wavelength.

[0312] Example 38 is a system according to any one of Examples 1 to 37, further comprising a filter for filtering electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0313] Example 39 is a system according to any one of Examples 1 to 38, further comprising a filter for filtering electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0314] Example 40 is a system according to any one of Examples 1 to 39, wherein the electromagnetic sensor is a photodiode.

[0315] Example 41 is a system according to any one of Examples 1 to 40, wherein sensing reflected electromagnetic radiation by the pixel array includes: generating a laser mapping exposure frame by sensing reflected electromagnetic radiation generated by the transmitter pulsed by the laser mapping mode, wherein the laser mapping exposure frame includes information for determining real-time measurements, the information including one or more of the following: the distance from the endoscope to the object; the angle between the endoscope and the object; or surface topology information about the object.

[0316] Example 42 is a system according to any one of Examples 1 to 41, wherein the laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 cm.

[0317] Example 43 is a system according to any one of Examples 1 to 42, wherein the laser mapping exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

[0318] Example 44 is a system according to any one of Examples 1 to 43, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter includes multiple tool-specific laser mapping modes for each of a plurality of tools in the scene.

[0319] Example 45 is a system according to any one of Examples 1 to 44, wherein the laser mapping mode emitted by the transmitter includes a first output and a second output that are independent of each other, wherein the first output is used for illumination and the second output is used for tool tracking.

[0320] Example 46 is a system according to any one of Examples 1 to 45, wherein the first transmitter is a first laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the first wavelength, and the second transmitter is a second laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the second wavelength.

[0321] Example 47 is a system according to any one of Examples 1 to 46, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to one or more electromagnetic radiation pulses emitted by the transmitter.

[0322] Example 48 is a system according to any one of Examples 1 to 47, wherein the fiber bundle comprises 2 to 150 fibers.

[0323] Example 49 is a system according to any one of Examples 1 to 48, wherein the third transmitter is a third laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the third wavelength.

[0324] Example 50 is a system according to any one of Examples 1 to 49, wherein the fourth transmitter is a fourth laser beam comprising a plurality of lasers for emitting the electromagnetic radiation pulse of the fourth wavelength.

[0325] Example 51 is a system according to any one of Examples 1 to 50, wherein the transmitter includes one or more hyperspectral transmitters for emitting electromagnetic radiation pulses of hyperspectral wavelengths to elicit a spectral response.

[0326] Example 52 is a system according to any one of Examples 1 to 51, wherein each of the one or more hyperspectral emitters includes a laser beam, and the laser beam includes a plurality of lasers.

[0327] Example 53 is a system according to any one of Examples 1 to 52, wherein the transmitter further includes an optical element for mixing electromagnetic radiation pulses before entering the fiber bundle, wherein the optical element includes one or more of a diffuser, a mixing rod, or a lens.

[0328] Example 54 is a system according to any one of Examples 1 to 53, further comprising a dichroic mirror for reflecting electromagnetic radiation of blue light wavelength.

[0329] Example 55 is a system according to any one of Examples 1 to 54, further comprising a dichroic mirror for reflecting electromagnetic radiation of green wavelength.

[0330] Example 56 is a system according to any one of Examples 1 to 55, further comprising a dichroic mirror for reflecting electromagnetic radiation of red wavelength.

[0331] Example 57 is a system according to any one of Examples 1 to 56, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 513 nm to about 545 nm.

[0332] Example 58 is a system according to any one of Examples 1 to 57, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 565 nm to about 585 nm.

[0333] Example 59 is a system according to any one of Examples 1 to 58, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0334] Example 60 is a system according to any one of Examples 1 to 59, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0335] Example 61 is a system according to any one of Examples 1 to 60, further comprising a dichroic mirror for reflecting electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0336] Example 62 is a system according to any one of Examples 1 to 61, further comprising a dichroic mirror for reflecting electromagnetic radiation of a certain wavelength band, wherein the dichroic mirror is transparent to electromagnetic radiation of other wavelengths.

[0337] Example 63 is a system according to any one of Examples 1 to 62, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a red light wavelength at least.

[0338] Example 64 is a system according to any one of Examples 1 to 63, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a green light wavelength.

[0339] Example 65 is a system according to any one of Examples 1 to 64, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a blue light wavelength at least.

[0340] Example 66 is a system according to any one of Examples 1 to 65, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 513 nm to about 545 nm.

[0341] Example 67 is a system according to any one of Examples 1 to 66, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 565 nm to about 585 nm.

[0342] Example 68 is a system according to any one of Examples 1 to 67, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0343] Example 69 is a system according to any one of Examples 1 to 68, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0344] Example 70 is a system according to any one of Examples 1 to 69, further comprising a dichroic mirror for reflecting electromagnetic radiation of a specific wavelength, wherein the dichroic mirror is transparent to electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0345] Example 71 is a system according to any one of Examples 1 to 70, wherein the emitter includes a plurality of laser emitters, and wherein the plurality of laser emitters have a Gaussian cross-sectional intensity profile.

[0346] Example 72 is a system according to any one of Examples 1 to 71, wherein the emitter includes a plurality of laser emitters, and wherein the plurality of laser emitters have a flat or substantially flat intensity profile.

[0347] Example 73 is a system according to any one of Examples 1 to 72, wherein the emitter includes a plurality of laser emitters, and wherein the plurality of laser emitters have a top-hat shaped intensity profile.

[0348] Example 74 is a system according to any one of Examples 1 to 73, wherein the first transmitter and the second transmitter are designed to collect an area such that the electromagnetic radiation pulse of the first wavelength and the electromagnetic radiation pulse of the second wavelength are mixed in the collection area and received by the fiber bundle.

[0349] Example 75 is a system according to any one of Examples 1 to 74, further comprising an intermediary optical element disposed between the transmitter and the fiber bundle, wherein the intermediary optical element is configured to mix the emissions from the first transmitter and the second transmitter before the emissions reach the fiber bundle.

[0350] Example 76 is a system according to any one of Examples 1 to 75, further comprising an intermediary optical element disposed between the transmitter and the fiber bundle, the intermediary optical element being configured to uniformly mix the independent emission of electromagnetic radiation from the first transmitter and the second transmitter before reaching the fiber bundle.

[0351] It should be understood that the various features disclosed herein offer significant advantages and advancements in the art. The following claims are examples of some of those features.

[0352] In the specific embodiments described above, for the purpose of simplification, the various features of this disclosure are concentrated in a single embodiment. The method of this disclosure should not be construed as implying an intention that the disclosure protected by the claims requires more features than expressly listed in each claim. Rather, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0353] It should be understood that any feature of the above-described arrangement, embodiments, and implementations may be combined in a single implementation that includes a combination of features obtained from any of the disclosed arrangement, embodiments, and implementations.

[0354] It should be understood that the above-described configuration is merely an exemplary application of the principles of this disclosure. Many modifications and alternative configurations can be devised by those skilled in the art without departing from the spirit and scope of this disclosure, and the appended claims are intended to cover such modifications and configurations.

[0355] Therefore, when this disclosure is illustrated and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications can be made without departing from the principles and ideas set forth herein, including but not limited to changes in size, material, shape, form, function and operation, assembly and use.

[0356] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to execute one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. Those skilled in the art will understand that components may have different names. This document is not intended to distinguish between components that are different in name rather than function.

[0357] For illustrative and descriptive purposes, the specific embodiments described above have been provided. These specific embodiments are not intended to be exhaustive or to limit this disclosure to the specific forms disclosed. Many modifications and changes can be made to this disclosure based on the foregoing teachings. Furthermore, it should be noted that any or all of the foregoing alternative embodiments can be used in any desired combination to form further hybrid embodiments of this disclosure.

[0358] Furthermore, while specific embodiments of this disclosure have been described and illustrated, this disclosure is not limited to the particular forms or arrangements of components as described and illustrated. The scope of this disclosure will be defined by the appended claims, any future claims filed herein and in different applications, and their equivalents.

Claims

1. A hyperspectral imaging system, comprising: A transmitter for emitting electromagnetic radiation pulses, wherein at least a portion of the electromagnetic radiation pulses includes one or more wavelengths suitable for color imaging, wherein the transmitter includes a plurality of hyperspectral sources, each hyperspectral source being configured to emit a wavelength band of electromagnetic spectrum capable of evoking a spectral response from a structure, tissue, or other material present in a light-deficient environment, wherein the wavelength band of electromagnetic spectrum includes infrared wavelengths, visible spectrum, ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands, and not merely based on the three wavelengths of the visible spectrum that are resolvable by the human eye; An image sensor, comprising an array of pixels for sensing reflected electromagnetic radiation; and A controller, which is in electronic communication with the transmitter and the image sensor; wherein the plurality of hyperspectral sources include: A first hyperspectral emitter is configured to emit electromagnetic radiation pulses within a first wavelength range at a first dichroic mirror, the first dichroic mirror being configured to reflect the pulses within the first wavelength range to a collection region. A second hyperspectral emitter is configured to emit electromagnetic radiation pulses in a second wavelength range at a second dichroic mirror, the second dichroic mirror being configured to reflect the pulses in the second wavelength range to a collection region. A third hyperspectral emitter is configured to emit electromagnetic radiation pulses in a third wavelength range at a third dichroic mirror, the third dichroic mirror being configured to reflect the pulses in the third wavelength range to a collection region. Specifically, the first, second, and third dichroic mirrors each reflect electromagnetic radiation pulses from the first, second, and third hyperspectral emitters, respectively, at an angle offset relative to the vertical collection region, back into the collection region; and The image sensor is configured to generate a plurality of exposure frames, each of the plurality of exposure frames corresponding to one or more electromagnetic radiation pulses emitted by the transmitter, and wherein at least a portion of the plurality of exposure frames includes: Hyperspectral exposure frames corresponding to pulses from at least two of a plurality of hyperspectral sources, wherein the hyperspectral exposure frames are the result of a spectral response sensed by the pixel array; and A color image frame, the color image frame corresponding to reflected electromagnetic radiation sensed by the pixel array in response to an electromagnetic radiation pulse containing one or more wavelengths suitable for color imaging; The hyperspectral exposure frame is superimposed on the color image frame.

2. The hyperspectral imaging system according to claim 1, wherein: The first dichroic mirror is transparent to electromagnetic radiation in the second wavelength range, and wherein: The third dichroic mirror reflects electromagnetic radiation pulses within a third wavelength range through the first dichroic mirror into the collection area; or The first and second dichroic mirrors are transparent to electromagnetic radiation in the third wavelength range.

3. The hyperspectral imaging system according to claim 1, in, The collection area includes an optical fiber bundle, wherein the transmitter emits the electromagnetic radiation pulse into the optical fiber bundle; The optical fiber bundle includes plastic fibers and glass fibers, wherein the plastic fibers and the glass fibers are coupled near the output of the optical fiber bundle.

4. The hyperspectral imaging system according to claim 3, wherein, The collection area also includes an intermediary optical component, wherein the electromagnetic radiation pulse passes through the intermediary optical component before entering the fiber bundle.

5. The hyperspectral imaging system of claim 4, wherein the intervening optical component comprises one or more of a diffuser or a mixing rod.

6. The hyperspectral imaging system according to claim 1, further comprising: An optical fiber bundle comprising a plurality of plastic optical fibers, wherein the transmitter transmits the electromagnetic radiation pulse into the optical fiber bundle; as well as Intermediate optical component, wherein the electromagnetic radiation pulse passes through the intermediate optical component before entering the fiber bundle; The intermediate optical component comprises multiple glass fibers.

7. The hyperspectral imaging system according to claim 1, further comprising: An optical fiber bundle, wherein the transmitter emits the electromagnetic radiation pulse into the optical fiber bundle; as well as A diffuser, wherein the diffuser is disposed at the distal end of the fiber bundle; The diffuser provides a light cone having an angle between 110 and 120 degrees or between 70 and 80 degrees.

8. The hyperspectral imaging system according to claim 1 further includes a second transmitter, a third transmitter, and a fourth transmitter, wherein, The second, third, and fourth transmitters are configured to emit electromagnetic radiation pulses comprising one or more wavelengths suitable for color imaging, wherein: The electromagnetic radiation emitted by the second transmitter is red light; The electromagnetic radiation emitted by the third transmitter is blue light; The electromagnetic radiation emitted by the fourth transmitter is green light; and The electromagnetic radiation in the first wavelength range emitted by the first hyperspectral emitter includes electromagnetic radiation with wavelengths from 513 nm to 545 nm, the electromagnetic radiation in the second wavelength range emitted by the second hyperspectral emitter includes the electromagnetic radiation with wavelengths from 565 nm to 585 nm, and the electromagnetic radiation in the third wavelength range emitted by the third hyperspectral emitter includes electromagnetic radiation with wavelengths from 900 nm to 1000 nm.

9. The hyperspectral imaging system of claim 1, wherein the first hyperspectral emitter is a first laser beam comprising a plurality of lasers for emitting electromagnetic radiation pulses in the first wavelength range, and the second hyperspectral emitter is a second laser beam comprising a plurality of lasers for emitting electromagnetic radiation pulses in the second wavelength range.

10. The hyperspectral imaging system of claim 1, wherein the pixel array of the image sensor senses reflected electromagnetic radiation during the readout period of the pixel array to generate the plurality of exposure frames, wherein the readout period includes a time period during which effective pixels in the pixel array are read out.

11. The hyperspectral imaging system of claim 1, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a hyperspectral wavelength for inducing a spectral response, wherein the hyperspectral wavelength includes one or more of the following: The electromagnetic radiation of the first hyperspectral emitter in a first wavelength range has a wavelength of 513 nm to 545 nm, and the electromagnetic radiation of the third hyperspectral emitter in a third wavelength range has a wavelength of 900 nm to 1000 nm; or The electromagnetic radiation of the second hyperspectral emitter in the second wavelength range has a wavelength of 565 nm to 585 nm, and the electromagnetic radiation of the third hyperspectral emitter in the third wavelength range has a wavelength of 900 nm to 1000 nm.

12. The hyperspectral imaging system of claim 1, wherein the transmitter is configured to emit a plurality of electromagnetic radiation subpulses having a sub-duration shorter than the pulse duration during the pulse duration.

13. The hyperspectral imaging system of claim 1, wherein one or more of the electromagnetic radiation pulses emitted by the transmitter comprise electromagnetic radiation emitted simultaneously as a single pulse or a single sub-pulse at two or more wavelengths.

14. The hyperspectral imaging system of claim 1, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is a hyperspectral emission that causes the image sensor to generate a hyperspectral exposure frame, and wherein the controller is configured to provide the hyperspectral exposure frame to a corresponding system that determines the location of key tissue structures within a scene based on the hyperspectral exposure frame.

15. The hyperspectral imaging system of claim 14, wherein the hyperspectral emission comprises: The electromagnetic radiation of the first hyperspectral emitter in a first wavelength range has a wavelength of 513 nm to 545 nm, and the electromagnetic radiation of the third hyperspectral emitter in a third wavelength range has a wavelength of 900 nm to 1000 nm; or The electromagnetic radiation of the second hyperspectral emitter in the second wavelength range has a wavelength of 565 nm to 585 nm, and the electromagnetic radiation of the third hyperspectral emitter in the third wavelength range has a wavelength of 900 nm to 1000 nm.

16. The hyperspectral imaging system of claim 15, wherein the controller is further configured to: Receive the location of the key organizational structure from the corresponding system; Generate a stacked frame including the location of the key organizational structure; and The overlay frame is combined with a color image frame depicting the scene to indicate the location of the key organizational structure within the scene.

17. The hyperspectral imaging system of claim 16, wherein the key tissue structure comprises one or more of a nerve, ureter, blood vessel, artery, blood flow, or tumor.

18. The hyperspectral imaging system of claim 1, wherein the controller is configured to synchronize the timing of the electromagnetic radiation pulses during a blanking period of the image sensor, wherein the blanking period corresponds to the time between the readout of the last row of valid pixels in the pixel array and the start of the next subsequent readout of valid pixels in the pixel array.

19. The hyperspectral imaging system of claim 1, wherein two or more electromagnetic radiation pulses emitted by the transmitter generate two or more instances of reflected electromagnetic radiation, the two or more instances being sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.

20. The hyperspectral imaging system of claim 1, wherein the transmitter is configured to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each exposure frame corresponds to one or more electromagnetic radiation pulses.

21. The hyperspectral imaging system of claim 1, wherein the electromagnetic radiation pulses are emitted in a pattern of electromagnetic radiation at different wavelengths, and wherein the transmitter repeats the pattern of electromagnetic radiation at different wavelengths.

22. The hyperspectral imaging system according to claim 1, wherein the electromagnetic radiation pulses suitable for color imaging include red light wavelengths, green light wavelengths, and blue light wavelengths, wherein, Overlaying a hyperspectral exposure frame onto a color image frame includes processing reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red, green, blue, and hyperspectral wavelengths to generate a superimposed red-green-blue (RGB) image frame including hyperspectral imaging data, wherein the electromagnetic radiation at the hyperspectral wavelengths includes: The electromagnetic radiation of the first hyperspectral emitter in a first wavelength range has a wavelength of 513 nm to 545 nm, and the electromagnetic radiation of the third hyperspectral emitter in a third wavelength range has a wavelength of 900 nm to 1000 nm; or The electromagnetic radiation of the second hyperspectral emitter in the second wavelength range has a wavelength of 565 nm to 585 nm, and the electromagnetic radiation of the third hyperspectral emitter in the third wavelength range has a wavelength of 900 nm to 1000 nm.

23. The hyperspectral imaging system of claim 1, wherein at least a portion of the electromagnetic radiation pulse comprises luminance emission, red hue emission, blue hue emission, and hyperspectral emission, such that electromagnetic radiation reflected by the pixel array corresponding to each of the luminance emission, the red hue emission, the blue hue emission, and the hyperspectral emission can be processed to generate a YCbCr image frame comprising superimposed hyperspectral imaging data, wherein the hyperspectral emission of the electromagnetic radiation comprises: The electromagnetic radiation of the first hyperspectral emitter in a first wavelength range has a wavelength of 513 nm to 545 nm, and the electromagnetic radiation of the third hyperspectral emitter in a third wavelength range has a wavelength of 900 nm to 1000 nm; or The electromagnetic radiation of the second hyperspectral emitter in the second wavelength range has a wavelength of 565 nm to 585 nm, and the electromagnetic radiation of the third hyperspectral emitter in the third wavelength range has a wavelength of 900 nm to 1000 nm.

Images