Controlling integrated energy of laser pulses in a laser mapping imaging system

Abstract

Controlling integrated energy of light pulses in a laser mapping imaging system is disclosed. A system is disclosed that includes an emitter for emitting pulses of electromagnetic radiation and an image sensor including an array of pixels for sensing reflected electromagnetic radiation. The system includes an electromagnetic sensor for sensing energy emitted by the emitter. The system includes a controller configured to synchronize timing of the emitter and the image sensor. The system causes at least a portion of the pulses of electromagnetic radiation emitted by the emitter to be a laser mapping pattern.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to digital imaging, and in particular to laser mapping imaging in light deficient environments. BACKGROUND

[0002] Advances in technology have provided advances in medical imaging capabilities. Endoscopes can be used to view inside the body and inspect the interior of organs or cavities of the body. Endoscopes are used to investigate symptoms in a patient, confirm a diagnosis, or provide medical treatment. Medical endoscopes can be used to view a variety of body systems and portions, such as the gastrointestinal tract, the respiratory tract, the urinary tract, the abdominal cavity, and the like. Endoscopes can also be used for surgical procedures, such as orthopedic procedures, procedures performed on joints or bones, procedures performed on the nervous system, procedures performed within the abdominal cavity, and the like.

[0003] In some cases of endoscopic imaging, it can be beneficial or necessary to view space in color. Digital color images include at least three layers or “color channels” that cumulatively form an image with a range of hues. Each of the color channels measures the intensity and hue of a band of the light spectrum. Typically, digital color images include color channels for the red, green, and blue spectrum bands (this can be referred to as a red-green-blue or RGB image). Each of the red, green, and blue color channels includes luminance information for the red, green, or blue spectrum band. The luminance information of the individual red, green, and blue layers is combined to produce a color image. Because color images are composed of individual layers, conventional digital camera image sensors include a color filter array that allows red, green, and blue visible wavelengths to hit selected pixel sensors. Each individual pixel sensor element is sensitive to red, green, or blue wavelengths, and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image. The at least three different types of pixel sensors occupy a significant amount of physical space, such that a complete pixel array cannot fit in the small distal tip of an endoscope.

[0004] Because conventional image sensors cannot fit in the distal tip of an endoscope, the image sensor is traditionally located in the handpiece unit of the endoscope, which is held by the endoscope operator and is not placed inside the body cavity. In such endoscopes, light is transmitted along the length of the endoscope from the handpiece unit to the distal tip of the endoscope. This configuration has significant limitations. Endoscopes with this configuration are delicate, and can easily become misaligned or damaged when they are bumped or impacted during routine use. This can significantly degrade the quality of the image, and the endoscope needs to be frequently repaired or replaced.

[0005] Conventional endoscopes with image sensors placed in the handpiece unit are further limited to capturing only color images. However, in some implementations, it can be desirable to capture images with laser mapping image data in addition to color image data. Laser mapping imaging can capture surface shapes of objects and landscapes and measure distances between objects within a scene. In some embodiments, it can be desirable to measure distances and surface shapes within a body cavity during an endoscopic imaging procedure.

[0006] However, the application of laser mapping technology known in the art generally requires highly specialized equipment that can not be available for a variety of applications. Further, laser mapping technology provides a limited field of view of the environment and must generally be used in conjunction with multiple independent systems. In the context of medical endoscopic imaging procedures, all sensors must fit within a small physical area within a body cavity. In some cases, the geographic area is particularly small and can only accommodate a very small tip of an endoscope. Thus, medical endoscopes known in the art are necessarily small and cannot accommodate multiple different imaging and ranging systems. Accordingly, it is desirable to develop an endoscopic imaging system that is capable of generating laser mapping data in small spaces such as a body cavity.

[0007] In light of the foregoing, systems, methods, and apparatus for laser mapping imaging in light deficient environments are described herein. Such systems, methods, and apparatus can provide multiple data sets for identifying key structures within the body and providing precise and valuable information about a body cavity. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present disclosure is described with reference to the following non-limiting and non-exhaustive specific embodiments, in which like elements are referred to with like numerals throughout the various figures of the drawing, unless otherwise described. Aspects of the disclosure will become apparent to those skilled in the art from the following description, in which:

[0009] FIG. 1 is a schematic of a system for digital imaging in light deficient environments with paired emitters and pixel arrays;

[0010] FIG. 2 is a system for providing illumination to a light deficient environment for endoscopic imaging;

[0011] FIG. 2A is a schematic of complementary system hardware;

[0012] FIGS. 3A-3D is an illustration of an operational cycle of a sensor for constructing an exposure frame;

[0013] FIG. 4A is an illustration of the operation of an embodiment of an electromagnetic emitter;

[0014] FIG. 4Bto change the duration and magnitude of the emitted electromagnetic pulses to provide an illustration of exposure control;

[0015] FIG. 5 to combine the operation cycle of the sensor, the electromagnetic emitter, and the emitted electromagnetic pulses of the embodiments of the present disclosure; FIGS. 3A-4B

[0016] FIG. 6A to illustrate a method for recording a video with full-spectrum light over a time period from t(0) to t(l);

[0017] FIG. 6B to illustrate a process for recording a video by pulsing a segmented spectrum of light over a time period from t(0) to t(l);

[0018] FIGS. 7A-7E to illustrate a method for recording video frames of both full-spectrum light and segmented spectrum light over a certain time interval;

[0019] FIG. 8 to illustrate an illumination system for use in conjunction with a pulsed imaging system, wherein the illumination system includes a photodiode for measuring energy emitted by a laser module;

[0020] FIG. 9 to illustrate a laser beam including an electromagnetic sensor in communication with an operational amplifier for measuring energy emitted by a laser unit;

[0021] FIG. 10 to illustrate an embodiment of an emitter including a plurality of laser beams, each laser beam including an electromagnetic sensor for measuring energy emitted by a laser unit;

[0022] FIG. 11 to illustrate an embodiment of a digital imaging system including an endoscope device;

[0023] FIG. 12 to illustrate a mode reconstruction process for generating an RGB image with laser mapping data overlaid thereon from a segmented spectrum of pulsed light;

[0024] FIGS. 13A-13C to illustrate a light source having a plurality of emitters;

[0025] FIG. 14 to illustrate a single optical fiber output via a diffuser at an output to illuminate a scene in a light deficient environment;

[0026] FIG. 15 to illustrate a portion of the electromagnetic spectrum divided into a plurality of different sub-spectra that can be emitted by emitters of a light source in accordance with the principles and teachings of the present disclosure; ​

[0027] FIG. 16 is a schematic diagram showing the timing of emission and readout for generating an image frame comprising a plurality of exposure frames produced by different partitions of pulsed light;

[0028] FIG. 17 shows an imaging system comprising a single cut filter for filtering electromagnetic radiation of a plurality of wavelengths;

[0029] FIG. 18 shows an imaging system comprising a plurality of cut filters for filtering electromagnetic radiation of a plurality of wavelengths;

[0030] FIG. 19 shows an exemplary laser mapping pattern that can be pulsed by an imaging system;

[0031] FIG. 20A and FIG. 20B shows an implementation having a plurality of pixel arrays for producing three-dimensional images in accordance with the principles and teachings of the present disclosure;

[0032] FIG. 21A and FIG. 21B shows perspective and side views, respectively, of an implementation of an imaging sensor built on a plurality of substrates, where a plurality of pixel columns forming a pixel array are located on a first substrate and a plurality of circuit columns are located on a second substrate, and showing electrical connections and communication between a column of pixels and its associated or corresponding column of circuitry; and

[0033] FIG. 22A and FIG. 22B shows perspective and side views, respectively, of an implementation of an imaging sensor having a plurality of pixel arrays for generating three-dimensional images, where the plurality of pixel arrays and the image sensor are built on a plurality of substrates. DETAILED DESCRIPTION

[0034] Systems, methods, and apparatuses for digital imaging that can be primarily applicable to medical applications such as medical endoscopic imaging are disclosed herein. One embodiment of the present disclosure is an endoscopic system for laser mapping and color imaging in light deficient environments. Such methods, systems, and computer-based products disclosed herein provide imaging or diagnostic capabilities for medical robotic applications, such as the use of robots for performing imaging procedures, surgical procedures, and the like.

[0035] One embodiment of the present disclosure is an imaging system having a light source for providing pulsed illumination in a light deficient environment. In this embodiment, the total energy emitted by the light source is controlled and maintained to a specified tolerance. The light source can be in communication with a camera control unit (CCU) such that the CCU controls the power level, enable / disable, duration, and power level of the light source. The light source can include digital light sources such as lasers and light emitting diodes, and can also include analog light sources.

[0036] In one embodiment, during operation of the imaging system, a PID (proportional, integral, and derivative) control algorithm is implemented to ensure that the captured scene maintains a desired video exposure level, maximizing the dynamic range of the image sensor or achieving a desired scene response as desired by the end user. The PID control algorithm can be generally referred to herein as automatic shutter control (ASC). In some embodiments, each light pulse is proportionally adjusted based on a calculated error measurement, and the error measurement is calculated by comparing a desired exposure level to a measured exposure level. The measured exposure level can be calculated using the average pixel value of all pixels or some portion of the pixels in the image sensor. The ASC can adjust the light pulse to change the duration and / or intensity of the light source. This ensures that a desired set value is obtained within a certain specified time. The imaging system can be implemented in an endoscopic device, where the image sensor is disposed in a spatially limited area in the distal tip of the endoscope.

[0037] Conventional endoscopes are designed such that the image sensor is placed at the proximal end of the device within the handpiece unit. This configuration requires the incident light to travel the length of the endoscope through precisely coupled optical elements. Precise optical elements can easily become misaligned during normal use, and this can result in image distortion or image loss. Embodiments of the present disclosure place the image sensor within the spatially limited environment in the distal end of the endoscope itself. This provides greater optical simplicity when compared to specific implementations known in the art. However, the acceptable solution for this approach is not simple, but rather presents a series of engineering challenges.

[0038] When the overall size of the image sensor is minimized such that the image sensor can fit within the distal tip of an endoscope, there can be a significant loss of image quality. The area of the pixel array of the image sensor can be reduced by reducing the number of pixels and / or the sensing area of each individual pixel. Each of these modifications affects the resolution, sensitivity, and dynamic range of the resulting image. Traditional endoscopic imaging systems are designed to sense stable broadband illumination and provide color information with the aid of a segmented pixel array, such as a Bayer pattern array. In light of the deficiencies associated with segmented pixel arrays, alternative systems and methods are disclosed herein that use a monochrome (which can be referred to as "color agnostic") pixel array that does not include individual pixel filters. In the embodiments disclosed herein, color information is provided by pulsing the emitter with different wavelengths of electromagnetic radiation. The pulsed imaging systems disclosed herein can generate color images with laser mapping data superimposed thereon.

[0039] In one embodiment, color information is determined by capturing independent exposure frames in response to different wavelength electromagnetic radiation pulses. The alternative pulses can include red, green, and blue wavelengths for generating an RGB image frame composed of a red exposure frame, a green exposure frame, and a blue exposure frame. In alternative implementations, the alternative pulses can include luminance ("Y"), red chrominance ("Cr"), and blue chrominance ("Cb") pulses of light for generating a YCbCr image frame composed of luminance data, red chrominance data, and blue chrominance data. The color image frame can also include data from a laser mapping exposure frame superimposed on the RGB or YCbCr image frame. The laser mapping pulses can include one or more pulses for measuring distances or dimensions within a scene, tracking the presence and location of tools in a scene, generating a three-dimensional topography map of a scene, etc. Alternating the wavelengths of the pulsed electromagnetic radiation allows for the utilization of a full pixel array and avoids artifacts introduced by a Bayer pattern pixel array.

[0040] In some cases, it is desirable to generate endoscopic imaging with multiple data types or multiple images that overlap one another. For example, it can be desirable to generate a color (red green blue “RGB”) image that also includes laser mapping imaging data superimposed on the RGB image. This type of superimposed image can enable a medical practitioner or computer program to identify dimensions and topology of a scene based on the laser mapping data. The laser mapping data can be superimposed on the color image in words, numbers, topography maps, etc. Historically, this would require the use of multiple sensor systems including an image sensor for color imaging and one or more additional image sensors for laser mapping imaging. In such systems, the multiple image sensors would have multiple types of pixel sensors that are each sensitive to a different range of electromagnetic radiation. In systems known in the art, this includes three separate types of pixel sensors for generating RGB color images, and additional pixel sensors for generating laser mapping image data at different wavelengths of the electromagnetic spectrum. These multiple different pixel sensors occupy excessive physical space and cannot be located at the distal tip of an endoscope. In systems known in the art, the camera(s) are not placed at the distal tip of the endoscope, but rather in the endoscope handpiece or robotic unit. This creates a number of drawbacks and results in an endoscope that is very delicate. When the endoscope is bumped or impacted during use, the delicate endoscope can be damaged and image quality can be reduced. With the foregoing in mind, disclosed herein are systems, methods, and devices for endoscopic imaging in low light environments. The systems, methods, and devices disclosed herein provide a device for employing multiple imaging techniques in a single imaging process while allowing one or more image sensors to be disposed in the distal tip of an endoscope.

[0041] Laser mapping imaging

[0042] In one embodiment, the systems, methods, and devices disclosed herein provide a device for generating laser mapping data with an endoscopic imaging system. Laser mapping data can be used to determine precise measurements and topographical profiles of a scene. In one particular implementation, laser mapping data is used to determine precise measurements between, for example, structures or organs in a body cavity, devices or tools in a body cavity, and / or critical structures in a body cavity. As discussed herein, the term “laser mapping” can encompass technologies referred to as laser mapping, laser scanning, topography scanning, three-dimensional scanning, laser tracking, tool tracking, etc. Laser mapping exposure frames as discussed herein can include topographical data of a scene, dimensions between objects or structures within a scene, dimensions or distances of tools or objects within a scene, etc.

[0043] Laser mapping generally includes controlled deflection of a laser beam. In the field of three-dimensional object scanning, laser mapping combines controlled steering of a laser beam with a laser rangefinder. By taking distance measurements in each direction, a laser rangefinder can quickly capture the surface shape of objects, tools, and panoramas. Construction of a full three-dimensional topology can include combining multiple surface models obtained from different perspectives. Various measurement systems and methods exist in the art for applications in archaeology, geography, atmospheric physics, autonomous vehicles, etc. One such system includes a light detection and ranging (LIDAR), which is a three-dimensional laser mapping system. LIDAR has been applied in navigation systems such as airplanes or satellites to determine the position and orientation of the sensor in combination with other systems and sensors. LIDAR uses an active sensor to illuminate an object and detect the energy reflected from the object and back to the sensor.

[0044] As discussed herein, the term "laser mapping" includes laser tracking. Laser tracking or tool tracking using a laser measures objects by determining the position of optical targets held relative to those objects. Laser trackers can be accurate to the order of 0.025 mm over distances of several meters. In one embodiment, an endoscope imaging system pulses light for use in conjunction with a laser tracking system so that positions within a scene or tools can be tracked and measured. In such an embodiment, the endoscope imaging system can pulse a laser tracking mode on tools, objects, or other structures within a scene imaged by the endoscope imaging system. A target can be placed on the tools, objects, or other structures within the scene. Measurements between the endoscope imaging system and the target can be triggered and acquired at selected points so that the endoscope imaging system can track the position of the target (and the tools, objects, or other structures to which the target is attached).

[0045] Pulse imaging

[0046] Some implementations of the present disclosure include various aspects of sensor and system combination designs that are capable of generating high definition images with reduced numbers of pixels in limited lighting environments. This is accomplished by pulsing a single color wavelength frame by frame and switching or alternating between each frame with a single different color wavelength using a controlled light source in combination with a high frame capture rate and specially designed corresponding monochromatic sensors. Additionally, electromagnetic radiation can be pulsed to enable the generation of laser mapping data, including size, distance, and three-dimensional topography data about a scene. Pixels can be color agnostic such that each pixel generates data for each electromagnetic radiation pulse, including pulses of red, green, and blue visible light wavelengths and other wavelengths for laser mapping data.

[0047] The system of the present disclosure is an endoscopic imaging system for use in light deficient environments. The system includes an endoscope including an image sensor, where the image sensor is configured to sense reflected electromagnetic radiation for generating a plurality of exposure frames that can be combined to generate an RGB image frame with laser mapping data superimposed thereon. The system includes an emitter for emitting pulses of electromagnetic radiation. The system includes a controller (alternatively referred to as "control circuitry") in electronic communication with the image sensor and the emitter. The controller controls a duty cycle of the emitter in response to a signal corresponding to the duty cycle of the emitter. The image sensor includes a bidirectional pad that can send and receive information. The bidirectional pad of the image sensor operates in a frame period that is divided into three defined states, including a rolling readout state, a service item state, and a configuration state. The system includes an oscillator disposed in the controller and a frequency detector connected to the controller. The frequency detector controls a clock frequency of the image sensor in response to a signal from the controller corresponding to the oscillator frequency. The system enables clock signal data to be transferred from the bidirectional pad of the image sensor to the controller during the service item phase and the configuration phase. The system enables exposure frames to be synchronized without using an input clock or a data transfer clock.

[0048] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe these embodiments. It will nevertheless be understood that no limitation of the scope of the disclosure is intended. Any alterations and further modifications of the described features of this application, and any additional or additional applications of the principles of the present disclosure as described herein are contemplated as falling within the scope of the disclosure as defined by the appended claims.

[0049] Before the structures, systems and methods for producing images in light deficient environments are disclosed and described, it is to be understood that the disclosure is not limited to the particular structures, configurations, process steps, and materials disclosed herein as such structures, configurations, process steps and materials can vary somewhat. It is also to be understood that the terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting since the scope of the present application will be limited only by the appended claims and equivalents thereof.

[0050] In describing and claiming the subject matter of the present disclosure, the following terminology will be used in accordance with the definitions set out below.

[0051] It should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0052] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," "containing," "characterized by," and the like, are open-ended terms that are not to be construed as limiting in any way.

[0053] As used herein, the phrase "consisting of" and grammatical equivalents thereof exclude any element or step not specified in the claim.

[0054] As used herein, the phrase "consisting essentially of" and grammatical equivalents thereof limits the scope of the claim to the specified materials or steps and those that do not materially affect the basic and novel characteristic(s) of the claimed disclosure.

[0055] As used herein, the term "proximal" broadly refers to the concept of a portion that is close to a starting point.

[0056] As used herein, the term "distal" generally refers to the concept of a portion that is opposite to proximal, and thus depending on the context, it refers to a portion that is further from the starting point or the furthest portion.

[0057] As used herein, color sensors or multispectral sensors are those known to have an array of color filters (CFA) on them in order to filter the incident electromagnetic radiation into its individual components. In the visible range of the electromagnetic spectrum, such a CFA can be based on a Bayer pattern or a modified version thereof in order to separate the green, red, and blue spectral components in the light.

[0058] As used herein, monochrome sensors refer to imaging sensors without a filtering function. Because the pixels are color agnostic, their effective spatial resolution is significantly higher than the pixel color (typically filtered in a Bayer pattern) counterparts in traditional single-sensor cameras. Monochrome sensors can also have higher quantum efficiency because there are fewer wasted incident photons between individual pixels.

[0059] As used herein, an emitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of an emitter can be configured to emit pulses and have very specific frequencies or ranges of frequencies from within the entire electromagnetic spectrum. The pulses can include wavelengths in the visible and non-visible ranges. The emitter can cycle on and off to produce pulses, or can produce pulses with a shutter mechanism. The emitter can have a variable power output level, or can be controlled with auxiliary devices such as an aperture or filter. The emitter can emit a broad spectrum or full spectrum of electromagnetic radiation that can be pulsed by color filtering or shutter action. The emitter can include multiple electromagnetic sources acting individually or in concert.

[0060] It should be noted that as used herein, the term "light" is both a particle and a wavelength, and is intended to mean electromagnetic radiation that can be detected by the pixel array 122, and can include wavelengths from the visible and non-visible spectrum of electromagnetic radiation. The term "bin" as used herein refers to a predetermined range of wavelengths of the electromagnetic spectrum that is less than the entire spectrum, or in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a source of light that is controllable with respect to the portion of the electromagnetic spectrum that is emitted, or the physical properties, emission intensity, or emission duration of the components that can be operated, or all of the above. The emitter can emit light in any of a dithered, diffuse, or collimated emission, and can be controlled digitally or by analog methods or systems. As used herein, an electromagnetic emitter is a source of bursts of electromagnetic energy, and includes light sources such as lasers, LEDs, incandescent light, or any light source that can be digitally controlled.

[0061] Referring now to the drawings, FIG. 1 A schematic diagram of a system 100 for sequential pulsed imaging in a light deficient environment is shown. The system 100 can be deployed to generate an RGB image with laser mapping data overlaid on the RGB image. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a bin of electromagnetic radiation in a light deficient environment 112, and the pixel array 122 senses instances of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 operate in sequence such that one or more pulses of the bin of electromagnetic radiation produces image data that is sensed by the pixel array 122.

[0062] It should be noted that as used herein, the term "light" is both a particle and a wavelength, and is intended to mean electromagnetic radiation that can be detected by the pixel array 122, and can include wavelengths from the visible and non-visible spectrum of electromagnetic radiation. The term "bin" as used herein refers to a predetermined range of wavelengths of the electromagnetic spectrum that is less than the entire spectrum, or in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a source of light that is controllable with respect to the portion of the electromagnetic spectrum that is emitted, or the physical properties, emission intensity, or emission duration of the components that can be operated, or all of the above. The emitter can emit light in any of a dithered, diffuse, or collimated emission, and can be controlled digitally or by analog methods or systems. As used herein, an electromagnetic emitter is a source of bursts of electromagnetic energy, and includes light sources such as lasers, LEDs, incandescent light, or any light source that can be digitally controlled.

[0063] The pixel array 122 of the image sensor can be electronically paired with the emitter 102 such that the emitter 102 and the pixel array 122 are synchronized for receiving the emission and adjustments made within the system during operation. The emitter 102 can be tuned to emit electromagnetic radiation in the form of laser light that can be pulsed to illuminate the light deficient environment 112. The emitter 102 can pulse at intervals corresponding to the operation and function of the pixel array 122. The emitter 102 can pulse light in a plurality of electromagnetic partitions such that the pixel array receives electromagnetic energy and produces a data set corresponding in time to each particular electromagnetic partition. For example, FIG. 1 A particular implementation is shown in which the emitter 102 emits four different partitioned electromagnetic radiation, including a red 104 wavelength, a green 106 wavelength, a blue 108 wavelength, and a laser mapping 110 pulse scheme. The laser mapping 110 pulse scheme can include a grid pattern for identifying the topology 120 of a scene in the light deficient environment 112 and further for measuring dimensions and distances within the scene. The laser mapping 110 pulse scheme is any suitable pulse scheme or pattern that can be used to generate laser mapping image data. Laser mapping image data includes data generated by techniques known as laser mapping, laser scanning, topography scanning, three-dimensional scanning, laser tracking, tool tracking, and the like.

[0064] In alternative embodiments not shown in FIG. 1 In alternative embodiments not shown in

[0065] The light deficient environment 112 includes structures, tissues, and other elements that reflect a combination of red 114, green 116, and / or blue 118 light. Structures perceived as red 114 will reflect back the pulsed red 104 light. Reflection off the red structures results in red 105 sensed by the pixel array 122 after the pulsed red 104 emission. The data sensed by the pixel array 122 produces a red exposure frame. Structures perceived as green 116 will reflect back the pulsed green 106 light. Reflection off the green structures results in green 107 sensed by the pixel array 122 after the pulsed green 106 emission. The data sensed by the pixel array 122 produces a green exposure frame. Structures perceived as blue 118 will reflect back the pulsed blue 108 light. Reflection off the blue structures results in blue 109 sensed by the pixel array 122 after the pulsed blue 108 emission. The data sensed by the pixel array 122 produces a blue exposure frame.

[0066] When a structure is a combination of colors, the structure will reflect back a combination of the pulsed red 104 emission, the pulsed green 106 emission, and / or the pulsed blue 108 emission. For example, a structure perceived as purple will reflect back light from the pulsed red 104 emission and the pulsed blue 108 emission. The resulting data sensed by the pixel array 122 will indicate that light was reflected in the same area after the pulsed red 104 emission and the pulsed blue 108 emission. When the resulting red exposure frame and blue exposure frame are combined to form an RGB image frame, the RGB image frame will indicate that the structure is purple.

[0067] In embodiments where the light deficient environment 112 includes fluorescent reagents or dyes or includes one or more fluorescent structures, tissues, or other elements, the pulsing scheme can include emission of certain fluorescent excitation wavelengths. Certain fluorescent excitation wavelengths can be selected to cause known fluorescent reagents, dyes, or other structures to fluoresce. The fluorescent structures will be sensitive to the fluorescent excitation wavelengths and will emit fluorescent relaxation wavelengths. After emission of the fluorescent excitation wavelengths, the fluorescent relaxation wavelengths will be sensed by the pixel array 122. The data sensed by the pixel array 122 produces a fluorescent exposure frame. The fluorescent exposure frame can be combined with a plurality of other exposure frames to form an image frame. The data in the fluorescent exposure frame can be superimposed on an RGB image frame that includes data from red exposure frames, green exposure frames, and blue exposure frames.

[0068] In embodiments in which the light deficient environment 112 includes structures, tissues, or other materials that emit a spectral response to certain partitions of the electromagnetic spectrum, the pulsing scheme can also include emission of the hyperspectral partitions of electromagnetic radiation for eliciting a spectral response from the structures, tissues, or other materials present in the light deficient environment 112. The spectral response includes emission or reflection of electromagnetic radiation of certain wavelengths. The spectral response can be sensed by the pixel array 122 and produce a hyperspectral exposure frame. The hyperspectral exposure frame can be combined with a plurality of other exposure frames to form an image frame. Data in the hyperspectral exposure frame can be overlaid on an RGB image frame that includes data from a red exposure frame, a green exposure frame, and a blue exposure frame.

[0069] In one embodiment, the pulsing scheme includes emission of a laser mapping 110 mode. The reflected electromagnetic radiation sensed by the pixel array 122 after emission of the laser mapping 110 mode produces a laser mapping exposure frame that includes sensed laser mapping 111 data. Data in the laser mapping exposure frame can be provided to a corresponding system to identify, for example, distances between tools present in the light deficient environment 112, a three-dimensional surface topology of a scene in the light deficient environment 112, distances, sizes, or locations of structures or objects within the scene, distances, sizes, or locations of tools within the scene, and the like. The data can be overlaid on an RGB image frame or otherwise provided to a user of the system.

[0070] The emitter 102 can be a laser emitter capable of emitting pulsed red 104 light for generating sensed red 105 data for identifying red 114 elements within the light deficient environment 112. The emitter 102 can also be capable of emitting pulsed green 106 light for generating sensed green 107 data for identifying green 116 elements within the light deficient environment. The emitter 102 can also be capable of emitting pulsed blue 108 light for generating sensed blue 109 data for identifying blue 118 elements within the light deficient environment. The emitter 102 can also be capable of emitting a laser mapping 110 pulsing scheme for mapping a topology 120 of a scene within the light deficient environment 112. The emitter 102 can emit the pulsed red 104, pulsed green 106, pulsed blue 108, and pulsed laser mapping 110 pulsing scheme in any desired order.

[0071] The pixel array 122 senses reflected electromagnetic radiation. Each of the sensed red 105, sensed green 107, sensed blue 109, and sensed laser mapping 111 data can be referred to as an “exposure frame.” Each exposure frame is assigned a particular color or wavelength partition, where the assignment is based on the timing of the pulsed color or wavelength partition from the emitter 102. The combination of an exposure frame and the assigned particular color or wavelength partition can be referred to as a data set. Even though the pixels 122 are not dedicated colors, a color can be assigned to any given data set based on prior information about the emitter.

[0072] For example, during operation, after a pulse of red 104 light is pulsed in the light deficient environment 112, the pixel array 122 senses reflected electromagnetic radiation. The reflected electromagnetic radiation produces an exposure frame, and the exposure frame is categorized as sensed red 105 data because it corresponds in time to the pulsed red 104 light. The exposure frame, along with an indication that it corresponds in time to the pulsed red 104 light, is a“data set.” This process is repeated for each allocation of electromagnetic radiation emitted by the emitter 102. The data created by the pixel array 122 includes a sensed red 105 exposure frame that identifies the red 114 component in the light deficient environment and corresponds in time to the pulsed red 104 light. The data also includes a sensed green 107 exposure frame that identifies the green 116 component in the light deficient environment and corresponds in time to the pulsed green 106 light. The data also includes a sensed blue 109 exposure frame that identifies the blue 118 component in the light deficient environment and corresponds in time to the pulsed blue 108 light. The data also includes a sensed laser mapping 111 exposure frame that identifies the topology 120 and corresponds in time to the laser mapping 110 pulse scheme.

[0073] In one embodiment, the three data sets representing the red, green, and blue electromagnetic pulses are combined to form a single image frame. Thus, the information in the red, green, and blue exposure frames is combined to form a single RGB image frame. One or more additional data sets representing other wavelength partitions can be overlaid on the single RGB image frame. The one or more additional data sets can represent, for example, laser mapping data, fluorescence imaging data, and / or hyperspectral imaging data.

[0074] It should be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition, and that any color combination or any electromagnetic partition can be used in place of RED, GREEN, and BLUE, such as cyan, magenta, and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible and non-visible wavelengths. In the figure, the light deficient environment 112 to be imaged includes a red 114 portion, a green 116 portion, and a blue 118 portion, and also includes a topology 120 that can be sensed and mapped into a three-dimensional rendering. As shown, the reflected light from the electromagnetic pulse contains data only for the portions of the object that have the particular color corresponding to the color partition of the pulse. These individual color (or color interval) data sets can then be used to reconstruct an image by combining the data sets at 126. The information in each of the multiple exposure frames (i.e., multiple data sets) can be combined by a controller 124, control unit, camera control unit, image sensor, image signal processing pipeline, or some other computing resource that can be configured to process the multiple exposure frames and combine the data sets at 126. The data sets can be combined to generate a single image frame either within the endoscope unit itself or off-site by some other processing resource.

[0075] FIG. 2 A system 200 for providing illumination to a light deficient environment, such as for endoscopic imaging. The system 200 can be used in conjunction with any of the systems, methods, or devices disclosed herein. The system 200 includes an emitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an endocavity waveguide 210, an endocavity 212, and an image sensor 214 with accompanying optical components such as lenses. The emitter 202, which can be collectively referred to as a “light source,” generates light that travels through the jumper waveguide 206 and the endocavity waveguide 210 to illuminate a scene at a distal end of the endocavity 212. The emitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, hyperspectral, fluorescence excitation, or other wavelengths. The endocavity 212 can be inserted into a patient’s body for imaging, such as during a procedure or examination. The light is output as shown by the dashed line 216. The scene illuminated by the light can be captured using the image sensor 214 and displayed to a physician or some other medical personnel. The controller 204 can provide control signals to the emitter 202 to control when illumination is provided to a scene. In one embodiment, the emitter 202 and the controller 204 are located within a camera control unit (CCU) or external control console to which the endoscope is connected. If the image sensor 214 includes a CMOS sensor, the light can be periodically provided to the scene during a series of illumination pulse periods between readout periods of the image sensor 214 during so-called blanking periods. Thus, the light can be pulsed in a controlled manner to avoid superimposing into the readout periods of image pixels in the pixel array of the image sensor 214.

[0076] In one embodiment, the lumen waveguide 210 includes one or more optical fibers. These optical fibers can be made of low cost materials, such as plastic, to allow for disposal of the lumen waveguide 210 and / or other portions of the endoscope. In one embodiment, the lumen waveguide 210 is a single glass fiber with a diameter of 500 microns. The jumper waveguide 206 can be permanently attached to the emitter 202. For example, the jumper waveguide 206 can receive light from an emitter within the emitter 202 and provide the light to the lumen waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 106 includes one or more glass optical fibers. The jumper waveguide can include any other type of waveguide for directing light to the lumen waveguide 210. The connector 208 can selectively couple the jumper waveguide 206 to the lumen waveguide 210 and allow light within the jumper waveguide 206 to pass through the lumen waveguide 210. In one embodiment, the lumen waveguide 210 is directly coupled to the light source without any intervening jumper waveguide 206.

[0077] The image sensor 214 includes an array of pixels. In one embodiment, the image sensor 214 includes two or more arrays of pixels for generating three-dimensional images. The image sensor 214 can constitute two additional image sensors each having an independent array of pixels and can be operated independently of one another. The array of pixels of the image sensor 214 includes active pixels and optical black (“OB”) or optically blind pixels. The active pixels can be transparent “color agnostic” pixels capable of sensing imaging data for electromagnetic radiation of any wavelength. The optical black pixels are read during a blanking period of the array of pixels when the array of pixels is “reset” or calibrated. In one embodiment, light is pulsed during the blanking period of the array of pixels when the optical black pixels are being read. After the optical black pixels have been read, the active pixels are read during a readout period of the array of pixels. The active pixels can be charged by electromagnetic radiation pulsed during the blanking period so that the active pixels are ready to be read by the image sensor during the readout period of the array of pixels.

[0078] FIG. 2A complementary system hardware such as a special or general purpose computer. Embodiments within the scope of the present disclosure also can include physical and other non-transitory computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: computer storage media (devices) and transmission media.

[0079] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSDs") (e.g., based on RAM), Flash memory, phase- change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer- executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

[0080] "Network" refers to one or more data links that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. In one specific implementation, the sensors and camera control units can be networked to communicate with each other, as well as with other components connected through the network to which they are connected. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and / or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0081] Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to computer storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received by way of network or data links can be buffered in RAM within a network interface module (e.g., a "NIC"), and then eventually transferred to computer system RAM and / or to less volatile computer storage media (devices) at a computer system. RAM can also include solid state drives (SSDs or PCIx-based real-time memory tiering storage devices such as FusionIO). Thus, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0082] Computer-executable instructions include, for example, instructions and data which, when executed at a general purpose computer, a special purpose computer, or a special purpose processing device cause the processing device to carry out a certain function or group of functions. The computer executable instructions can be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0083] Those skilled in the art will appreciate that the disclosure can be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, hand-held devices, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, storage devices, and the like. It should be noted that any of the

[0084] Additionally, the functions described herein can be performed, where appropriate, by one or more hardware, software, firmware, digital component, or analog component. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays can be programmed to perform one or more of the systems and procedures described herein. Certain terminology used throughout the following description and claims is used for the purpose of reference only. Those skilled in the art will appreciate that the components can have different names. It is not intended to distinguish between components that do not functionally differ from one another.

[0085] FIG. 2A A block diagram of an example computing device 250 is shown. The computing device 250 can be used to perform various procedures, such as those discussed herein. The computing device 250 can be used as a server, a client, or any other computing entity. The computing device 250 can perform various monitoring functions as discussed herein and can execute one or more application programs, such as those discussed herein. The computing device 250 can be any of various computing devices, such as a desktop computer, a laptop computer, a server computer, a handheld computer, a camera control unit, a tablet, etc.

[0086] Computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all of which are coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory device 254 and / or mass storage device 258. The processor 252 can also include various types of computer-readable media, such as cache memory.

[0087] The memory device 254 includes various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 264) and / or nonvolatile memory (e.g., read-only memory (ROM) 266). The memory device 254 can also include rewritable ROM, such as flash memory.

[0088] The mass storage device 258 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., Flash memory), and so forth. As FIG. 2 indicated in FIG. 2, a particular mass storage device is a hard disk drive 274. Various drives can also be included in the mass storage device 258 to enable reading from and / or writing to various computer- readable media. The mass storage device 258 includes removable media 276 and / or non-removable media.

[0089] The I / O device 260 includes various devices that enable a user or user's of the computing device 250 to input to and / or retrieve data and / or other information from the computing device 250. Exemplary I / O devices 260 include a digital imaging device, an electromagnetic sensor and emitter, a cursor control device, a keyboard, a keypad, a microphone, a monitor or other display device, a speaker, a printer, a network interface card, a modem, a lens, a CCD or other image capture device, and so forth.

[0090] The display device 280 includes any type of device capable of displaying information to one or more users of the computing device 250. Examples of the display device 280 include a monitor, a display terminal, a video projection device, and so forth.

[0091] The interface 256 includes various interfaces that enable the computing device 250 to interact with various other systems, devices, or computing environments. Exemplary interfaces 256 can include any number of different network interfaces 270, such as interfaces for connecting to local area networks (LAN), wide area networks (WAN), wireless networks, and the Internet. Other interfaces include a user interface 268 and a peripheral device interface 272. The interface 256 can also include one or more user interface elements 268. The interface 256 can also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mice, trackpads, etc.), keyboards, etc.

[0092] The bus 262 enables the processor 252, the memory device 254, the interface 256, the mass storage device 258, and the I / O device 260 to communicate with one another, as well as with other devices or components coupled to the bus 262. The bus 262 represents what in practice can be one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, etc.

[0093] For exemplary purposes, the programs and other executable program devices shown herein are discrete blocks, but it is understood that such programs and components can reside in various types of memory devices and can be executed by the processor(s) 252 at various times and in various sequences, unless otherwise specified. Alternatively, the systems and programs described herein can be implemented in hardware, or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to perform one or more of the systems and processes described herein.

[0094] FIG. 3A An operational cycle of a sensor used during a rolling readout mode or during sensor readout 300 is shown. A frame readout can begin at and can be represented by the vertical line 310. The readout period is represented by the diagonal or slanted line 302. The active pixels of the pixel array of the image sensor can be read out row by row, with the top of the downward sloping edge being the top sensor row 312 and the bottom of the downward sloping edge being the bottom sensor row 314. The time between the last row readout and the next readout cycle can be referred to as the blanking period 316. It is noted that some of the sensor pixel rows can be covered with a light shield (e.g., a metallic coating or any other substantially black layer of another material type). These covered pixel rows can be referred to as optical black rows 318 and 320. The optical black rows 318 and 320 can be used as inputs to a correction algorithm. As shown, these optical black rows 318 and 320 can be located on the top of the pixel array or on the bottom of the pixel array or on both the top and the bottom of the pixel array. FIG. 3A

[0095] FIG. 3B ​A method of controlling the amount of electromagnetic radiation (e.g., light) that is exposed to a pixel and thus integrated or accumulated by the pixel is shown. It should be appreciated that a photon is a basic particle of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted to an electrical charge or current. An electronic shutter or a rolling shutter (shown by dotted line 322) can be used to start the integration time by resetting the pixel. Light will then be integrated until the next readout phase. The position of the electronic shutter 322 can be moved between two readout cycles 302 in order to control the pixel saturation of a given amount of light. It should be noted that this technique allows the integration time to be constant between two different rows but introduces a delay when moving from the top row to the bottom row.

[0096] FIG. 3C A situation where the electronic shutter 322 has been removed is shown. In this configuration, the integration of the incident light can start during the readout 302 and can end at the next readout cycle 302, which also defines the start of the next integration.

[0097] FIG. 3D A configuration without the electronic shutter 322 but with a controlled and pulsed light 330 during the blanking period 316 is shown. This ensures that all rows see the same light emanating from the same light pulse 330. In other words, each row will start its integration in a dark environment, which can be located at the optical black back row 320 of the readout frame (m) for maximum light pulse width, and will then receive the light pass and will end its integration in a dark environment, which can be located at the optical black front row 318 of the next subsequent readout frame (m+1) for maximum light pulse width. In, for example, FIG. 3D The image generated by the light pulse will be available only during the frame (m+1) readout without interfering with frame (m) and frame (m+2). It should be noted that the condition to have the light pulse read out in only one frame and not interfere with adjacent frames is to fire a given light pulse during the blanking period 316. Because the optical black rows 318, 320 are not sensitive to light, the optical black back row 320 time of frame (m) and the optical black front row 318 time of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 330.

[0098] As shown in FIG. 3A The sensor can be cycled multiple times to receive data for each pulsed color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum) as shown in, for example, FIG. 3. Each cycle can be timed. In one embodiment, the cycles can be timed to operate within a 16.67 ms interval. In another embodiment, the cycles can be timed to operate within an 8.3 ms interval. It should be appreciated that other timing intervals are contemplated by the present disclosure and are intended to fall within the scope of the present disclosure.

[0099] FIG. 4AThe operation of embodiments of the electromagnetic emitter is shown graphically. The emitter can be timed to correspond to the cycle of the sensor, such that electromagnetic radiation is emitted during the sensor operating cycle and / or a portion of the sensor operating cycle. FIG. 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the emitter can be pulsed during the readout period 302 of the sensor operating cycle. In one embodiment, the emitter can be pulsed during the blanking portion 316 of the sensor operating cycle. In one embodiment, the emitter can be pulsed for a duration that is during a portion of two or more sensor operating cycles. In one embodiment, the emitter can start the pulse during the blanking portion 316 or during the optical black portion 320 of the readout period 302 and end the pulse during the readout period 302 or the optical black portion 318 of the readout period 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of the present disclosure, so long as the pulse of the emitter and the cycle of the sensor correspond.

[0100] FIG. 4B The duration and magnitude of the emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, pulse 3 at 416) are represented graphically to control exposure. An emitter with a fixed output magnitude can be pulsed for a longer time interval, providing more electromagnetic energy to the pixel, or the emitter can be pulsed for a shorter time interval, providing less electromagnetic energy. Whether a longer or shorter time interval is needed depends on the operating conditions. FIG. 3D and FIG. 4A The emitter can be pulsed for a certain interval during any of the cycles described above to provide the pixel array with the required electromagnetic energy. An emitter with a fixed output magnitude can be pulsed for a longer time interval, providing more electromagnetic energy to the pixel, or the emitter can be pulsed for a shorter time interval, providing less electromagnetic energy. Whether a longer or shorter time interval is needed depends on the operating conditions.

[0101] The magnitude of the emission itself can be increased to provide more electromagnetic energy to the pixel compared to adjusting the time interval of the emitter pulse fixed output magnitude. Similarly, decreasing the magnitude of the pulse can provide less electromagnetic energy to the pixel. It should be noted that embodiments of the system can have the ability to adjust both the magnitude and the duration, if desired. Additionally, the sensor can be adjusted to increase its sensitivity and duration as needed for optimal image quality. FIG. 4BThe magnitude and duration of the pulses are shown to change. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. In addition, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416, such that the electromagnetic energy provided by the pulse is shown by the area under the pulse shown in the illustration. In the illustration, pulse 2 at 414 has a relatively low magnitude or intensity and a longer duration when compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration when compared to pulse 1 at 412 and pulse 2 at 414.

[0102] FIG. 5 for operation cycles, electromagnetic emitters, and emitted electromagnetic pulses to display embodiments of the present disclosure of an imaging system during operation in accordance with the principles and teachings of the present disclosure FIGS. 3A-3D and FIG. 4A An illustration of an embodiment of the present disclosure of an operation cycle, electromagnetic emitter, and emitted electromagnetic pulses to display an imaging system during operation. As can be seen in the figure, the electromagnetic emitter pulses radiation primarily during the blanking period 316 of the image sensor, such that the pixels will be charged and ready to be read during the readout period 302 of the image sensor cycle. FIG. 5 The dashed lines in the figure represent pulses of electromagnetic radiation (from FIG. 4A ) The electromagnetic radiation pulses are emitted primarily during the blanking period 316 of the image sensor, but can be overlaid with the readout period 302 of the image sensor.

[0103] The exposure frame includes data read by the pixel array of the image sensor during the readout period 302. The exposure frame can be combined with an indication of the type of pulse emitted by the emitter prior to the readout period 302. The combination of the exposure frame and the indication of the type of pulse can be referred to as a data set. Multiple exposure frames can be combined to generate a black and white or RGB color image. In addition, hyperspectral, fluorescence, and / or laser mapping imaging data can be overlaid on the black and white or RGB image.

[0104] In one embodiment, the exposure frame is data sensed by the pixel array during the readout period 302 that occurs after the blanking period 316. The emission of electromagnetic radiation is emitted during the blanking period 316. In one embodiment, a portion of the emission of electromagnetic radiation overlaps with the readout period 316. The blanking period 316 occurs when the optical black pixels of the pixel array are being read, and the readout period 302 occurs when the active pixels of the pixel array are being read. The blanking period 316 can overlap with the readout period 302.

[0105] FIG. 6A and FIG. 6BA process for recording image frames is shown. Multiple image frames can be strung together to generate a video stream. A single image frame can include data from multiple exposure frames, where an exposure frame is data sensed by a pixel array after an emission of electromagnetic radiation. FIG. 6A A conventional process is shown that is typically implemented with a color image sensor that has a color filter array (CFA) to filter out certain wavelengths of light for each pixel. FIG. 6B A process disclosed herein, and can be implemented with a monochrome "color agnostic" image sensor that receives electromagnetic radiation of all wavelengths.

[0106] FIG. 6A The process shown occurs during time t(0) to time t(l). The process starts with an emission of white light 602 and senses white light 604. At 606, an image is processed and displayed based on the sensing at 604.

[0107] FIG. 6B The process shown occurs during time t(0) to time t(l). The process starts with an emission of green light 612 and senses reflected electromagnetic radiation 614 after the emission of green light 612. The process continues with an emission of red light 616 and senses reflected electromagnetic radiation 618 after the emission of red light 616. The process continues with an emission of blue light 620 and senses reflected electromagnetic radiation 622 after the emission of blue light 620. The process continues with one or more emissions of a laser mapping 624 pulse scheme and senses reflected electromagnetic energy 626 after each of the one or more emissions of the laser mapping 624 pulse scheme. An image is processed and displayed at 628 based on each of the sensed reflected electromagnetic energy instances 614, 618, 622, and 626.

[0108] FIG. 6B The process shown provides a higher resolution image and provides a means for generating an RGB image that also includes laser mapping data. When using a partitioned spectrum of light (as shown FIG. 6B The sensor can be made sensitive to electromagnetic energy of all wavelengths. In FIG. 6BIn the illustrated process, the monochromatic pixel array is instructed to sense electromagnetic energy from a predetermined partition of the full spectrum of electromagnetic energy in each cycle. Thus, to form an image, the sensor need only cycle through the different partitions of the full spectrum light. The final image is assembled based on the multiple cycles. Because the image from each color partition frame cycle (as compared to a CFA pixel array) has a higher resolution, the resulting image produced when the partitioned light frames are assembled also has a higher resolution. In other words, because every pixel within the array (as opposed to every other pixel in a sensor with a CFA) senses the amplitude of the energy of a given pulse and a given scene, only a period of time apart, a higher resolution image is produced for each scene.

[0109] As shown in the embodiment illustrated in FIG. 1, the sensor for the partitioned spectral system in FIGS. 6A-6B can cycle through FIG. 6B at least four times per full spectrum system in FIG. 6A . In one embodiment, the display device (LCD panel) operates at a rate of 50 to 60 frames per second. In such an embodiment, the partitioned light system in FIG. 6B can operate at a rate of 200 to 240 frames per second to maintain continuity and smoothness of the displayed video. In other embodiments, there can be different capture and display frame rates. Further, the average capture rate can be any multiple of the display rate.

[0110] In one embodiment, it can be desirable that not all partitions are equally represented within the system frame rate. In other words, not all light sources must be pulsed with the same regularity in order to emphasize and de-emphasize aspects of the recorded scene as desired by the user. It should also be understood that the non-visible and visible partitions of the electromagnetic spectrum can be pulsed together within the system, where their respective data values are stitched into the video output for display to the user.

[0111] Embodiments can include the following pulse cycle pattern:

[0112] i. a green pulse;

[0113] ii. a red pulse;

[0114] iii. a blue pulse;

[0115] iv. a green pulse;

[0116] v. a red pulse;

[0117] vi. a blue pulse;

[0118] vii. a laser mapping pulse;

[0119] viii. (repeat)

[0120] Embodiments can include a pulse cycle pattern as follows:

[0121] i. a luminance pulse;

[0122] ii. a red chrominance pulse;

[0123] iii. a luminance pulse;

[0124] iv. a blue chrominance pulse;

[0125] v. a laser mapping pulse;

[0126] vi. (repeat)

[0127] Embodiments can include a pulse cycle pattern as follows:

[0128] i. a luminance pulse;

[0129] ii. a red chrominance pulse;

[0130] iii. a luminance pulse;

[0131] iv. a blue chrominance pulse;

[0132] v. a luminance pulse;

[0133] vi. a red chrominance pulse;

[0134] vii. a luminance pulse;

[0135] viii. a blue chrominance pulse;

[0136] ix. a laser mapping pulse;

[0137] x. (repeat)

[0138] As can be seen in this example, the laser mapping partition can pulse at a different rate than the other partitions. Doing so can emphasize a certain aspect of the scene, where the laser mapping data only overlaps with other data in the video output to make the desired emphasis. It should be noted that adding a laser mapping partition on top of the red, green, and blue partitions does not necessarily require a serialized system to operate at four times the rate of a full-spectrum non-serialized system, as each partition does not have to be equally represented in the pulse pattern. As seen in this embodiment, adding a partition pulse that is represented less in the pulse pattern (laser mapping in the above example) will result in less than a 20% increase in the cycle speed of the sensor to accommodate the irregular partition sampling.

[0139] The partition cycle can be divided to accommodate or approximate various imaging and video standards. In one embodiment, the partition cycle includes the following as outlined inFIGS. 7A-7D The pulses of electromagnetic energy in the red, green, and blue spectra are best shown in FIGS. 7A-7D The timing relationship between the emission of pulses of electromagnetic radiation by the emitters and the readout of the pixel array is further shown in

[0140] In FIG. 7A Different light intensities have been achieved in by modulating the light pulse width or duration within the working range shown by the vertical gray dashed line. FIG. 7A The general timing relationship between the mixing of pulses of three wavelengths within a four frame cycle and the readout cycle of the pixel array of an image sensor is shown. In one embodiment, there are three monochromatic pulsed light sources under the control of a controller. A periodic sequence of monochromatic red, monochromatic green, and monochromatic blue exposure frames are captured, for example, with an R-G-B-G pulse pattern, and combined by an image signal processor chain into an sRGB image frame.

[0141] In FIG. 7B Different light intensities have been achieved in by modulating the light power or power of the electromagnetic emitters, which can be laser or LED emitters, but keeping the pulse width or duration constant.

[0142] FIG. 7C The case where both the light power and the light pulse width are modulated for greater flexibility is shown. Partitioned cycles can use cyan, magenta, yellow (CMY), infrared, ultraviolet, hyperspectral, and fluorescent, use invisible pulse sources mixed with visible pulse sources, and any other color space needed to produce an image or approximate a desired video standard currently known or yet to be developed. It should also be understood that the system is capable of switching between color spaces on the fly to provide the required image output quality.

[0143] In an embodiment using the color space green-blue-green-red (as shown in FIG. 7D It can be desirable to pulse the luminance component more frequently than the chroma components because users are generally more sensitive to light magnitude differences than to light color differences. This principle can be exploited using a monochromatic image sensor as shown in FIG. 7D In FIG. 7D Green, which contains the most luminance information, can be pulsed more frequently or with greater intensity in a (G-B-G-R-G-B-G-R...) scheme to obtain luminance data. Such a configuration will create a video stream with perceptibly more detail without creating and transmitting imperceptible data.

[0144] In one embodiment, all three light sources are pulsed in unison with light energy modulated to provide pure luminance information in the same exposure frame. The light energy can be modulated according to color conversion coefficients that convert from an RGB color space to a YCbCr color space. It should be understood that the color conversion can be implemented according to any suitable standard such as the ITU-R BT.709 HD standard, the ITU-R BT.601 standard, the ITU-R BT.2020 standard, or any other suitable standard or formula. The conversion can be performed according to the ITU-R BT.709 HD standard as follows:

[0145]

[0146] In addition to modulation of the luminance information, full color images also require a red chrominance component and a blue chrominance component. However, the algorithm applied to the luminance component cannot be applied directly to the chrominance components because the algorithm is signed, as reflected in the fact that some of the RGB coefficients are negative. In one embodiment, the degree of luminance is boosted so that all of the final pulse energies are positive. As long as the color fusion process in the image signal processor knows the composition of the chrominance exposure frames, they can be decoded by subtracting an appropriate amount of luminance from the adjacent frames. The pulse energy proportions are given by:

[0147] Y = 0.183 · R + 0.614 · G + 0.062 · B

[0148] Cb = λ · Y - 0.101 · R - 0.339 · G + 0.439 · B

[0149] Cr = δ · Y + 0.439 · R - 0.399 · G - 0.040 · B

[0150] where

[0151]

[0152]

[0153] If the lambda factor is equal to 0.552, then the red and green components are canceled out. In this case, the blue chrominance information can be provided with pure blue light. Similarly, if the delta factor is equal to 0.650, then the blue and green components are canceled out, and the red chrominance information can be provided with pure red light. This embodiment is a convenient approximation of digital frame reconstruction.

[0154] In embodiments where white balancing is performed in the illumination domain, in addition to the white balancing modulation, a modulation is also applied.

[0155] In one embodiment, pulsing of the weaker partitioned regions can be used to produce an output that has been adjusted for the weaker pulse. For example, blue laser light is considered weak relative to the sensitivity of silicon-based pixels, and is difficult to produce compared to red or green light, and can be pulsed more frequently during a frame cycle to compensate for the weakness of the light. These additional pulses can be done continuously over time, or by pulsing multiple lasers simultaneously to produce the desired compensation effect. It should be noted that by pulsing during a blanking period (time during which the sensor is not reading out the pixel array), the sensor is not sensitive to differences / mismatches between lasers of the same kind, and simply accumulates the light for the desired output. In another embodiment, the range of maximum light pulses can be different from frame to frame. This is shown in FIG. 7E where the light pulses are different from frame to frame. The sensor can be constructed to be able to program different blanking periods in a two-frame or three-frame or four-frame or n-frame repeating pattern.

[0156] In FIG. 7E , four different light pulses are shown, and pulse 1 can repeat, for example, after pulse 4, and can have a four-frame pattern with different blanking periods. This technique can be used to place the most powerful partition on the smallest blanking period, and thus allow the weakest partition to have a wider pulse on one of the subsequent frames without increasing the readout speed. The reconstructed frame can still have a regular pattern from frame to frame, as it is composed of many pulse frames.

[0157] FIG. 8 A system 800 for controlling the integrated energy of laser pulses in an environment for hyperspectral, fluorescence, and / or laser mapping imaging is shown. In conventional systems known in the art, lasers or other light sources deliver a specified light level consistently. These conventional systems do not have a means to offset problems that arise during normal use. Such problems can be caused by pre-heat requirements, temperature variations, manufacturing defects, and the like. These problems can reduce image quality due to flickering light and image artifacts. These problems can be ameliorated by the systems disclosed herein, including the system 800 shown. FIG. 8

[0158] FIG. 8 ​The illustrated system 800 represents one example implementation, and many other implementations can be realized without departing from the scope of the present disclosure. The system 800 includes a laser module 802, an optical fiber 804 connected to the laser module 802, and an electromagnetic sensor such as a photodiode 806 connected to the optical fiber 804. The laser module 802 can be referred to generally herein as a “transmitter,” and the term transmitter as used herein can include multiple laser modules. The laser module 802 and the photodiode 806 are each in communication with a controller 808. The controller 808 receives light-sensing readings from the photodiode 806 and changes the wavelength and / or power of light emitted by the laser module 802 in response to real-time light readings received from the photodiode 806.

[0159] In FIG. 8 In the illustrated example implementation, the system 800 includes two separate laser modules 802, two separate optical fibers 804, and two separate photodiodes 806. It should be understood that the system 800 can include any number of laser modules 802, photodiodes 806, and optical fibers 804 without departing from the scope of the present disclosure.

[0160] In one implementation, the system 800 is implemented to control the duration and / or intensity of light pulsed by the transmitter to illuminate a light deficient environment. The laser controller 808 is in communication with each of the plurality of laser modules 802 and is configured to control the duration and / or intensity of light emitted by the laser module 802. The photodiode 806 is configured to measure the duration and / or intensity of light emitted by the laser module 802 by measuring light traveling through the optical fiber 804. The laser controller 808 is used to limit the output light energy of the laser module 802 within acceptable tolerances.

[0161] The laser module 802 can include one or more lasers or other light emitters capable of emitting a wide range of wavelengths of electromagnetic radiation. The wavelengths and power of the emitted electromagnetic radiation can be selected to meet application requirements. In one implementation, the laser module 802 includes a plurality of lasers arranged in a linear array or some other geometric pattern as FIG. 9 The laser module 802 can generally include ten or more individual laser units to ensure precise light output.

[0162] The optical fiber 804 can include a single optical fiber or a bundle of optical fibers. The optical fiber 804 is directly connected to the laser module 802 and can be directly connected to the optical output of the lasers within the laser module 802. In some implementations, the optical fiber 804 can be in the range of 0.05 mm to 0.5 mm.

[0163] The photodiode 806 is an electromagnetic sensor. In alternative embodiments, different types of electromagnetic sensors can be used. The photodiode 806 senses the light level traveling through the optical fiber 804 by reading the energy transmission on the individual optical fiber 804 or bundle of optical fibers 804. In embodiments, the photodiode 806 measures the light level information by sensing the voltage or current of the optical fiber 804, and in such embodiments, the photodiode 806 can be directly attached to the optical fiber 804. The photodiode 806 sends the light level information to the controller 808. In some embodiments, the photodiode 806 converts the raw data into a usable light level measurement, and in other embodiments, the controller 808 converts the raw data into a usable light level measurement.

[0164] In one embodiment, the total amount of light in the scene is calculated based on the measurement of one, two, or N photodiodes 806. The one, two, or N photodiodes 806 can be used with an amplifier to compare the light output measurement to a desired reference voltage or current level. This circuit can provide direct feedback to the bias current or voltage of the laser module 802 to ensure that the desired output light level is met.

[0165] In one embodiment, a photodiode 806 or another electromagnetic sensor is built-in on each laser module 802. A portion of the electromagnetic radiation emitted by the laser module 802 can be directed at the photodiode 806 or other sensing element capable of converting light into a voltage or current level to measure the light output.

[0166] The controller 808 receives the light level information from the photodiode 806 in real-time. The controller 808 calculates the amount of light emitted by the laser module 802 based on the energy transmission readings received from the photodiode 806. These energy transmission readings serve as independent feedback from the photodiode 806 to the controller 808. The controller 808 can compare this independent feedback to a register or variable value that indicates a predetermined desired light level. The register or variable value can be retrieved from computer memory.

[0167] In one embodiment, the controller 808 calculates the light level and integrates the light energy in less than 1 ms. The controller 808 implements an automatic exposure control (AEC) system to adjust the wavelength and / or duration of the light emitted by the laser module 802 in response to the current light readings received from the photodiode 806. The AEC can include a PID (proportional, integral, and derivative) control algorithm, and this PID control algorithm can be implemented by the controller 808.

[0168] In one embodiment, each pulse of electromagnetic radiation is scaled based on an error measurement. The controller 808 calculates the error measurement by comparing the desired exposure level to the measured exposure level. The measured exposure level is calculated based on the average pixel value of all pixels within the pixel array of the image sensor. The ASC can request adjustments to the duration and / or intensity of the light emitted by the laser module 802 to ensure the desired exposure level is achieved.

[0169] The controller 808 ensures that once the light energy level is reached, the laser module 802 is turned off or disabled to maintain the desired image quality.

[0170] In one embodiment, the controller 808 is or includes a camera control unit (CCU). The controller 808 can include a microcontroller, a field programmable gate array (FPGA) board, an application specific integrated circuit (ASIC), hardware, software, an image signal processor (ISP), support circuitry, and the like. The controller 808 controls the activation and deactivation of the laser module 802. The controller 808 also controls the power level and light level of the laser module 802. The controller 808 receives information from the photodiode 806 and changes the light energy emitted by the laser module 802 based on the information received from the photodiode 806.

[0171] FIG. 9 A system 900 for emitting electromagnetic radiation is shown. The system 900 can be implemented in conjunction with a digital imaging system for a light deficient environment as described herein. The system 900 can be deployed to control the output of electromagnetic radiation to illuminate a scene.

[0172] The system 900 includes a laser beam 902 having a plurality of laser units. The laser beam 902 can be referred to generally herein as a “emitter” and the term emitter as used herein can include a plurality of laser beams. The laser beam 902 is connected to an optical fiber 904. The system includes an electromagnetic sensor 906, such as a photodiode or other light sensing element. The electromagnetic sensor 906 senses the output of electromagnetic radiation through at least one laser unit of the laser beam 902. The system 900 includes an operational amplifier (OPA) 908 circuit electrically connected to the electromagnetic sensor 906. The operational amplifier 908 can apply feedback, or in an alternative embodiment, an optical frequency multiplier can apply feedback.

[0173] The electromagnetic sensor 906 senses the output of at least one laser unit of the laser beam 902. Based on this measured value, the total output of the laser beam 902 is controlled and adjusted in real time to ensure proper exposure of the scene. In one embodiment, the light output is controlled to a precise level of 0.01% to 10%.

[0174] FIG. 10An implementation of a system 1000 for laser illumination is shown. The system 1000 includes multiple laser beams for emitting electromagnetic energy at multiple wavelengths. The system 1000 can be implemented for generating color RGB image frames based on separate pulses of red, green, and blue electromagnetic radiation that respectively produce red, green, and blue exposure frames. The system 1000 can further be implemented for generating laser mapping exposure frames to be superimposed on the RGB image frames.

[0175] In FIG. 10 In the example implementation shown, the system 1000 includes a red laser beam 1010, a green laser beam 1012, a blue laser beam 1014, and a laser mapping laser beam 1016. FIG. 10 The laser beams shown are merely exemplary, and the system can include alternative laser beams for different applications. For example, the system 1000 can include a hyperspectral laser beam, a fluorescence laser beam 1016, and / or a laser mapping laser beam in addition to the red laser beam 1010, the green laser beam 1012, and the blue laser beam 1014.

[0176] The laser mapping laser beam 1016 can include multiple different laser beams for emitting electromagnetic radiation to measure distances, dimensions, and three-dimensional topography maps of a scene. In one embodiment, the laser mapping laser beam 1016 includes multiple laser beams attached to, for example, tools within the environment, such as surgical tools, as well as the endoscope device itself, and other tools within the environment. The multiple laser mapping laser beams 1016 can work together to track the position of the tools in the environment relative to other objects, such as tissue, organs, and other critical structures. The laser beams 1010, 1012, 1014, 1016 can be referred to generally herein as “emitters,” and the term “emitter” can include any suitable number of laser beams as desired.

[0177] If the system includes a fluorescence laser beam, the fluorescence laser beam can include multiple different laser beams for emitting electromagnetic radiation at different fluorescence excitation wavelengths for causing different reagents within the scene to fluoresce.

[0178] If the system 1000 includes a hyperspectral laser beam, the system 1000 can include multiple independent hyperspectral laser beams for emitting different partitions of the electromagnetic spectrum. For example, the system 1000 can include a first hyperspectral laser beam for emitting electromagnetic radiation having a wavelength from about 513 nm to about 545 nm. The system 1000 can further include a second hyperspectral laser beam for emitting electromagnetic radiation having a wavelength from about 565 nm to about 585 nm. The system 1000 can further include a second hyperspectral laser beam for emitting electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm.

[0179] The laser beams 1010, 1012, 1014, 1016 can be referred to generally herein as “emitters,” and the term “emitter” can include any suitable number of laser beams as desired. In one embodiment, the fluorescence laser beam 1016 includes two separate fluorescence laser beams 1016, with one fluorescence laser beam configured to emit electromagnetic radiation having a wavelength from about 770 nm to about 790 nm, and the other fluorescence laser beam configured to emit electromagnetic radiation having a wavelength from about 795 nm to about 815 nm.

[0180] It will be appreciated that the system 1000 can include any number of laser beams, and each of the laser beams can be configured to emit electromagnetic radiation at different wavelengths. As described herein, the system 1000 can include “emitters” that include multiple laser beams, such as laser beams for red, green, and blue wavelengths, as well as laser beams for hyperspectral, fluorescence, and / or laser mapping imaging. In one embodiment, the system 1000 includes laser beams for emitting electromagnetic radiation at red, green, and blue wavelengths, and further includes laser beams for emitting electromagnetic radiation at hyperspectral wavelengths, for emitting fluorescence excitation wavelengths for causing a reagent to fluoresce, and for emitting laser mapping patterns.

[0181] In one embodiment, the system 1000 includes a red laser beam 1010, a green laser beam 1012, and a blue laser beam 1014, and further includes a laser mapping laser beam 1016, and one or more of a hyperspectral laser beam or a fluorescence laser beam. The hyperspectral laser beam can emit electromagnetic radiation at multiple wavelengths for eliciting a spectral response. Information collected in response to the emission of electromagnetic radiation at the hyperspectral wavelengths can be analyzed to identify locations of key tissue structures such as diseased tissue, tumors, blood vessels, blood flow direction, certain chemical processes, certain biological processes, etc. This information can be used in conjunction with information collected in response to electromagnetic radiation pulses of the laser mapping laser beam 1016. For example, information sensed by an image sensor in response to the laser mapping laser beam 1016 emitting electromagnetic radiation can identify dimensional, distance, and three-dimensional topographical information about a scene. This information can be combined with information collected in response to electromagnetic radiation pulses of the hyperspectral laser beam to calculate distance, dimensional, and three-dimensional topographical information of key tissue structures such as diseased tissue, tumors, blood vessels, blood flow direction, certain chemical processes, certain biological processes, etc. It will be appreciated that the same process can be implemented with information collected in response to emissions of the laser mapping laser beam 1016 and information collected in response to emissions of a fluorescence laser beam configured to emit fluorescence excitation wavelengths for causing a reagent within a scene to fluoresce.

[0182] Each of the laser beams includes a plurality of laser units for emitting pulses of electromagnetic radiation. The system 1000 includes a dedicated electromagnetic sensor 1006 for measuring the electromagnetic radiation emitted by each laser beam. As shown, the system 1000 includes a separate electromagnetic sensor 1006 for each of the red laser beam 1010, the green laser beam 1012, the blue laser beam 1014, and the laser mapping laser beam 1016. The electromagnetic sensor 1006 can be connected to one or more laser units of the laser beam 1010, 1012, 1014, 1016. The electromagnetic sensor 1006 senses the output of electromagnetic radiation by measuring the current or voltage emitted by at least one laser unit. FIG. 10

[0183] Light is transmitted from the laser beams through optical fibers 1004. Each of the laser beams 1010, 1012, 1014, 1016 includes an optical fiber 1004. The optical fibers 1004 are combined to include a single exit optical fiber 1018.

[0184] FIG. 11 A schematic diagram of a system 1100 for providing illumination to a light deficient environment, such as for endoscopic imaging, is shown. The system 1100 can be used in conjunction with any of the systems, methods, or apparatuses disclosed herein. The system 1100 includes a light source 1102, a controller 1104, a jumper waveguide 1106, a waveguide connector 1108, a lumen waveguide 1110, a lumen 1112, and an image sensor 1114 with attendant optical components such as lenses. The light source 1102, which can be referred to as an “emitter” in some embodiments, generates light that travels through the jumper waveguide 1106 and the lumen waveguide 1110 to illuminate a scene at a distal end of the lumen 1112. The light source 1100 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared wavelengths, ultraviolet wavelengths, or other wavelengths. The lumen 1112 can be inserted into a patient’s body for imaging, such as during a procedure or examination. The light is output as shown by the dashed lines 1116. The scene illuminated by the light can be captured using the image sensor 1114 and displayed to a physician or some other medical personnel. The controller 1104 can provide control signals to the light source 1102 to control when illumination is provided to a scene. In one embodiment, the light source 1102 and the controller 1104 are located within a camera control unit (CCU) or external control console to which an endoscope is connected. If the image sensor 1114 includes a CMOS sensor, the light can be provided to the scene periodically in a series of illumination pulses during a so-called blanking period between readout periods of the image sensor 1114. Thus, the light can be pulsed in a controlled manner to avoid superimposing onto the readout periods of image pixels in the pixel array of the image sensor 1114.

[0185] ​In one embodiment, the lumen waveguide 1110 includes one or more optical fibers. These optical fibers can be made of low cost materials, such as plastic, to allow for disposal of the lumen waveguide 1110 and / or other portions of the endoscope. In one embodiment, a single glass fiber having a diameter of 500 microns can be used. The jumper waveguide 1106 can be permanently attached to the light source 1102. For example, the jumper waveguide 1106 can receive light from an emitter within the light source 1102 and provide the light to the lumen waveguide 1110 at the location of the connector 1108. In one embodiment, the jumper waveguide 106 can include one or more glass optical fibers. The jumper waveguide can include any other type of waveguide for directing light to the lumen waveguide 1110. The connector 1108 can selectively couple the jumper waveguide 1106 to the lumen waveguide 1110 and allow light within the jumper waveguide 1106 to pass through the lumen waveguide 1110. In one embodiment, the lumen waveguide 1110 can be directly coupled to the light source without any intervening jumper waveguide 1106.

[0186] FIG. 11 A digital imaging system 1100 is shown that utilizes a minimal pad interconnect to reduce the size of an image sensor used with an endoscope device within a light deficient environment. FIG. 11 The digital imaging system 1100 shown includes an endoscope device 1102 used in a light deficient environment. The endoscope device 1102 includes an endoscope 1104, an endoscope housing 1106, a controller 1108, an electronic communication device 1120, a light source 1110, an optical cable 1126, a display 1112, and an imaging device 1114. The electronic communication 1120 can include an electronic cable or other form of wired or wireless communication. In some embodiments, the optical cable 1126 can be an optical fiber cable. The imaging device 1114 can be an image sensor, such as a CMOS image sensor having an array of pixels.

[0187] In FIG. 11In the illustrated example, for ease of discussion, the endoscope device 1104, the endoscope housing 1106, the controller 1108, the light source 1110, the display 1112, and the imaging device 1114 are shown separately relative to one another. However, it should be understood and appreciated that this is not to be construed as limiting, and any one or more of these components can be integrated and / or connected in any suitable manner. It should be understood that the image sensor senses the reflected electromagnetic radiation with a pixel array. The pixel array generates an exposure frame comprising image data in response to the pulse of electromagnetic radiation. The processor 1124 can detect image texture and edges within the image frame and can further enhance the texture and edges within the image frame. The processor 1124 can also retrieve characteristics from memory related to the pixel technology and the applied sensor gain, whether in the housing 1106 or at the controller 1108, to assess an expectation of the noise magnitude within the image frame produced by the image sensor and use that noise expectation to control the edge enhancement application. The stream of image frames can be produced by sequentially combining multiple image frames, where each image frame comprises data from multiple exposure frames.

[0188] It should be understood that traditional rod-lens endoscopes for laparoscopy, arthroscopy, urology, gynecology, and ENT (ear, nose, and throat) procedures require high manufacturing costs due to their complex optical makeup. Incident image information is transmitted in the optical domain along the entire length of the endoscope. Typically, these conventional endoscopes are optically coupled to a handpiece unit that includes an image sensor. This type of conventional endoscope is delicate and easily damaged during handling, use, and sterilization. The necessary repair and sterilization processes add additional expense to each procedure in which they are utilized.

[0189] An endoscope 1102 can be improved by placing an image sensing device at the distal end of the endoscope. In such embodiments, the optical transmission assembly can be replaced with a simple plastic lens stack. The price of such endoscopes can be very low, so it can be more economical to manufacture them for single use only, followed by disposal or recycling, as there would no longer be a need for repair and sterilization processes.

[0190] However, when the image sensor is located at the distal end of the endoscope 1102, the image sensor must be very small. The distal end of the endoscope 1102 is spatially constrained in the x-dimension and the y-dimension. One way to reduce the size of the image sensor is to reduce the number of bond pads within the image sensor chip. Each bond pad takes up significant physical space on the image sensor chip. Each bond pad is used to provide power or input / output signals to and from the image sensor chip. Thus, to minimize area, it is desirable to reduce the number of bond pads. The present disclosure describes systems and methods for reducing the bond pad count by combining digital input and output functions into the same bidirectional pad. During image transfer, these bidirectional pads act as differential outputs. In one embodiment, during a defined portion of each exposure frame, the bidirectional pads switch direction to receive commands. In such embodiments, the camera control electronics are synchronized so that commands are issued to the image sensor bidirectional pads at a specific time when the bidirectional pads are configured to receive commands.

[0191] In addition to this, in the context of an endoscope system, simplicity and manufacturability can be enhanced by customizing the image sensor to receive commands and information from the endoscope handpiece. Information can be incorporated into the output data emitted by the image sensor. This reduces the total conductor count from the endoscope to the camera system. Such sources of information can include user initiated button events or a measurement of the angle of rotation of the endoscope relative to the handpiece. Certain embodiments of the endoscope require angle measurement where the image sensor of the endoscope is placed at the distal end.

[0192] CMOS image sensors typically incorporate two different power supplies, requiring three pads: VDD1, VDD2, and GND. The higher of the two voltages is primarily used for the purpose of biasing the pixel array. Occasionally, the higher of the two voltages is also used to power the input and output circuits. The lower of the two voltages is typically used to power the peripheral analog circuits and the digital portion of the image sensor, where applicable.

[0193] In one embodiment, the bond pad count is reduced by using only a single external power supply. This can be accomplished by providing multiple internal power supplies using, for example, an internal DC-to-DC converter or regulator. In addition, the bond pad count can be reduced by providing only a single power level. A second power level can then be derived on-chip. This embodiment can effectively remove power supply circuitry, such as a regulator, from the camera system.

[0194] FIG. 12 is a schematic diagram of a mode reconstruction process. FIG. 12The example pattern shown includes red, green, blue, and laser mapping light pulses each of duration T1. In various embodiments, the light pulses can have the same duration or different durations. The red, green, blue, and laser mapping exposure frames are combined to generate an RGB image with laser mapping data superimposed on top. A single image frame that includes the red, green, blue, and laser mapping exposure frames requires a period of 4*T1 to generate. FIG. 12 The durations shown are merely exemplary and can vary for different implementations. In other embodiments, different pulse schemes can be employed. For example, embodiments can be based on the timing of each color component or frame (T1), and the reconstructed frame has a period that is twice the period of the input color frame (2 x T1). Different frames within the sequence can have different frame periods, and the average capture rate can be any multiple of the final frame rate.

[0195] In one embodiment, the dynamic range of the system is increased by varying the pixel sensitivity of the pixels within the pixel array of the image sensor. Some pixels can sense reflected electromagnetic radiation at a first level of sensitivity, other pixels can sense reflected electromagnetic radiation at a second level of sensitivity, and so on. The different pixel sensitivities can be combined to increase the dynamic range provided by the pixel configuration of the image sensor. In one embodiment, adjacent pixels are set at different sensitivities such that each cycle includes data generated by pixels that are more sensitive and less sensitive relative to each other. When multiple sensitivities are recorded in a single cycle of the pixel array, the dynamic range is increased. In one embodiment, a wide dynamic range can be achieved by having multiple global TXs, each firing on only a different set of pixels. For example, in a global mode, global TX1 signal is firing on pixel set 1, global TX2 signal is firing on pixel set 2, global TXn signal is firing on pixel set n, and so on.

[0196] FIGS. 13A-13CEach shows a light source 1300 with multiple emitters. The emitters include a first emitter 1302, a second emitter 1304, and a third emitter 1306. Additional emitters can be included, as discussed further below. The emitters 1302, 1304, and 1306 can include one or more laser generators that emit light having different wavelengths. For example, the first emitter 1302 can emit a wavelength consistent with a blue laser, the third emitter 1304 can emit a wavelength consistent with a green laser, and the third emitter 1306 can emit a wavelength consistent with a red laser. For example, the first emitter 1302 can include one or more blue lasers, the second emitter 1304 can include one or more green lasers, and the third emitter 1306 can include one or more red lasers. The lasers 1302, 1304, 1306 emit laser beams toward a collection region 1308, which can be a waveguide, a mirror, or other optical component for collecting light and / or providing light to a jumper waveguide 206 or a bore waveguide 210 of FIG. 2

[0197] In one implementation, the emitters 1302, 1304, and 1306 emit electromagnetic radiation at hyperspectral wavelengths. Certain hyperspectral wavelengths can penetrate tissue and enable a practitioner to "see through" tissue in front to identify chemical processes, structures, compounds, biological processes, etc. located behind the tissue in front. The hyperspectral wavelengths can be specifically selected to identify certain diseases, tissue conditions, biological processes, chemical processes, types of tissue, etc. that are known to have a particular spectral response.

[0198] In implementations in which an agent or dye has been administered to a patient that assists in identifying certain tissues, structures, chemical reactions, biological processes, etc., the emitters 1302, 1304, and 1306 can emit wavelengths for causing the agent or dye to fluoresce. Such wavelengths can be determined based on the agent or dye administered to the patient. In such implementations, the emitters can need to be highly precise in order to emit the required wavelengths to cause certain agents or dyes to fluoresce or activate.

[0199] In one implementation, the emitters 1302, 1304, and 1306 emit a laser mapping pattern for mapping the topology of a scene and / or for calculating the size of objects in a scene and the distances between them. In one implementation, an endoscopic imaging system is used in conjunction with multiple tools such as surgical knives, retractors, clamps, etc. In such implementations, each of the emitters 1302, 1304, and 1306 can emit a laser mapping pattern such that the laser mapping patterns are individually projected onto each tool. In such implementations, the laser mapping data for each tool can be analyzed to identify the distances between the tool and other objects in the scene.

[0200] In​FIG. 13B In embodiments of the system 1300, the emitters 1302, 1304, 1306 each deliver light to the collection region 1308 at different angles. The variation in angle can result in a variation in the location of the electromagnetic energy in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at the collection region 1308, the varying angles can result in different amounts of light entering different fibers. For example, the angles can result in a variation in intensity across the collection region 1308. In addition, the light from the different emitters can not be mixed uniformly, so some fibers can receive different amounts of different colors of light. The variation in color or intensity of the light in different fibers can result in non-optimal illumination of the scene. For example, the variation in delivered light or light intensity can result at the scene and in the captured image.

[0201] In one embodiment, an intervening optical element can be placed between the fiber bundle and the emitters 1302, 1304, 1306 to mix the different colors (wavelengths) of light before entering the fibers or other waveguides. Exemplary intervening optical elements include a diffuser, a mixing rod, one or more lenses, or other optical components to mix the light so that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle can have the same color. This mixing can result in the same color in each fiber, but, in some embodiments, can still result in different total brightness delivered to different fibers. In one embodiment, the intervening optical element can also spread or uniformly spread the light over the collection region so that each fiber carries the same total amount of light (e.g., the light can spread out in a top hat profile). A diffuser or mixing rod can result in loss of light.

[0202] Although the collection region 1308 is represented as a physical component in FIG. 13A some cases, the collection region 1308 can simply be the region in which light from the emitters 1302, 1304, and 1306 is delivered. In some cases, the collection region 1308 can include optical components, such as a diffuser, a mixing rod, a lens, or any other intervening optical component between the emitters 1302, 1304, 1306 and the output waveguide.

[0203] FIG. 13CAn embodiment of a light source 1300 is shown having emitters 1302, 1304, 1306 that provide light to a collection area 1308 at the same or approximately the same angle. The light is provided at an angle that is substantially normal to the collection area 1308. The light source 1300 includes a plurality of dichroic mirrors, including a first dichroic mirror 1310, a second dichroic mirror 1312, and a third dichroic mirror 1314. The dichroic mirrors 1310, 1312, 1314 include mirrors that reflect light of a first wavelength but transmit light of a second wavelength (or are transparent to it). For example, the third dichroic mirror 1314 can reflect blue laser light provided by the third emitter, while being transparent to the red and green light provided by the first emitter 1302 and the second emitter 1304, respectively. The second dichroic mirror 1312 can be transparent to light from the first emitter 1302, but reflective to light from the second emitter 1304. If other colors or wavelengths are included, the dichroic mirrors can be selected to reflect light corresponding to at least one emitter and be transparent to the other emitters. For example, the third dichroic mirror 1314 reflects light from the third emitter 1306, but is transparent to emitters behind it, such as the first emitter 1302 and the second emitter 1304. In embodiments where there are tens or hundreds of emitters, each dichroic mirror can reflect the corresponding emitter and the emitters in front of it, while being transparent to the emitters behind it. This can allow tens or hundreds of emitters to emit electromagnetic energy to the collection area 1308 at substantially the same angle.

[0204] Because these dichroic mirrors allow other wavelengths to be transmitted or pass through, each of these wavelengths can reach the collection area 1308 from the same angle and / or at the same center point or focal point. Providing light from the same angle and / or the same focal / center point can significantly improve reception and color mixing at the collection area 1308. For example, a particular fiber can receive different colors in the same proportion as they are transmitted / reflected by the emitters 1302, 1304, 1306 and the mirrors 1310, 1312, 1314. In contrast to embodiments of FIG. 13B Light mixing can be significantly improved at the collection area compared to embodiments of

[0205] FIG. 13CAn embodiment of a light source 1300 with emitters 1302, 1304, 1306 that also provide light to the collection region 1308 at the same or approximately the same angle is shown. For example, the light incident on the collection region 1308 is offset from the vertical by an angle 1316. The angle 1316 indicates the angle of the offset from the vertical. In one embodiment, the laser emitters 1302, 1304, 1306 can have a Gaussian cross-sectional intensity profile. As previously described, improved distribution of light energy between the fibers can be achieved by forming a more flat or top-hat shaped intensity profile. In one embodiment, as the angle 1316 is increased, the intensity across the collection region 1308 approaches a top-hat profile. For example, by increasing the angle 1316 until the profile is sufficiently flat, the top-hat profile can even approximate a non-flat output beam. The top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intervening optical components between the emitters 1302, 1304, 1306 and the output waveguide, fiber, or bundle of optical fibers.

[0206] FIG. 14 is a schematic diagram showing a single optical fiber 1402 outputting at the output via a diffuser 1404. In one embodiment, the optical fiber 1402 has a diameter of 500 microns, a numerical aperture of 0.65, and emits a light cone 1406 of about 70 or 80 degrees without the diffuser 1404. With the diffuser 1404, the light cone 1406 can have an angle of about 110 or 120 degrees. The light cone 1406 can be the majority of the place where all the light reaches and is uniformly distributed. The diffuser 1404 can allow for a more uniform distribution of electromagnetic energy of the scene observed by the image sensor.

[0207] In one embodiment, the lumen waveguide 210 includes a single plastic or glass optical fiber of about 500 microns. Plastic fibers are less expensive, but by coupling, diffusing, or other losses, their width can allow the fiber to carry a sufficient amount of light to the scene. For example, a smaller fiber can not be able to carry as much light or power as a larger fiber. The lumen waveguide 210 can include a single or multiple optical fibers. The lumen waveguide 210 can receive light directly from a light source or via a jumper waveguide. A diffuser can be used to widen the light output 206 to obtain a desired field of view of the image sensor 214 or other optical components.

[0208] While in FIGS. 13A-13CThree emitters are shown, but in some embodiments, a number of emitters ranging from one to hundreds or more can be used. The emitters can have different wavelengths or spectrums of light that they emit, and these lights can be used to continuously cover a desired portion of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectrums). The emitters can be configured to emit visible light such as red, green, and blue light, and can further be configured to emit hyperspectral emissions of electromagnetic radiation, fluorescence excitation wavelengths for causing reagents to fluoresce, and / or laser mapping patterns for calculating parameters and distances between objects in a scene.

[0209] FIG. 15 A portion of the electromagnetic spectrum 1500 is shown that is divided into twenty different sub-spectrums. The number of sub-spectrums is merely exemplary. In at least one embodiment, the spectrum 1500 can be divided into hundreds of sub-spectrums, each having a small waveband. The spectrum can extend from the infrared spectrum 1502, through the visible spectrum 1504, and into the ultraviolet spectrum 1506. The sub-spectrums each have a waveband 1508 that covers a portion of the spectrum 1500. Each waveband can be defined by an upper wavelength and a lower wavelength.

[0210] Hyperspectral imaging includes imaging information from the entire electromagnetic spectrum 1500. A hyperspectral pulse of electromagnetic radiation can include multiple sub-pulses across one or more portions of the electromagnetic spectrum 1500 or the entire electromagnetic spectrum 1500. A hyperspectral pulse of electromagnetic radiation can include a single partition of wavelengths of electromagnetic radiation. The resulting hyperspectral exposure frame includes information sensed by the pixel array following the hyperspectral pulse of electromagnetic radiation. Thus, a hyperspectral exposure frame can include data for any suitable partition of the electromagnetic spectrum 1500, and can include multiple exposure frames for multiple partitions of the electromagnetic spectrum 1500. In one embodiment, the hyperspectral exposure frame includes multiple hyperspectral exposure frames such that the combined hyperspectral exposure frames include data for the entire electromagnetic spectrum 1500.

[0211] In one embodiment, at least one emitter, such as a laser emitter, is included in the light source, such as light source 202, 1300, to provide complete and continuous coverage of the entire optical spectrum 1500. For example, a light source for providing coverage of the illustrated sub-spectra can include at least 20 different emitters, one for each sub-spectrum. In one embodiment, each emitter covers a 40 nanometer band of the spectrum. For example, one emitter can emit light within the 500 nm to 540 nm band, while another emitter can emit light within the 540 nm to 580 nm band. In another embodiment, the emitters can cover other sized bands, depending on the type of emitters available or the imaging needs. For example, a plurality of emitters can include a first emitter covering the 500 nm to 540 nm band, a second emitter covering the 540 nm to 640 nm band, and a third emitter covering the 640 nm to 650 nm band. Each emitter can cover a different segment of the electromagnetic spectrum ranging from far infrared, mid infrared, near infrared, visible light, near ultraviolet, and / or far ultraviolet. In some cases, multiple emitters of the same type or wavelength can be included to provide sufficient output power for imaging. The number of emitters required for a particular band can depend on the sensitivity of the monochromatic sensor to the band and / or the power output capability of the emitters in that band.

[0212] The band width and coverage provided by the emitters can be selected to provide any desired combination of the spectrum. For example, continuous coverage of the spectrum using very small band widths (e.g., 10 nm or less) can allow highly selective hyperspectral and / or fluorescence imaging. The band width can allow selective emission of the excitation wavelengths of one or more specific fluorescent agents. Additionally, the band width can allow selective emission of certain segments of the hyperspectral electromagnetic radiation for identifying specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths come from selectively activatable emitters, great flexibility in fluorescing one or more specific fluorescent agents during an examination can be achieved. Additionally, great flexibility in identifying one or more objects or processes by hyperspectral imaging can be achieved. Thus, more fluorescence and / or hyperspectral information can be achieved in less time and within a single examination, which would otherwise require multiple examinations, be delayed due to application or staining of dyes, etc.

[0213] FIG. 16is a schematic diagram showing a timing diagram 1600 for generating emission and readout of images. The solid lines represent readout (peaks 1602) and blanking periods (valleys) for capturing a series of exposure frames 1604-1614. The series of exposure frames 1604-1614 can include a series of repeated exposure frames that can be used to generate laser mapping, hyperspectral, and / or fluorescence data that can be overlaid on an RGB video stream. In one embodiment, a single image frame includes information from multiple exposure frames, with one exposure frame including red image data, another exposure frame including green image data, and another exposure frame including blue image data. Additionally, a single image frame can include one or more of hyperspectral image data, fluorescence image data, and laser mapping data. The multiple exposure frames are combined to produce a single image frame. The single image frame is an RGB image with hyperspectral imaging data. The series of exposure frames includes a first exposure frame 1604, a second exposure frame 1606, a third exposure frame 1608, a fourth exposure frame 1610, a fifth exposure frame 1612, and an Nth exposure frame 1626.

[0214] Additionally, the hyperspectral image data, fluorescence image data, and laser mapping data can be used in combination to identify key tissues or structures, and further measure the dimensions of those key tissues or structures. For example, the hyperspectral image data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, cancerous tissue, and the like. The location and identification of the key structures can be received from the corresponding system and also used to generate a topology of the key structures using the laser mapping data. For example, the corresponding system determines the location of a cancerous tumor based on the hyperspectral imaging data. Since the location of the cancerous tumor is known based on the hyperspectral imaging data, the topography and distance of the cancerous tumor can be calculated based on the laser mapping data. This example can also apply when a cancerous tumor or other structure is identified based on fluorescence imaging data.

[0215] In one embodiment, each exposure frame is generated based on at least one pulse of electromagnetic energy. The pulses of electromagnetic energy are reflected by the image sensor and detected, and subsequently read out in a subsequent readout (1602). Thus, each blanking period and readout results in an exposure frame for a particular electromagnetic energy spectrum. For example, the first exposure frame 1604 can be generated based on the spectrum of a first one or more pulses 1616, the second exposure frame 1606 can be generated based on the spectrum of a second one or more pulses 1618, the third exposure frame 1608 can be generated based on the spectrum of a third one or more pulses 1620, the fourth exposure frame 1610 can be generated based on the spectrum of a fourth one or more pulses 1622, the fifth exposure frame 1612 can be generated based on the spectrum of a fifth one or more pulses 2424, and the Nth exposure frame 1626 can be generated based on the spectrum of an Nth one or more pulses 1626.

[0216] Pulses 1616-1626 can include energy from a single emitter or a combination from two or more emitters. For example, the spectra included in a single readout cycle or within multiple exposure frames 1604-1614 can be selected for a desired examination or detection of a particular tissue or condition. According to one embodiment, one or more pulses can include visible spectra for generating RGB or black images, while one or more additional pulses are emitted to sense a spectral response to electromagnetic radiation at hyperspectral wavelengths. For example, pulse 1616 can include red light, pulse 1618 can include blue light, and pulse 1620 can include green light, while the remaining pulses 1622-1626 can include wavelengths and spectra for detecting particular tissue types, causing reagents to fluoresce, and / or mapping the topology of a scene. As another example, the pulses of a single readout cycle include spectra (e.g., different segments of the electromagnetic spectrum) generated by multiple different emitters that can be used to detect a particular tissue type. For example, if the combination of wavelengths causes a pixel to have a value that exceeds or falls below a threshold, the pixel can be classified as corresponding to a particular type of tissue. Each frame can be used to further narrow down the type of tissue present at that pixel (e.g., as well as each pixel in the image) to provide a very specific classification of the tissue and / or the state (diseased / healthy) of the tissue based on the spectral response of the tissue and / or whether a fluorescent reagent is present in the tissue.

[0217] Multiple frames 1604-1614 are shown as having different lengths of readout cycles and having pulses of different lengths or intensities. The blanking periods, pulse lengths or intensities, etc. can be selected based on the sensitivity of the monochromatic sensors to particular wavelengths, the power output capabilities of the emitters, and / or the carrying capacity of the waveguides.

[0218] In one embodiment, a dual image sensor can be used to obtain a three- dimensional image or video feed. Three-dimensional examination can allow for improved understanding of the three-dimensional structure of an examination region and mapping of different tissue or substance types within the region.

[0219] In one example implementation, a patient is provided with a fluorescent reagent, and the fluorescent reagent is configured to adhere to cancer cells. The fluorescent reagent is known to fluoresce when irradiated by a particular partition of electromagnetic radiation. The relaxation wavelength of the fluorescent reagent is also known. In the example implementation, the patient is imaged with an endoscopic imaging system as described herein. The endoscopic imaging system pulses partitions of light at red, green, and blue wavelengths to generate an RGB video stream of the interior of the patient's body. Additionally, the endoscopic imaging system pulses electromagnetic radiation at the excitation wavelength of the fluorescent reagent administered to the patient. In the example, the patient has cancer cells, and the fluorescent reagent has adhered to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength of the fluorescent reagent, the fluorescent reagent will fluoresce and emit the relaxation wavelength. If cancer cells are present in the scene imaged by the endoscopic imaging system, then the fluorescent reagent will also be present in the scene, and will emit its relaxation wavelength upon fluorescing due to the emission of the excitation wavelength. The endoscopic imaging system senses the relaxation wavelength of the fluorescent reagent, thereby sensing the presence of the fluorescent reagent in the scene. Because the fluorescent reagent is known to adhere to cancer cells, the presence of the fluorescent reagent is further indicative of the presence of cancer cells within the scene. The endoscopic imaging system thereby identifies the location of cancer cells within the scene. The endoscopic imaging system can also emit a laser mapping pulse scheme for generating a topology of the scene and calculating dimensions of objects within the scene. The location of the cancer cells (as identified by the fluorescent imaging data) can be combined with the topology and dimension information calculated based on the laser mapping data. Thus, the precise location, size, dimensions, and topology of the cancer cells can be identified. This information can be provided to a medical practitioner to aid in resecting the cancer cells. Additionally, this information can be provided to a robotic surgical system to enable the surgical system to resect the cancer cells.

[0220] In another example implementation, a patient is imaged with an endoscopic imaging system to identify quantitative diagnostic information about a lesion in the patient's tissue. In this example, the patient is suspected or known to have a disease that can be tracked with hyperspectral imaging to observe the progression of the disease in the patient's tissue. The endoscopic imaging system pulses light of red, green, and blue wavelengths to generate an RGB video stream of the inside of the patient's body. Additionally, the endoscopic imaging system pulses light of one or more hyperspectral wavelengths, allowing the system to "see through" some tissue and generate an image of the tissue affected by the disease. The endoscopic imaging system senses the reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue, identifying the location of the diseased tissue within the patient's body. The endoscopic imaging system can also emit a laser mapping pulse scheme to generate the topology of the scene and calculate the dimensions of objects within the scene. The location of the diseased tissue (as identified by the hyperspectral imaging data) can be combined with the topology and dimensional information calculated with the laser mapping data. Thus, the precise location, size, dimensions, and topology of the diseased tissue can be identified. This information can be provided to a medical practitioner to assist in resecting, imaging, or studying the diseased tissue. Additionally, this information can be provided to a robotic surgical system to enable the surgical system to resect the diseased tissue.

[0221] FIG. 17 is a schematic diagram of an imaging system 1700 with a single cut filter. The system 1700 includes an endoscope 1706 or other suitable imaging device with a light source 1708 for a light deficient environment. The endoscope 1706 includes an image sensor 1704 and a filter 1702 to filter out unwanted wavelengths of light or other electromagnetic radiation before reaching the image sensor 1704. The light source 1708 transmits light that can illuminate a surface 1712 in a light deficient environment such as a body cavity. Light 1710 reflects off the surface 1712 and passes through the filter 1702 before hitting the image sensor 1704.

[0222] The filter 1702 can be used in implementations where a fluorescent reagent or dye has been applied. In such embodiments, the light source 1708 emits an excitation wavelength for causing the fluorescent reagent or dye to fluoresce. Typically, the relaxation wavelength emitted by the fluorescent reagent or dye will have a different wavelength than the excitation wavelength. The filter 1702 can be selected to filter out the excitation wavelength and only allow the relaxation wavelength to pass through the filter and be sensed by the image sensor 1704.

[0223] In one embodiment, the optical filter 1702 is configured to filter out excitation wavelengths of electromagnetic radiation that cause a reagent or dye to fluoresce, such that only the intended relaxation wavelengths of the fluorescing reagent or dye are allowed to pass through the optical filter 1702 and reach the image sensor 1704. In one embodiment, the optical filter 1702 filters out fluorescent reagent excitation wavelengths of at least between 770 nm and 790 nm. In one embodiment, the optical filter 1702 filters out fluorescent reagent excitation wavelengths of at least between 795 nm and 815 nm. In one embodiment, the optical filter 1702 filters out fluorescent reagent excitation wavelengths of at least between 770 nm and 790 nm and between 795 nm and 815 nm. In these embodiments, the optical filter 1702 filters out the excitation wavelengths of the reagent and allows only the relaxation wavelengths of the fluorescent reagent to be read by the image sensor 1704. The image sensor 1704 can be a wavelength-agnostic image sensor, and the optical filter 1702 can be configured to allow the image sensor 1704 to receive only the relaxation wavelengths of the fluorescent reagent and not the emission excitation wavelengths of the reagent. The data determined by the image sensor 1704 can then be indicative of the presence of a key body structure, tissue, biological process, or chemical process determined by the location of the reagent or dye.

[0224] The optical filter 1702 can also be used in implementations where a fluorescent reagent or dye has not been applied. The optical filter 1702 can be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read by the image sensor 1704. The image sensor 1704 can be a monochromatic image sensor, such that pixels of a captured image above or below a threshold can be characterized as corresponding to certain spectral responses or fluorescent emissions. The spectral responses or fluorescent emissions determined by the pixels captured by the image sensor 1704 can be indicative of the presence of certain body tissues or structures, certain conditions, certain chemical processes, etc.

[0225] FIG. 18 is a schematic diagram of an imaging system 1800 having multiple cut filters. The system 1800 includes a scope 1806 or other suitable imaging device having a light source 1808 for a light deficient environment. The scope 1806 includes an image sensor 1804 and two optical filters 1802a, 1802b. It should be understood that in alternative embodiments, the system 1800 can include any number of optical filters, and the number of optical filters and the type of optical filters can be selected for certain purposes, e.g., to collect imaging information of particular body tissues, body conditions, chemical processes, etc. The optical filters 1802a, 1802b are configured to prevent the image sensor 1804 from sensing light or other electromagnetic radiation of unwanted wavelengths. The optical filters 1802a, 1802b can be configured to filter out unwanted wavelengths from white light or other electromagnetic radiation that can be emitted by the light source 1808.

[0226] With respect toFIG. 17 Further describing the invention, filters 1802a and 1802b can be used in embodiments where a fluorescent reagent or dye has been applied. Filters 1802a and 1802b can be configured to block the emission excitation wavelength of the reagent or dye and allow the image sensor 1804 to read only the relaxation wavelength of the reagent or dye. Furthermore, filters 1802a and 1802b can be used in embodiments where a fluorescent reagent or dye has not been applied. In such embodiments, filters 1802a and 1802b can be selected to allow wavelengths corresponding to the desired spectral response to pass through and be read by the image sensor 1804.

[0227] Multiple filters 1802a, 1802b can each be configured to filter out wavelengths of different ranges in the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than the desired wavelength range, and additional filters can be configured to filter out wavelengths shorter than the desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 1804.

[0228] In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 513 nm and 545 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 565 nm and 585 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 900 nm and 1000 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 417 nm and 475 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 520 nm and 545 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 617 nm and 645 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 760 nm and 795 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 795 nm and 815 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 370 nm and 420 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 600 nm and 670 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are configured to allow only certain fluorescent relaxation emissions to pass through the optical filters 1802a, 1802b and contact the image sensor 1804. In one embodiment, the first optical filter blocks electromagnetic radiation having a wavelength from about 770 nm to about 790 nm, and the second optical filter blocks electromagnetic radiation having a wavelength from about 795 nm to about 815 nm.

[0229] In one embodiment, the system 1800 includes multiple image sensors 1804, and can specifically include two image sensors for generating three-dimensional images. The image sensors 1804 can be color / wavelength agnostic, and configured to read electromagnetic radiation of any wavelength reflected from the surface 1812. In one embodiment, the image sensors 1804 are each color-dependent or wavelength-dependent, and configured to read electromagnetic radiation of a specific wavelength reflected from the surface 1812 and back to the image sensors 1804. Alternatively, the image sensors 1804 can include a single image sensor with multiple different pixel sensors configured to read different wavelengths or colors of light, such as a Bayer filter color filter array. Alternatively, the image sensors 1804 can include one or more color agnostic image sensors, which can be configured to read different wavelengths of electromagnetic radiation according to a pulse schedule, such as those shown in FIGS. 1-3. FIGS. 5-7E

[0230] FIG. 19 is a schematic diagram illustrating a system 1900 for mapping a surface and / or tracking an object in a light deficient environment by laser mapping imaging. In one embodiment, an endoscope 1906 pulses a grid array 1906 (which can be referred to as a laser mapping pattern) onto a surface 1904 in a light deficient environment. In one embodiment as shown in FIG. 19B, the grid array 1906 includes vertical hashes 1908 and horizontal hashes 1910. It should be understood that the grid array 1906 can include any suitable array for mapping the surface 1904, including, for example, a raster grid of discrete points, an occupancy grid map, a dot array, etc. Additionally, the endoscope 1906 can pulse multiple grid arrays 1906, and can, for example, pulse one or more separate grid arrays on each of multiple objects or structures within the light deficient environment. FIG. 19

[0231] In one embodiment, the system 1900 pulses the grid array 1906, which can be used to map the three-dimensional topology of a surface and / or track the location of an object, such as a tool or another device, in a light deficient environment. In one embodiment, the system 1900 provides data to a third party system or computer algorithm for determining the surface dimensions and configuration by way of light detection and ranging (LIDAR) mapping. The system 1900 can pulse light or electromagnetic radiation of any suitable wavelength in the grid array 1906, including, for example, ultraviolet light, visible light, light, and / or infrared or near-infrared light. The surface 1904 and / or objects within the environment can be mapped and tracked with very high resolution and with very high accuracy and precision.

[0232] ​​In one embodiment, system 1900 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. System 1900 can include a light engine having an emitter that generates one or more pulses of electromagnetic radiation and an internal lumen that transmits the one or more pulses of electromagnetic radiation to a distal tip of an endoscope within a light deficient environment, such as a body cavity. In one embodiment, at least a portion of the one or more pulses of electromagnetic radiation includes a laser mapping pattern that is emitted onto a surface within the light deficient environment, such as a surface of body tissue within the body cavity and / or a surface of a tool or other device. Endoscope 1906 can include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations within the light deficient environment.

[0233] In one embodiment, system 1900 includes a processor for determining a distance of an endoscope or tool from an object, such as surface 1904. The processor can also determine an angle between the endoscope or tool and the object. The processor can also determine surface area information about the object, including, for example, dimensions of a surgical tool, dimensions of a structure, dimensions of an anatomical structure, location information, and other location data and metrics. System 1900 can include one or more image sensors that provide image data to a control system for determining a distance of an endoscope or tool from an object, such as surface 1904. The image sensors can output information to a control system for determining an angle between the endoscope or tool and the object. Additionally, the image sensors can output information to a control system for determining surface area information about the object, dimensions of a surgical tool, dimensions of a structure, dimensions of an anatomical structure, location information, and other location data and metrics.

[0234] In one embodiment, the grid array 1906 is pulsed by the emitter of the endoscope 1906 at a sufficient speed such that the grid array 1906 is not visible to the user. In various implementations, seeing the grid array 1906 during an endoscopic imaging procedure and / or endoscopic surgery can distract the user. The grid array 1906 can be pulsed at a sufficiently short period such that the grid array 1906 cannot be detected by the human eye. In alternative embodiments, the endoscope 1906 pulses the grid array 1906 at a sufficient recurrence frequency such that the grid array 1906 is viewable by the user. In such embodiments, the grid array 1906 can be superimposed on the image of the surface 1904 on the display. The grid array 1906 can be superimposed on the black and white or RGB image of the surface 1904 such that the grid array 1906 is visible to the user during use of the system 1900. The user of the system 1900 can indicate whether the grid array 1906 should be superimposed on the image of the surface 1904 and / or whether the grid array 1906 should be visible to the user. The system 1900 can include a display that provides real-time measurements of the distance from the endoscope 1906 to the surface 1904 or another object within the light deficient environment. The display can also provide real-time surface area information about the surface 1904 and / or any objects, structures, or tools within the light deficient environment. The accuracy of the measurements can be precise to less than one millimeter.

[0235] In one embodiment, the system 1900 pulses multiple grid arrays 1906. In one embodiment, each of the multiple grid arrays 1906 corresponds to a tool or other device that is present within the light deficient environment. The precise location and parameters of each of the tools and other devices can be tracked by pulsing and sensing the multiple grid arrays 1906. The information generated by sensing the reflected grid arrays 1906 can be evaluated to identify the relative locations of the tools and other devices within the light deficient environment.

[0236] The endoscope 1906 can pulse electromagnetic radiation according to pulse schedules such as those shown herein, for example, this can also include pulsing the grid array 1906 and pulsing red, green, and blue light that is used to generate RGB images and also generate grid arrays 1906 that can be superimposed on the RGB images and / or for mapping and tracking surfaces 1904 and objects within the light deficient environment. The grid array 1906 can additionally be pulsed in conjunction with hyperspectral or fluorescence excitation wavelengths of electromagnetic radiation. Data from each of RGB imaging, laser mapping imaging, hyperspectral imaging, and fluorescence imaging can be combined to identify the location, size, and surface topology of key structures within the body.

[0237] In one embodiment, the endoscope 1906 includes one or more color agnostic image sensors. In one embodiment, the endoscope 1906 includes two color agnostic image sensors for generating three-dimensional images or maps of a light deficient environment. The image sensors can generate RGB images of the light deficient environment according to a pulse schedule as disclosed herein. Additionally, the image sensors can determine data for mapping the light deficient environment and tracking one or more objects within the light deficient environment based on data determined at the pulse grid array 1906. Additionally, the image sensors can determine spectral or hyperspectral data and fluorescence imaging data according to a pulse schedule, which can be modified by a user to suit the particular needs of an imaging procedure. In one embodiment, the pulse schedule includes red, green, and blue pulses and pulses of the grid array 1906 and / or pulses for generating hyperspectral image data and / or fluorescence image data. In various implementations, the pulse schedule can include any suitable combination of pulses of electromagnetic radiation according to the needs of a user. The frequency of repetition of different wavelengths of electromagnetic radiation can be determined based on, for example, the energy of certain pulses, the needs of a user, whether certain data, such as hyperspectral data and / or fluorescence imaging data, needs to be continuously updated or can be updated less frequently, etc.

[0238] The pulse schedule can be modified in any suitable manner and certain pulses of electromagnetic radiation can be repeated at any suitable frequency according to the needs of a user or a computer-implemented program for certain imaging procedures. For example, in embodiments in which surface tracking data generated based on the grid array 1906 is provided to a computer-implemented program for, for example, robotic surgery, the grid array 1906 can be pulsed more frequently than if the surface tracking data is provided to a user who visualizes the scene during an imaging procedure. In such embodiments in which the surface tracking data is used for robotic surgery, the surface tracking data can need to be updated more frequently or can need to be extremely accurate so that the computer-implemented program can perform the robotic surgery with precision and accuracy.

[0239] In one embodiment, the system 1900 is configured to generate an occupancy grid map including an array of cells divided into a grid. The system 1900 is configured to store a height value for each of the respective grid cells to determine a surface map of a three-dimensional environment in the light deficient environment.

[0240] FIG. 20A and FIG. 20BPerspective and side views of an implementation of a monolithic sensor 2000 having multiple pixel arrays for producing a three-dimensional image are shown, respectively, in accordance with the teachings and principles of the present disclosure. Such an implementation can be desirable for three-dimensional image capture, where during use the two pixel arrays 2002 and 2004 can be offset. In another implementation, the first pixel array 2002 and the second pixel array 2004 can be dedicated to receiving electromagnetic radiation of a predetermined wavelength range, where the first pixel array is dedicated to electromagnetic radiation of a different wavelength range than the second pixel array.

[0241] FIG. 21A and FIG. 21B Perspective and side views of an implementation of an imaging sensor 2100 built on multiple substrates are shown, respectively. As shown, multiple pixel columns 2104 forming the pixel array are located on a first substrate 2102 and multiple circuit columns 2108 are located on a second substrate 2106. Electrical connections and communication between a pixel column and its associated or corresponding circuit column are also shown. In one implementation, the image sensor can have a pixel array separate from all or most of the supporting circuitry, while it can otherwise be fabricated with its pixel array and supporting circuitry on a single, monolithic substrate / chip. The present disclosure can use at least two substrates / chips that will be stacked together using three-dimensional stacking technology. The first substrate / chip 2102 of the two substrates / chips can be processed using image CMOS technology. The first substrate / chip 2102 can consist of only a pixel array, or can consist of a pixel array surrounded by limited circuitry. The second or subsequent substrate / chip 2106 can be processed using any technology, not necessarily from image CMOS technology. The second substrate / chip 1306 can be, but is not limited to, a high-density digital technology in order to integrate various and multiple functions into very limited space or area on the substrate / chip, or a mixed-mode or analog technology in order to integrate, for example, precise analog functions, or an RF technology in order to implement wireless capabilities, or a MEMS (Micro Electro Mechanical System) in order to integrate MEMS devices. The image CMOS substrate / chip 2102 can be stacked with the second or subsequent substrate / chip 2106 using any three-dimensional technology. The second substrate / chip 2106 can support the majority or most of the circuitry that would otherwise be implemented in the first image CMOS chip 2102 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increase the overall system area while keeping the pixel array size constant and optimized as much as possible. Electrical connections between the two substrates / chips can be done through interconnects, which can be bond wires, lugs, and / or TSVs (Through Silicon Vias).

[0242] FIG. 22A and FIG. 22BPerspective and side views of a particular implementation of an imaging sensor 2200 with multiple pixel arrays for producing a three-dimensional image are shown. A three-dimensional image sensor can be built on multiple substrates and can include multiple pixel arrays and other associated circuitry, where multiple columns of pixels 2204a forming a first pixel array and multiple columns of pixels 2204b forming a second pixel array are on respective substrates 2202a and 2202b, respectively, and multiple columns of circuitry 2208a and 2208b are on a separate substrate 2206. Electrical connections and communication between the columns of pixels and associated or corresponding columns of circuitry are also shown.

[0243] Multiple pixel arrays can sense information simultaneously, and information from the multiple pixel arrays can be combined to generate a three-dimensional image. In one embodiment, an endoscopic imaging system includes two or more pixel arrays that can be deployed to generate three-dimensional imaging. The endoscopic imaging system can include an emitter to emit a pulse of electromagnetic radiation during a blanking period of the pixel arrays. The pixel arrays can be synchronized such that, for the two or more pixel arrays, an optical black pixel is read simultaneously (i.e., a blanking period occurs). The emitter can emit a pulse of electromagnetic radiation for charging each of the two or more pixel arrays. The two or more pixel arrays can read their respective charged pixels simultaneously, such that a readout period of the two or more pixel arrays occurs simultaneously or approximately simultaneously. In one embodiment, the endoscopic imaging system includes multiple emitters, where each emitter is individually synchronized with one or more of the multiple pixel arrays. Information from the multiple pixel arrays can be combined to generate three-dimensional image frames and video streams.

[0244] It should be understood that the teachings and principles of the present disclosure can be used in a reusable device platform, a limited use device platform, a repositionable use device platform, or a single use / disposable device platform without departing from the scope of the present disclosure. It should be understood that in a reusable device platform, an end user is responsible for cleaning and sterilization of the device. In a limited use device platform, a device can be used a certain defined number of times before becoming inoperable. A typical new device has been sterilized prior to delivery and should be cleaned and sterilized by the end user prior to other uses if used for other purposes. In a repositionable use device platform, a third party can reprocess (e.g., clean, package, and sterilize) single use devices for additional use at a lower cost than a new unit. In a single use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0245] EMBODIMENTS

[0246] The following embodiments relate to preferred features of additional embodiments:

[0247] Example 1 is a system. A system is disclosed that includes an emitter for emitting pulses of electromagnetic radiation and an image sensor including an array of pixels for sensing reflected electromagnetic radiation. The system includes an electromagnetic sensor for sensing energy emitted by the emitter. The system includes a controller in electronic communication with the emitter, the electromagnetic sensor, and the image sensor, wherein the controller is configured to synchronize timing of the emitter and the image sensor to generate a plurality of exposure frames. The system is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter include a laser mapping pattern.

[0248] Example 2 is the system of Example 1, wherein the emitter includes a laser beam and the laser beam includes a plurality of discrete laser units, each laser unit capable of emitting electromagnetic radiation.

[0249] Example 3 is the system of any of Examples 1-2, wherein the electromagnetic sensor is embedded within the emitter, and wherein the electromagnetic sensor senses the energy emitted by the emitter by sensing energy emitted by at least one, but less than all, of the plurality of discrete laser units within the laser beam.

[0250] Example 4 is the system of any of Examples 1-3, wherein: the electromagnetic sensor provides a value of the energy emitted by the emitter to the controller; and the controller controls a duty cycle of the emitter based on a predetermined known output and the value of the energy emitted by the emitter such that the emitter illuminates a scene with a desired exposure of an image captured by the image sensor.

[0251] Example 5 is the system of any of Examples 1-4, wherein: the emitter includes a plurality of laser beams and each of the plurality of laser beams includes a plurality of laser units; the system includes a dedicated electromagnetic sensor for each of the plurality of laser beams such that energy emitted by each of the plurality of laser beams is sensed by a different dedicated electromagnetic sensor; and the dedicated electromagnetic sensors are configured to sense energy emitted by at least one, but less than all, of the plurality of laser units of the laser beam for which it is assigned.

[0252] Example 6 is the system of any of Examples 1-5, wherein the controller comprises one or more processors configurable to execute instructions stored in a non-transitory computer readable storage medium, the instructions comprising: receiving, from the electromagnetic sensor, a value of the energy emitted by the emitter; comparing the value of the energy emitted by the emitter to a desired light output of the emitter; determining whether the value of the energy emitted by the emitter is within an acceptable tolerance with respect to the desired light output of the emitter; and in response to the value of the energy emitted by the emitter not being within the acceptable tolerance, adjusting a duration of at least one of the pulses of electromagnetic radiation.

[0253] Example 7 is the system of any of Examples 1-6, wherein the emitter comprises a plurality of laser beams, and each of the plurality of laser beams comprises a plurality of laser units, wherein the plurality of laser beams comprises: a red laser beam to emit electromagnetic radiation of a red wavelength; a green laser beam to emit electromagnetic radiation of a green wavelength; a blue laser beam to emit electromagnetic radiation of a blue wavelength; and a laser mapping beam to emit the laser mapping pattern.

[0254] Example 8 is the system of any of Examples 1-7, wherein the laser mapping beam is configured to emit the laser mapping pattern such that one or more of a distance, a dimension, or a three-dimensional topography map can be calculated based on the reflected electromagnetic radiation sensed by the image sensor in response to the emission of the laser mapping pattern.

[0255] Example 9 is the system of any of Examples 1-8, wherein the pixel array of the image sensor senses reflected electromagnetic radiation during a readout period of the pixel array to generate the plurality of exposure frames, wherein the readout period is a duration when active pixels in the pixel array are read out.

[0256] Example 10 is the system of any of Examples 1-9, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter is a fluorescence excitation wavelength to cause a reagent to fluoresce, wherein the fluorescence excitation wavelength comprises one or more of electromagnetic radiation having a wavelength from about 770 nm to about 790 nm or about 795 nm to about 815 nm.

[0257] Example 11 is the system of any of Examples 1-10, wherein the emitter is configured to emit, during a pulse duration, a plurality of sub-pulses of electromagnetic radiation having a sub-duration shorter than the pulse duration.

[0258] Example 12 is the system of any of Examples 1-11, wherein one or more of the pulses of electromagnetic radiation emitted by the emitter comprises electromagnetic radiation emitted at two or more wavelengths simultaneously as a single pulse or single sub-pulse.

[0259] Example 13 is the system of any of Examples 1-12, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter are fluorescence excitation emissions that result in a fluorescence exposure frame generated by the image sensor, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding system that determines a location of a key tissue structure within a scene based on the fluorescence exposure frame.

[0260] Example 14 is the system of any of Examples 1-13, wherein the fluorescence excitation emissions comprise each of: electromagnetic radiation having a wavelength from about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength from about 795 nm to about 815 nm.

[0261] Example 15 is the system of any of Examples 1-14, wherein the controller is further configured to: receive the location of the key tissue structure from the corresponding system; generate an overlay frame comprising the location of the key tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the key tissue structure within the scene.

[0262] Example 16 is the system of any of Examples 1-15, wherein sensing reflected electromagnetic radiation by the array of pixels comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and / or dimensions of one or more objects within the scene; provide the location of the key tissue structure to the corresponding laser mapping system; and receive a topology and / or dimensions of the key tissue structure from the corresponding laser mapping system. The system such that the key structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

[0263] Example 17 is the system of any of Examples 1-16, wherein the controller is configured to synchronize timing of the pulses of electromagnetic radiation during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the array of pixels and a start of a next subsequent readout of active pixels in the array of pixels.

[0264] Example 18 is the system of any of Examples 1-17, wherein two or more pulses of electromagnetic radiation emitted by the emitter result in two or more instances of reflected electromagnetic radiation that are sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.

[0265] Example 19 is the system of any of Examples 1-18, wherein the image sensor comprises a first image sensor and a second image sensor such that the image sensor is capable of generating a three-dimensional image.

[0266] Example 20 is the system of any of Examples 1-19, wherein the emitter is configured to repeatedly emit a sequence of pulses of electromagnetic radiation sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to a pulse of electromagnetic radiation.

[0267] Example 21 is the system of any of Examples 1-20, wherein the pulses of electromagnetic radiation are emitted in a pattern of different wavelengths of electromagnetic radiation, and wherein the emitter repeatedly emits the pattern of different wavelengths of electromagnetic radiation.

[0268] Example 22 is the system of any of Examples 1-21, wherein at least a portion of the pulses of electromagnetic radiation comprise a red emission, a green emission, a blue emission, and a laser mapping emission such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red emission, the green emission, the blue emission, and the laser mapping emission can be processed to generate a red-green-blue (RGB) image frame comprising superimposed laser mapping data.

[0269] Example 23 is the system of any of Examples 1-22, wherein at least a portion of the pulses of electromagnetic radiation comprise a luminance emission, a red chrominance emission, a blue chrominance emission, and a laser mapping emission such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the luminance emission, the red chrominance emission, the blue chrominance emission, and the laser mapping emission can be processed to generate a YCbCr image frame comprising superimposed laser mapping data.

[0270] Example 24 is the system of any of Examples 1-23, further comprising an operational amplifier circuit in electronic communication with the electromagnetic sensor.

[0271] Example 25 is the system of any of Examples 1-24, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each exposure frame of the plurality of exposure frames corresponds to a pulse of electromagnetic radiation emitted by the emitter.

[0272] Example 26 is the system of any of Examples 1-25, wherein the emitter comprises a laser beam, and the system further comprises an optical fiber connected to the laser beam, wherein the electromagnetic sensor is connected to the optical fiber.

[0273] Example 27 is the system of any of Examples 1-25, wherein the electromagnetic sensor is a photodiode.

[0274] Example 28 is the system of any of Examples 1-27, wherein the emitter comprises a plurality of laser beams for emitting a plurality of different wavelengths of electromagnetic radiation, and the system comprises a plurality of electromagnetic sensors for independently sensing each laser beam of the plurality of laser beams.

[0275] Example 29 is the system of any of Examples 1-28, wherein the emitter comprises a laser beam, and the electromagnetic sensor is disposed in an optical fiber connected to the laser beam.

[0276] Example 30 is the system of any of Examples 1-29, further comprising an operational amplifier circuit, wherein the electromagnetic sensor is in electronic communication with the operational amplifier circuit.

[0277] Example 31 is the system of any of Examples 1-30, wherein the controller comprises one or more processors configurable to execute instructions stored in a non-transitory computer readable storage medium, the instructions comprising: determining an average pixel value of pixels in the pixel array based on an exposure level of the image sensor; calculating a measured exposure level of the image sensor based on the average pixel value; calculating an error measurement based on the measured exposure level and a desired exposure level; determining whether the error measurement is within a tolerance threshold; and in response to the error measurement not being within the tolerance threshold, adjusting a wavelength and / or duration of at least one of the pulses of electromagnetic radiation to bring the error measurement within the tolerance threshold.

[0278] Example 32 is the system of any of Examples 1-31, further comprising a light filter that filters electromagnetic radiation having a wavelength from about 770 nm to about 790 nm.

[0279] Example 33 is the system of any of Examples 1-32, further comprising a filter that filters electromagnetic radiation having a wavelength from about 795 nm to about 815 nm.

[0280] Example 34 is the system of any of Examples 1-33, wherein sensing reflected electromagnetic radiation by the array of pixels comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, wherein the laser mapping exposure frame comprises information for determining real-time measurements, the information comprising one or more of: a distance from an endoscope to an object; an angle between an endoscope and the object; or surface topology information about the object.

[0281] Example 35 is the system of any of Examples 1-34, wherein the laser mapping exposure frame comprises information for determining the real-time measurements with an accuracy of less than 10 centimeters.

[0282] Example 36 is the system of any of Examples 1-35, wherein the laser mapping exposure frame comprises information for determining the real-time measurements with an accuracy of less than one millimeter.

[0283] Example 37 is the system of any of Examples 1-36, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter comprises a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

[0284] Example 38 is the system of any of Examples 1-37, wherein the laser mapping pattern emitted by the emitter comprises a first output and a second output that are independent of each other, wherein the first output is for light illumination and the second output is for tool tracking.

[0285] Example 39 is the system of any of Examples 1-38, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter is a hyperspectral emission that results in a hyperspectral exposure frame produced by the image sensor, and wherein the controller is configured to provide the hyperspectral exposure frame to a corresponding hyperspectral system that determines a location of a critical tissue structure within a scene based on the hyperspectral exposure frame.

[0286] Example 40 is the system of any of Examples 1-39, wherein the hyperspectral emission comprises one or more of: electromagnetic radiation having a wavelength from about 513 nm to about 545 nm; electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; or electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm.

[0287] Example 41 is the system of any of Examples 1-40, wherein the controller is further configured to: receive the location of the critical tissue structure from the corresponding hyperspectral system; generate an overlay frame comprising the location of the critical tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

[0288] Example 42 is the system of any of Examples 1-41, wherein sensing the reflected electromagnetic radiation by the array of pixels comprises generating a laser mapping exposure frame by sensing the reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and / or dimensions of one or more objects within the scene; provide the location of the critical tissue structure to the corresponding laser mapping system; and receive a topology and / or dimensions of the critical tissue structure from the corresponding laser mapping system.

[0289] Example 43 is the system of any of Examples 1-42, wherein the critical structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

[0290] It should be appreciated that various features disclosed herein provide significant advantages over the state of the art. The following claims are in the example of some of those features.

[0291] In the foregoing Detailed Description, various features are grouped together in example embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the innovations apply to or require more features than the claims do. Rather, inventive aspects lie in the novel features disclosed and not necessarily in particular combinations with other features.

[0292] It should be appreciated that any of the above-described arrangements, embodiments and implementations can be combined in single embodiments.

[0293] It should be appreciated that the preceding description is merely exemplary of the principles of the application. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the present disclosure. Specifically, those skilled in the art will recognize that the principles of the present disclosure can be applied to any device that includes a display and a camera. Accordingly, the attached claims are intended to cover all such modifications and adaptations that fall within the scope of the present disclosure.

[0294] Thus, numerous modifications can be made to the specific implementations described above without departing from the scope of the present disclosure which is not to be limited to particular forms or arrangements of parts and steps disclosed. Rather, the scope of the disclosure is that of the appended claims, including modifications made to the disclosure in the same way as those made to embodiments of all applications which follow the same or similar principles as taught herein.

[0295] Additionally, functions described herein can be performed in one or more of: hardware, software, firmware, digital components, or analog components, as appropriate or desired. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to perform one or more of the systems and processes described herein. Certain terms are used throughout the following description and claims to refer to particular features or aspects of the disclosure. As one skilled in the art will appreciate, these features are exemplary and do not limit the scope of the disclosure. Embodiments can be implemented in any of various forms, from about which there can be multiple possible variations. Furthermore, specific embodiments of the present disclosure can be implemented in any of a variety of systems including handheld devices, laptops, desktops, servers, mobile telephones, tablet computers, netbooks, etc.

[0296] The specific implementations described above are provided for the purpose of illustration and description. They are not intended to be exhaustive or to limit the disclosure to the precise forms described. Many modifications and variations are possible in light of the above teaching. Further, it should be noted that any or all of the aforementioned alternative implementations can be used in any desired combination, to form additional hybrid implementations of the disclosure.

[0297] Additionally, although specific implementations of the disclosure have been described and illustrated, the disclosure is not to be limited to the specific forms or arrangements of parts so described and illustrated. The scope of the disclosure is to be defined by the claims appended hereto and including any future claims, and their equivalents, as issued by the patent office of the various jurisdictions of the world.

Claims

1. An endoscope system comprising: an endoscope; a transmitter to emit pulses of electromagnetic radiation; an electromagnetic sensor to sense energy emitted by the transmitter; an image sensor comprising an array of pixels to sense reflected electromagnetic radiation; a controller in electronic communication with the transmitter, the electromagnetic sensor, and the image sensor, wherein the controller is configured to synchronize timing of the transmitter and the image sensor; wherein at least a portion of the pulses of electromagnetic radiation emitted by the transmitter comprises a laser mapping pattern; wherein the image sensor is configured to generate a plurality of exposure frames, wherein each exposure frame of the plurality of exposure frames corresponds to a pulse of electromagnetic radiation emitted by the transmitter; wherein a portion of the plurality of exposure frames comprises a laser mapping exposure frame.

2. The endoscope system of claim 1, wherein, the transmitter comprises a zoned laser beam to emit wavelengths of the electromagnetic spectrum, and the laser beam comprises a plurality of discrete laser units, each discrete laser unit capable of emitting electromagnetic radiation.

3. The endoscope system of claim 2, wherein, the electromagnetic sensor is embedded within the transmitter, and wherein the electromagnetic sensor senses the energy emitted by the transmitter by sensing energy emitted by at least one, but less than all, of the plurality of discrete laser units within the laser beam.

4. The endoscope system of claim 1, wherein: the electromagnetic sensor provides a value of the energy emitted by the transmitter to the controller; and the controller controls a duty cycle of the transmitter based on a predetermined known output and the value of the energy emitted by the transmitter such that the transmitter illuminates a scene with a desired exposure of an image captured by the image sensor.

5. The endoscope system of claim 1, wherein: the transmitter comprises a plurality of laser beams, and each laser beam of the plurality of laser beams comprises a plurality of laser units; the endoscope system comprises a dedicated electromagnetic sensor for each laser beam of the plurality of laser beams such that energy emitted by each laser beam of the plurality of laser beams is sensed by a different dedicated electromagnetic sensor; and the dedicated electromagnetic sensor is configured to sense energy emitted by at least one, but less than all, of the plurality of laser units of the laser beam assigned thereto.

6. The endoscope system of claim 1, wherein, the controller comprises one or more processors capable of being configured to execute instructions stored in a non-transitory computer readable storage medium, the instructions comprising: receiving a value of the energy emitted by the transmitter from the electromagnetic sensor; comparing the value of the energy emitted by the transmitter to a desired light output of the transmitter; determining whether the value of the energy emitted by the transmitter is within a tolerance threshold relative to the desired light output of the transmitter; and in response to the value of the energy emitted by the emitter not being within the tolerance threshold, adjusting a wavelength and / or a duration of at least one of the pulses of electromagnetic radiation to cause the value of the energy emitted by the emitter to be within the tolerance threshold.

7. The endoscope system of claim 1, wherein, The controller includes one or more processors that can be configured to execute instructions stored in a non-transitory computer-readable storage medium, the instructions including: determining an average pixel value of pixels in the pixel array based on an exposure level of the image sensor; calculating a measured exposure level of the image sensor based on the average pixel value; calculating an error measurement based on the measured exposure level and a desired exposure level; determining whether the error measurement is within a tolerance threshold; and in response to the error measurement not being within the tolerance threshold, adjusting a wavelength and / or a duration of at least one of the pulses of electromagnetic radiation to cause the error measurement to be within the tolerance threshold.

8. The endoscope system of claim 1, wherein, The emitter includes a plurality of laser beams, and each of the plurality of laser beams includes a plurality of laser units, wherein the plurality of laser beams includes: a red laser beam to emit electromagnetic radiation at a red wavelength; a green laser beam to emit electromagnetic radiation at a green wavelength; a blue laser beam to emit electromagnetic radiation at a blue wavelength; and a laser mapper beam to emit the laser mapping pattern.

9. The endoscope system of claim 8, wherein, The laser mapper beam is configured to emit the laser mapping pattern such that one or more of a distance, a size, or a three-dimensional topography map can be calculated based on the reflected electromagnetic radiation sensed by the image sensor in response to the emission of the laser mapping pattern.

10. The endoscope system of claim 1, further comprising an operational amplifier circuit in electronic communication with the electromagnetic sensor.

11. The endoscope system of claim 1, wherein, The pixel array of the image sensor senses the reflected electromagnetic radiation during a readout period of the pixel array to generate the plurality of exposure frames, wherein the readout period is a duration of time when active pixels in the pixel array are read.

12. The endoscope system of claim 1, wherein, The emitter is configured to emit a plurality of sub-pulses of electromagnetic radiation during a pulse duration, the sub-pulses having a sub-duration that is shorter than the pulse duration.

13. The endoscope system of claim 1, wherein, One or more of the pulses of electromagnetic radiation emitted by the emitter include electromagnetic radiation emitted at two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

14. The endoscope system of claim 1, wherein, At least a portion of the pulses of electromagnetic radiation emitted by the emitter are fluorescence excitation emissions that result in a fluorescence exposure frame produced by the image sensor, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding fluorescence system that determines a location of a key tissue structure within a scene based on the fluorescence exposure frame.

15. The endoscope system of claim 14, wherein, The fluorescence excitation emissions include one or more of: electromagnetic radiation having a wavelength from 770 nm to 790 nm; or electromagnetic radiation having a wavelength from 795 nm to 815 nm.

16. The endoscope system of claim 14, wherein, The controller is further configured to: receive the location of the critical tissue structure from the corresponding fluorescence system; generate an overlay frame including the location of the critical tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

17. The endoscope system of claim 16, wherein, sensing the reflected electromagnetic radiation by the pixel array includes generating the laser mapping exposure frame by sensing the reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topology of the scene and / or dimensions of one or more objects within the scene; provide the location of the critical tissue structure to the corresponding laser mapping system; and receive the topology and / or dimensions of the critical tissue structure from the corresponding laser mapping system.

18. The endoscope system of claim 17, wherein, The critical tissue structure includes one or more of a nerve, a ureter, a blood vessel, a blood flow, or a tumor.

19. The endoscope system of claim 17, wherein, The critical tissue structure includes an artery.

20. The endoscope system of claim 1, wherein, The controller is configured to synchronize timing of the pulses of electromagnetic radiation during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a start of a next subsequent readout of active pixels in the pixel array.

21. The endoscope system of claim 1, wherein, The two or more pulses of electromagnetic radiation emitted by the emitter result in two or more instances of reflected electromagnetic radiation that are sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.

22. The endoscope system of claim 1, wherein, The emitter is configured to repeatedly emit a sequence of pulses of electromagnetic radiation sufficient to generate a video stream including a plurality of image frames, wherein each image frame in the video stream includes data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to a pulse of electromagnetic radiation.

23. The endoscope system of claim 1, wherein, The pulses of electromagnetic radiation are emitted in a pattern of different wavelengths of electromagnetic radiation, and wherein the emitter repeatedly the pattern of different wavelengths of electromagnetic radiation.

24. The endoscope system of claim 1, wherein, At least a portion of the pulses of electromagnetic radiation include a red emission, a green emission, a blue emission, and a laser mapping emission, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the red emission, the green emission, the blue emission, and the laser mapping emission can be processed to generate a red-green-blue (RGB) image frame that includes superimposed laser mapping data.

25. The endoscope system of claim 1, wherein, At least a portion of the pulses of electromagnetic radiation include a luminance emission, a red chrominance emission, a blue chrominance emission, and a laser mapping emission, such that reflected electromagnetic radiation sensed by the pixel array corresponding to each of the luminance emission, the red chrominance emission, the blue chrominance emission, and the laser mapping emission can be processed to generate a YCbCr image frame that includes superimposed laser mapping data.

26. The endoscope system of claim 1, wherein, sensing reflected electromagnetic radiation by the pixel array includes generating the laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, wherein the laser mapping exposure frame includes information for determining real-time measurements, the information including one or more of: a distance from an endoscope to an object; an angle between an endoscope and the object; or surface topography information about the object.

27. The endoscope system of claim 26, wherein, the laser mapping exposure frame includes information for determining the real-time measurements with an accuracy of less than 10 centimeters.

28. The endoscope system of claim 26, wherein, the laser mapping exposure frame includes information for determining the real-time measurements with an accuracy of less than one millimeter.

29. The endoscope system of claim 1, wherein, at least a portion of the electromagnetic radiation pulses emitted by the emitter includes a plurality of tool-specific laser mapping patterns for each of a plurality of tools within a scene.

30. The endoscope system of claim 1, wherein, the laser mapping pattern emitted by the emitter includes a first output and a second output independent of one another, wherein the first output is for light illumination and the second output is for tool tracking.

Images