Laser scanning and tool tracking imaging in light deficient environments

Abstract

Systems, methods, and apparatuses for laser scanning in a light deficient environment are disclosed. A system includes an emitter to emit pulses of electromagnetic radiation and an image sensor including an array of pixels to sense reflected electromagnetic radiation. The system includes a controller including a processor in electrical communication with the image sensor and the emitter, where the controller synchronizes timing of the pulses of electromagnetic radiation during a blanking period of the image sensor. The system causes at least a portion of the pulses of electromagnetic radiation emitted by the emitter to include a laser scanning pattern.

Description

Technical Field

[0001] This patent application relates to digital imaging, and more particularly to laser scanning and tool tracking imaging in low-light environments. Background Technology

[0002] Advances in technology have enabled advancements in medical imaging capabilities. Endoscopes can be used to observe the inside of the body and examine the interior of organs or cavities. Endoscopes are used to investigate patient symptoms, confirm diagnoses, or provide medical treatment. Medical endoscopes can be used to observe a variety of body systems and parts, such as the gastrointestinal tract, respiratory tract, urethra, and abdominal cavity, through small incisions. Endoscopes are also used in surgical procedures, such as orthopedic surgery, surgery on joints or bones, surgery on the nervous system, and surgery within the abdominal cavity.

[0003] In some cases of endoscopic imaging, observing the color space may be beneficial or necessary. A digital color image comprises at least three layers, or "color channels," that accumulate to form an image with a range of hues. Each color channel measures the intensity and chromaticity of light in a spectral band. Typically, a digital color image comprises color channels with red, green, and blue spectral bands (this may be referred to as a red-green-blue or RGB image). Each of the red, green, and blue color channels includes luminance information for the red, green, or blue spectral band. The luminance information from the individual red, green, and blue layers is combined to generate a color image. Because a color image is composed of individual layers, a conventional digital camera image sensor includes a color filter array that allows red, green, and blue visible light wavelengths to strike selected pixel sensors. Each individual pixel sensor element is sensitive to a red, green, or blue wavelength and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image. These at least three different types of pixel sensors consume a significant amount of physical space, making it impossible to fit the entire pixel array into the small distal end of an endoscope.

[0004] Because conventional image sensors cannot be fitted into the distal end of an endoscope, they are traditionally located in the endoscope's handheld unit, which is held by the operator and not placed within the body cavity. In such endoscopes, light travels along the length of the endoscope from the handheld unit to the distal end. This configuration has significant limitations. Endoscopes with this configuration are delicate and can easily become misaligned or damaged by impacts or shocks during normal use. This can significantly degrade image quality, and the endoscope requires frequent repair or replacement.

[0005] Traditional endoscopes with image sensors housed in a handheld unit are limited to capturing only color images. However, in some implementations, it may be desirable to capture images using laser scanning or tool-tracking image data in addition to color image data. Laser scanning imaging can capture the surface shape of objects and panoramas and measure distances between objects within a scene. In some implementations, it may be desirable to measure distances and surface shapes within body cavities during endoscopic imaging procedures.

[0006] However, applications of laser scanning technologies known in the art typically require highly specialized equipment that may be unavailable for many applications. Furthermore, laser scanning technologies offer a limited field of view and often must be used in conjunction with multiple separate systems. In the context of endoscopic medical imaging procedures, all sensors must be fitted into a small physical area within a body cavity. In some cases, the geographic area may be particularly small and may only accommodate a very small tip of the endoscope. Therefore, medical endoscopes known in the art are necessarily small and cannot accommodate multiple different imaging and ranging systems. Thus, it is desirable to develop endoscopic imaging systems capable of generating laser scanning data in small spaces such as body cavities.

[0007] In light of the above, this paper describes systems, methods, and apparatuses for laser scanning and tool tracking imaging in low-light environments. Such systems, methods, and apparatuses can provide multiple datasets for identifying key structures within the body and providing accurate and valuable information about body cavities. Attached Figure Description

[0008] Non-limiting and incomplete embodiments of this disclosure are described with reference to the following accompanying drawings, wherein, unless otherwise specified, similar reference numerals in the various views indicate similar parts. The advantages of this disclosure will be better understood with reference to the following description and drawings, wherein:

[0009] Figure 1 This is a schematic diagram of a system with paired emitters and pixel arrays for digital imaging in low-light environments;

[0010] Figure 2 It is a system used to provide illumination to light-deficient environments used for endoscopic imaging;

[0011] Figure 2A This is a schematic diagram of the hardware of the complementary system;

[0012] Figures 3A-3D This is a diagram illustrating the operation loop of a sensor used to construct an image frame;

[0013] Figure 4A It is a graphical representation of the operation of the implementation scheme of the electromagnetic transmitter;

[0014] Figure 4B It is a graphical representation of changing the duration and magnitude of the emitted electromagnetic pulse to provide exposure control;

[0015] Figure 5 It is Figures 3A-4B The illustration of an embodiment of the present disclosure is shown in the form of a sensor operating cycle, an electromagnetic transmitter, and a combination of emitted electromagnetic pulses, illustrating an imaging system during operation.

[0016] Figure 6A This is a schematic diagram of a method for recording video using full-spectrum light during the time period from t(0) to t(1);

[0017] Figure 6B This is a schematic diagram of the process of recording video by pulsating partial spectral light during the time period from t(0) to t(1);

[0018] Figures 7A-7E A schematic diagram illustrates a method for recording video frames of both full-spectrum light and partitioned-spectrum light within a time interval;

[0019] Figure 8 This is a schematic diagram of the process flow for simultaneously adjusting the electromagnetic transmitter and the image sensor;

[0020] Figure 9 This is a schematic diagram of the process for adjusting the sensitivity of an image sensor;

[0021] Figure 10 It is a schematic diagram of the process flow for simultaneously adjusting the image signal processor and the transmitter based on the histogram of the frame cycle;

[0022] Figure 11 It is a schematic diagram of the process flow for limiting the adjustment of the image signal processor and / or transmitter based on the desired output;

[0023] Figure 12 It is a schematic diagram of a process for increasing the dynamic range of an image by cyclically passing the sensor through a first emission intensity and a second emission intensity and combining the data from the first emission intensity and the second emission intensity;

[0024] Figure 13 It is a schematic diagram of the process of increasing the dynamic range of an image by cyclically passing the sensor through multiple light intensities and combining the data from each of the multiple light intensities;

[0025] Figure 14 It is a schematic diagram of the process flow for performing correction and adjustment on digital image data;

[0026] Figure 15 This is a schematic diagram of system hardware used for writing, storing, and retrieving data from digital video data streams;

[0027] Figure 16 This is a schematic diagram illustrating the method and hardware used in conjunction with a partitioned optical system;

[0028] Figure 17 This is a schematic diagram of a pattern reconstruction process used to generate an RGB image with laser scanning data superimposed on it by pulse partitioning spectrum;

[0029] Figures 18A-18B This is a schematic diagram illustrating a timing example for deploying two different pixel sensitivities in a dual-sensitivity pixel array;

[0030] Figures 19A-19C The use of pulsed and / or synchronized white light emission and the corresponding color sensor is shown;

[0031] Figures 20A-20C A light source with multiple emitters is shown;

[0032] Figure 21 A single optical fiber is shown, which illuminates a scene in a dark environment by outputting via a diffuser at the output point;

[0033] Figure 22 A portion of an electromagnetic spectrum, divided into multiple distinct sub-spectrums that can be emitted by an emitter of a light source, is shown in accordance with the principles and teachings of this disclosure.

[0034] Figure 23 This is a schematic diagram illustrating the timing of emission and readout for generating an image frame that includes multiple exposure frames produced by different partitions of pulsed light;

[0035] Figure 24 It is an imaging system that includes a single cutoff filter for generating fluorescence exposure frames;

[0036] Figure 25 It is an imaging system that includes multiple cutoff filters for generating fluorescence exposure frames;

[0037] Figure 26 It is an imaging system for scanning grid patterns with pulsed lasers, which are used to map the topology of a scene and / or to calculate distances and dimensions within the scene;

[0038] Figure 27A and Figure 27B A specific implementation of a plurality of pixel arrays for generating a three-dimensional image is shown, based on the principles and teachings of this disclosure;

[0039] Figure 28A and Figure 28BPerspective and side views of a specific implementation of an imaging sensor constructed on multiple substrates are shown, respectively. Multiple pixel columns forming a pixel array are located on a first substrate, and multiple circuit columns are located on a second substrate. The figures illustrate the electrical connections and communication between a column of pixels and its associated or corresponding circuit columns.

[0040] Figure 29A and Figure 29B Perspective and side views of a specific implementation of an imaging sensor with multiple pixel arrays for generating three-dimensional images are shown, wherein the multiple pixel arrays and image sensor are constructed on multiple substrates. Detailed Implementation

[0041] This document discloses systems, methods, and apparatuses for digital imaging that are primarily applicable to medical applications such as medical endoscopy. Embodiments of this disclosure are endoscopic systems for laser scanning and color imaging.

[0042] In one embodiment, the system includes a transmitter for emitting electromagnetic radiation pulses, and an image sensor including an array of pixels for sensing reflected electromagnetic radiation. The system includes a controller comprising a processor in electrical communication with the image sensor and the transmitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor. The system is such that at least a portion of the electromagnetic radiation pulses emitted by the transmitter includes a laser scanning pattern.

[0043] The systems, methods, and apparatuses disclosed herein can generate RGB images and also generate fluorescent laser scans and / or tool tracking data. The laser scan data can be evaluated to generate a three-dimensional panoramic view of a scene and calculate distances between objects within the scene. In one embodiment, a medical endoscopic imaging system generates laser scan data for mapping topology and / or calculating the dimensions of objects within a body cavity. The laser scan data can be overlaid on an RGB video stream and used in real time by a physician or computer program to calculate distances between objects within a body cavity. Real-time laser scan data can be used as a non-destructive means of mapping and measuring body cavities. Laser scan data can assist physicians during exploratory or surgical procedures. Additionally, laser scan data can be provided to a robotic surgical system, enabling the robotic system to precisely perform surgical or other medical procedures.

[0044] In some cases, it is desirable to generate endoscopic images with multiple data types or multiple images superimposed on each other. For example, it may be desirable to generate a color (“RGB”) image that also includes laser scan data superimposed on the RGB image. This type of superimposed image allows a physician or computer program to identify dimensions within a body cavity based on the laser scan data. Historically, this would have required the use of multiple sensor systems, including an image sensor for color imaging and one or more additional image sensors and / or transmitters for generating the laser scan data. These multiple image sensors consumed excessive physical space and could not be located at the distal end of the endoscope.

[0045] Conventional endoscopes are designed such that the image sensor is placed at the proximal end of the device within the handheld unit. This configuration requires incident light to travel the length of the endoscope through precisely coupled optics. Precise alignment of these optics can be difficult during normal use, leading to image distortion or loss. The embodiments disclosed herein place the image sensor within the distal end of the endoscope itself. This offers greater optical simplicity compared to specific implementations known in the art. However, an acceptable solution to this method is important and introduces its own set of engineering challenges, particularly the need to fit the image sensor into a highly constrained area. This document discloses systems, methods, and apparatuses for digital imaging in low-light environments, employing image sensors with minimal area and configurable for both laser scanning and color imaging.

[0046] Laser scanning imaging

[0047] In one embodiment, the systems, methods, and apparatus disclosed herein provide means for generating laser scanning data using an endoscopic imaging system. The laser scanning data can be used to determine precise measurements and topographic contours of a scene. In one specific embodiment, the laser scanning data is used to determine precise measurements between, for example, structures or organs within a body cavity, devices or tools within a body cavity, and / or critical structures within a body cavity.

[0048] Laser scanning typically involves the controlled deflection of a laser beam. In the field of 3D object scanning, laser scanning combines the controlled deflection of the laser beam with a laser rangefinder. By measuring distances in each direction, the laser rangefinder can rapidly capture the surface shape of objects, tools, and panoramas. The construction of a full 3D topology can include combining multiple surface models obtained from different perspectives. Various measurement systems and methods exist in this field for applications in archaeology, geography, atmospheric physics, autonomous vehicles, and more. One such system includes Light Detection and Ranging (LIDAR), a 3D laser scanning system. LIDAR has been used in navigation systems such as aircraft or satellites to determine the position and orientation of sensors combined with other systems and sensors. LIDAR uses active sensors to illuminate the object and detects the energy reflected from the object and back to the sensor.

[0049] Laser tracking, or tool tracking using lasers, measures objects by determining the position of an optical target relative to them. Laser trackers can be accurate to the order of 0.025 mm over distances of several meters. In one embodiment, an endoscopic imaging system pulses light in conjunction with a laser tracking system to enable the tracking and measurement of a position or tool within a scene. In such embodiments, the endoscopic imaging system can pulse laser tracking patterns onto tools, objects, or other structures within a scene imaged by the endoscopic imaging system. A target can be placed on a tool, object, or other structure within the scene. Measurements between the endoscopic imaging system and the target can be triggered and acquired at selected points, allowing the endoscopic imaging system to track the position of the target (and the tools, objects, or other structures attached to the target).

[0050] Pulsed imaging

[0051] Some specific embodiments of this disclosure include aspects of a sensor and system combination design capable of generating high-resolution images with a reduced number of pixels in highly constrained lighting environments. This is achieved by pulsed monochromatic wavelengths frame by frame and switching or alternating between single different color wavelengths each frame using a controlled light source combined with a high frame capture rate and a specially designed corresponding monochromatic (“color-indeterminate”) sensor. The pixels are color-indeterminate, such that each pixel generates data for each electromagnetic radiation pulse, including pulses of red, green, and blue visible wavelengths in addition to the relaxation wavelengths, hyperspectral wavelengths, and reflected laser scanning patterns of one or more fluorescent reagents. The system disclosed herein can generate RGB images overlaid with additional imaging data. In one embodiment, the system pulses electromagnetic radiation of certain wavelengths to excite fluorescent reagents, thereby generating fluorescent imaging data that can be used to identify key tissues and structures. In one embodiment, the system pulses hyperspectral wavelengths of electromagnetic radiation to generate hyperspectral imaging data that can be used to identify key tissues and structures. In one embodiment, the system pulses laser scanning patterns to generate a three-dimensional topology of a scene and / or to measure the distance and size of objects within the scene. In one embodiment, a combination of fluorescence, hyperspectral, and / or laser scanning emission is deployed to generate a combined image frame that includes RGB image data and also includes hyperspectral, fluorescence, and / or laser scanning data.

[0052] For the purpose of facilitating an understanding of the principles of this disclosure, reference will now be made to embodiments shown in the accompanying drawings, and these embodiments will be described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any changes and further modifications to the features of the invention shown herein, as well as any additional applications of the principles of this disclosure as shown herein (which will generally occur to those skilled in the art and those familiar with the contents of this disclosure), will be considered within the scope of the disclosure protected by the claims.

[0053] Before disclosing and describing the structures, systems, and methods for generating images in low-light environments, it should be understood that this disclosure is not limited to the specific structures, configurations, process steps, and materials disclosed herein, as such structures, configurations, process steps, and materials can vary to some extent. Furthermore, it should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the invention will be defined only by the appended claims and their equivalents.

[0054] In describing and claiming the subject matter of this disclosure, the following terms will be used in accordance with the following definitions.

[0055] It should be noted that the singular forms “a,” “an,” and “the” used in this specification and the appended claims include multiple referents unless the context clearly indicates otherwise.

[0056] As used herein, the terms “comprising,” “including,” “characterized by,” and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unmentioned elements or method steps.

[0057] As used herein, the phrase “consisting of” and its grammatical equivalents exclude any element or step not included in the claims.

[0058] As used herein, the phrase “consistent with…” and its grammatical equivalents limit the scope of the claims to the specified materials or steps and to materials or steps that do not substantially affect one or more of the essential and novel features of the claimed disclosure.

[0059] As used in this article, the term "proximal" broadly refers to the concept of the part closer to the starting point.

[0060] As used in this article, the term "distal" generally refers to the opposite of the proximal, and therefore, depending on the context, it refers to the part that is farther from the starting point or the farthest part.

[0061] As used herein, color sensors or multispectral sensors are those known to have a color filter array (CFA) on which incident electromagnetic radiation is filtered to its single component. In the visible range of the electromagnetic spectrum, such CFAs can be based on a Bayer template or a modified form thereof to separate the green, red, and blue spectral components of light.

[0062] As used in this article, a monochrome sensor refers to an imaging sensor without filtering capabilities. Because pixels are color-invariant, their effective spatial resolution is significantly higher than that of pixel color equivalents in traditional single-sensor cameras (which typically employ Bayer pattern filtering). Monochrome sensors also offer higher quantum efficiency because fewer incident photons are wasted between individual pixels.

[0063] As used herein, a transmitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of a transmitter can be configured to emit pulses and have very specific frequencies or frequency ranges from the entire electromagnetic spectrum. Pulses may include wavelengths in both the visible and invisible ranges. The transmitter may cycle on and off to generate pulses, or may generate pulses with a shutter mechanism. The transmitter may have a variable power output level, or may be controlled by auxiliary devices such as apertures or filters. The transmitter may emit broad-spectrum or full-spectrum electromagnetic radiation that can generate pulses through color filtering or light blocking. The transmitter may include multiple electromagnetic sources, acting individually or in concert.

[0064] Now refer to the attached diagram, Figure 1 A schematic diagram of a system 100 for sequential pulse imaging in a low-light environment is shown. The system 100 can be deployed to generate RGB images, wherein laser scan data is overlaid on the RGB image. The system 100 includes a transmitter 102 and a pixel array 122. The transmitter 102 pulses a partition of electromagnetic radiation in the low-light environment 112, and the pixel array 122 senses instances of reflected electromagnetic radiation. The transmitter 102 and the pixel array 122 operate sequentially such that one or more pulses of the partition of electromagnetic radiation generate image data sensed by the pixel array 122.

[0065] It should be noted that, as used herein, the term "light" refers to both a particle and a wavelength, and is intended to represent electromagnetic radiation detectable by the pixel array 122, and may include wavelengths from both the visible and invisible spectra of electromagnetic radiation. The term "partition" as used herein refers to a predetermined wavelength range of the electromagnetic spectrum that is smaller than the entire spectrum, or in other words, wavelengths that constitute a portion of the electromagnetic spectrum. As used herein, an emitter is a controllable light source with respect to a portion of the emitted electromagnetic spectrum, or a light source with physical properties of its components, emission intensity, or emission duration, or all of the above. An emitter may emit light in any dithered, diffuse, or collimated emission, and may be controlled digitally or through analog methods or systems. As used herein, an electromagnetic emitter is a source of electromagnetic energy bursts, and includes light sources such as lasers, LEDs, incandescent light, or any digitally controllable light source.

[0066] The pixel array 122 of the image sensor can be electronically paired with the transmitter 102, such that the transmitter 102 and the pixel array 122 are synchronized during operation for both receiving emissions and adjustments performed within the system. The transmitter 102 can be tuned to emit electromagnetic radiation in the form of a laser, which can be pulsed to illuminate a darkened environment 112. The transmitter 102 can correspond to interval pulses for the operation and function of the pixel array 122. The transmitter 102 can pulse light in multiple electromagnetic zones, causing the pixel array to receive electromagnetic energy and generate a dataset corresponding in time to each specific electromagnetic zone. For example, Figure 1 A specific implementation is illustrated, in which transmitter 102 emits electromagnetic radiation in four distinct zones, including a red wavelength of 104, a green wavelength of 106, a blue wavelength of 108, and a laser scanning 110 pulse scheme. The laser scanning 110 pulse scheme may include a grid pattern of topology 120 for identifying a scene in a dark environment 112, and further for measuring dimensions and distances within the scene.

[0067] The dark environment 112 includes structures, organizations, and other elements that reflect a combination of red 114, green 116, and / or blue 118 light. The red 114 organization is sensitive to pulsed red 104 light and will be perceived by the human eye as having a red 114 hue. The scene within the dark environment 112 includes a topology 120 having one or more objects that can be scanned to generate a 3D model of the scene.

[0068] Emitter 102 may be a laser emitter capable of emitting pulsed red light 104 to generate sensing red data 105, thereby identifying red features 114 within the dark environment 112. Emitter 102 may also emit pulsed green light 106 to generate sensing green data 107, thereby identifying green features 116 within the dark environment. Emitter 102 may also emit pulsed blue light 108 to generate sensing blue data 109, thereby identifying blue features 118 within the dark environment. Emitter 102 may also emit a laser scanning pulse pattern 110 for mapping the topology 120 of the scene within the dark environment 112. Emitter 102 may emit the pulsed red 104, pulsed green 106, pulsed blue 108, and pulsed laser scanning 110 pulse patterns in any desired order.

[0069] Pixel array 122 senses reflected electromagnetic radiation. Each of the data sensed—red 105, green 107, blue 109, and laser scan 111—can be referred to as an "exposure frame." A specific color or wavelength partition is assigned to each exposure frame, where the assignment is based on the timing of pulsed color or wavelength partitions from transmitter 102. The combination of exposure frames and assigned specific color or wavelength partitions can be referred to as a dataset. Even if pixel 122 is not a dedicated color, colors can be assigned to any given dataset based on prior information about the transmitter.

[0070] For example, during operation, after the pulsed red 104 light is pulsed in the dark environment 112, the pixel array 122 senses the reflected electromagnetic radiation. The reflected electromagnetic radiation generates an exposure frame, and this exposure frame is classified as sensing red 105 data because it corresponds temporally to the pulsed red 104 light. The exposure frame, along with its indication of temporal correspondence to the pulsed red 104 light, constitutes a “dataset.” This process is repeated for each partition of electromagnetic radiation emitted by the transmitter 102. The data generated by the pixel array 122 includes sensing red 105 exposure frames that identify the red 114 component in the dark environment and correspond temporally to the pulsed red 104 light. The data also includes sensing green 107 exposure frames that identify the green 116 component in the dark environment and correspond temporally to the pulsed green 106 light. The data also includes sensing blue 109 exposure frames that identify the blue 118 component in the dark environment and correspond temporally to the pulsed blue 108 light. The data also includes sensing laser scan 111 exposure frames, which identify topology 120 and correspond in time to the laser scan 110 pulse scheme.

[0071] In one implementation, three datasets representing red, green, and blue electromagnetic pulses are combined to form a single image frame. Thus, information from red, green, and blue exposure frames is combined to form a single RGB image frame. One or more additional datasets representing other wavelength partitions may be overlaid on the single RGB image frame. These additional datasets may represent, for example, laser scanning data, fluorescence imaging data, and / or hyperspectral imaging data.

[0072] It should be understood that, without departing from the scope of this disclosure, this disclosure is not limited to any particular color combination or any particular electromagnetic partition, and any color combination or any electromagnetic partition can be used in place of RED, GREEN, and BLUE, such as cyan, magenta, and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible and invisible wavelengths. In the figure, the dark environment 112 to be imaged includes a red portion 114, a green portion 116, and a blue portion 118, and also includes a topology 120 that can be sensed and mapped into the 3D rendering. As shown, the reflected light from the electromagnetic pulse contains only data of the portion of the object having a specific color corresponding to the pulse's color partition. These individual color (or color interval) datasets can then be used to reconstruct the image by combining the datasets at 126. Information in each of the plurality of exposure frames (i.e., the plurality of datasets) can be processed by a controller 124, a control unit, a camera control unit, an image sensor, an image signal processing pipeline, or some other combination of computing resources that can be configured to process the plurality of exposure frames and combine the datasets at 126. The dataset can be combined to generate a single image frame either within the endoscope unit itself or off-site by some other processing resources.

[0073] Figure 2 This is a system 200 for providing illumination to light-deficient environments, such as those used for endoscopic imaging. System 200 can be used in conjunction with any of the systems, methods, or apparatuses disclosed herein. System 200 includes a transmitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, an internal cavity waveguide 210, an internal cavity 212, and an image sensor 214 with accompanying optical components, such as lenses. The transmitter 202 (generally referred to as a “light source”) generates light that passes through the jumper waveguide 206 and the internal cavity waveguide 210 to illuminate the scene at the distal end of the internal cavity 212. The transmitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, hyperspectral, fluorescence excitation, laser scanning pulse schemes, or other wavelengths. The internal cavity 212 can be inserted into a patient for imaging, such as during surgery or examination. The output light is shown as dashed line 216. The image sensor 214 can be used to capture the scene illuminated by the light and display the scene to a physician or other medical personnel. Controller 204 can provide control signals to transmitter 202 to control when illumination is provided to a scene. In one embodiment, transmitter 202 and controller 204 are located within a camera control unit (CCU) or external console to which the endoscope is attached. If image sensor 214 includes a CMOS sensor, light can be periodically provided to the scene in a series of illumination pulses between readout cycles of image sensor 214 during a so-called blanking period. Therefore, light can be pulsed in a controlled manner to avoid overlaying onto the readout cycles of image pixels in the pixel array of image sensor 214.

[0074] In one embodiment, the cavity waveguide 210 includes one or more optical fibers. These optical fibers may be made of low-cost materials such as plastic to allow for the handling of the cavity waveguide 210 and / or other parts of the endoscope. In one embodiment, the cavity waveguide 210 is a single glass fiber with a diameter of 500 micrometers. A jumper waveguide 206 may be permanently attached to the transmitter 202. For example, the jumper waveguide 206 may receive light from a transmitter within the transmitter 202 and provide light to the cavity waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 206 includes one or more glass optical fibers. The jumper waveguide may include any other type of waveguide for guiding light to the cavity waveguide 210. The connector 208 may selectively couple the jumper waveguide 206 to the cavity waveguide 210 and allow light within the jumper waveguide 206 to pass through the cavity waveguide 210. In one implementation, the cavity waveguide 210 is directly coupled to the light source without any intervening jumper waveguide 206.

[0075] Image sensor 214 includes a pixel array. In one embodiment, image sensor 214 includes two or more pixel arrays for generating a three-dimensional image. Image sensor 214 may constitute two additional image sensors, each with an independent pixel array and capable of operating independently of each other. The pixel array of image sensor 214 includes effective pixels and optical black (“OB”) pixels or light-blind pixels. Effective pixels may be transparent “color-indeterminate” pixels capable of sensing imaging data of electromagnetic radiation at any wavelength. Optical black pixels are read during the blanking period of the pixel array when the pixel array is “reset” or calibrated. In one embodiment, light pulses during the blanking period of the pixel array when reading optical black pixels. After the optical black pixels have been read, effective pixels are read during the readout period of the pixel array. Effective pixels may be charged by electromagnetic radiation pulsed during the blanking period, such that the effective pixels are ready to be read by the image sensor during the readout period of the pixel array.

[0076] Figure 2A This is a schematic diagram of complementary system hardware such as a dedicated or general-purpose computer. Embodiments within the scope of this disclosure may also include physical and other non-transitory computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible through a general-purpose or dedicated computer system. A computer-readable medium storing computer-executable instructions is a computer storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, specific embodiments of this disclosure may include at least two distinctly different types of computer-readable media: computer storage media (devices) and transmission media.

[0077] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to store program code tools in the form of desired computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0078] A “network” refers to one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. In one implementation, sensors and camera control units may be networked to communicate with each other and with other components connected through the network to which they are connected. When information is transmitted or provided to a computer via a network or other communication connection (hard-wired, wireless, or a combination of hard-wired and wireless), the computer reasonably considers that connection as a transmission medium. The transmission medium may include networks and / or data links that can be used to carry program code tools in the form of desired computer-executable instructions or data structures and are accessible via general-purpose or special-purpose computers. The above combinations should also be covered within the scope of computer-readable media.

[0079] Furthermore, upon arrival at various computer system components, program code tools in the form of computer-executable instructions or data structures can be automatically transferred by a transmission medium to computer storage media (devices) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or the computer system's non-volatile computer storage media (devices). RAM may also include solid-state drives (SSDs) or PCIx-based real-time memory tiered storage devices, such as FusionIO. Therefore, it should be understood that computer storage media (devices) may be included in computer system components that also (or even primarily) utilize transmission media.

[0080] Computer-executable instructions include, for example, instructions and data that, when executed by one or more processors, cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform certain functions or groups of functions. Computer-executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter of the invention has been set forth in terms of language with respect to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or steps described above. Rather, the features and steps described above are disclosed as examples of implementing the claims.

[0081] Those skilled in the art will understand that this disclosure can be implemented in a network computing environment with various types of computer system configurations, including personal computers, desktop computers, laptop computers, information processors, control units, camera control units, handheld devices, handheld components, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, various storage devices, etc. It should be noted that any of the aforementioned computing devices can be provided by or located within an entity. This disclosure can also be implemented in a distributed system environment, wherein local and remote computer systems are connected via a network (through a hard-wired data link, a wireless data link, or a combination of hard-wired and wireless data links), and both can perform tasks. In a distributed system environment, program modules can reside in both local and remote memory storage devices.

[0082] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to execute one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. Those skilled in the art will understand that components may have different names. This document is not intended to distinguish between components that are different in name rather than function.

[0083] Figure 2A This is a block diagram illustrating an exemplary computing device 250. The computing device 250 can be used to execute various programs, such as those discussed herein. The computing device 250 can be used as a server, client, or any other computing entity. The computing device 250 can perform various monitoring functions discussed herein and can execute one or more applications, such as those described herein. The computing device 250 can be any of a variety of computing devices, such as a desktop computer, laptop computer, server computer, handheld computer, camera control unit, tablet computer, etc.

[0084] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all of which are coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory devices 254 and / or mass storage devices 258. The processor 252 may also include various types of computer-readable media, such as cache memory.

[0085] The memory device 254 includes various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 264) and / or non-volatile memory (e.g., read-only memory (ROM) 266). The memory device 254 may also include rewritable ROM, such as flash memory.

[0086] Mass storage devices 258 include various computer-readable media, such as magnetic tape, magnetic disks, optical disks, solid-state storage (e.g., flash memory), etc. Figure 2 As shown, a specific mass storage device is hard disk drive 274. Various drives may also be included in mass storage device 258 to enable reading and / or writing from various computer-readable media. Mass storage device 258 includes removable media 276 and / or non-removable media.

[0087] I / O device 260 includes various means that allow input to or retrieval of data and / or other information to or from computing device 250. Exemplary I / O device 260 includes digital imaging device, electromagnetic sensor and transmitter, cursor control device, keyboard, keypad, microphone, monitor or other display device, speaker, printer, network interface card, modem, lens, CCD or other image capture device, etc.

[0088] Display device 280 includes any type of device capable of displaying information to one or more users of computing device 250. Examples of display device 280 include monitors, display terminals, video projection devices, etc.

[0089] Interface 256 includes various interfaces that allow computing device 250 to interact with other systems, devices, or computing environments. Exemplary interface 256 may include any number of different network interfaces 270, such as interfaces for connecting to a local area network (LAN), wide area network (WAN), wireless network, and the Internet. Other interfaces include user interface 268 and peripheral device interface 272. Interface 256 may also include one or more user interface elements 268. Interface 256 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touchpad, etc.), keyboards, etc.

[0090] Bus 262 allows processor 252, memory device 254, interface 256, mass storage device 258, and I / O device 260 to communicate with each other and with other devices or components coupled to bus 262. Bus 262 represents one or more of several types of bus architectures (such as system bus, PCI bus, IEEE 1394 bus, USB bus, etc.).

[0091] For illustrative purposes, the programs and other executable program components shown herein are discrete blocks; however, it should be understood that such programs and components may reside at various times in different storage devices of computing device 250 and be executed by processor 252. Alternatively, the systems and programs described herein may be implemented in hardware, or in a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays may be programmed to execute one or more systems and programs described herein.

[0092] Figure 3A The diagram illustrates the sensor's operational cycle during rolling readout mode or during sensor readout 300. Frame readout may begin at and be represented by a vertical line 310. The readout cycle is represented by a diagonal or slanted line 302. The sensor can be read out line by line, with the top of the downward-sloping edge being the sensor top row 312 and the bottom of the downward-sloping edge being the sensor bottom row 314. The time between the last row readout and the next readout cycle may be referred to as the blanking cycle 316. It should be noted that some of the sensor pixel rows may be covered with light-shielding material (e.g., a metallic coating or any other generally black layer of another material type). These covered pixel rows may be referred to as optical black rows 318 and 320. Optical black rows 318 and 320 can be used as input to correction algorithms. Figure 3A As shown, these optical black rows 318 and 320 can be located on top of the pixel array, at the bottom of the pixel array, or at both the top and bottom of the pixel array.

[0093] Figure 3BA method is shown for controlling the amount of electromagnetic radiation (e.g., light) exposed to pixels so that it is integrated or accumulated by the pixels. It should be understood that photons are the fundamental particles of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted into electrical charge or current. An electronic shutter or rolling shutter (shown as dashed 322) can be used to begin the integration time by resetting the pixels. Light will then be integrated until the next readout stage. The position of the electronic shutter 322 can be moved between two readout cycles 302 to control the pixel saturation of a given amount of light. It should be noted that this technique allows for a constant integration time between two different rows, but introduces a delay when moving from the top row to the bottom row.

[0094] Figure 3C The diagram shows the case where the electronic shutter 322 has been removed. In this configuration, the integration of incident light can begin during readout 302 and end at the next readout cycle 302, which also defines the start of the next integration.

[0095] Figure 3D This illustrates a configuration without an electronic shutter 322 but with controlled and pulsed light 330 during the blanking period 316. This ensures that all lines see the same light emitted from the same light pulse 330. In other words, each line will begin its integration in a dark environment, which may be located after the optically black line 320 of the readout frame (m) to obtain the maximum light pulse width, and will then receive the light pass and end its integration in a dark environment, which may be located before the optically black line 318 of the next subsequent readout frame (m+1) to obtain the maximum light pulse width. For example... Figure 3D In this process, the image generated by the light pulse will only be available during the readout of frame (m+1) without interfering with frames (m) and (m+2). It should be noted that the condition for the light pulse to be read out only in one frame and not interfere with adjacent frames is that a given light pulse is fired during the blanking period 316. Because the optical black lines 318 and 320 are not sensitive to light, the optical black line following time 320 of frame (m) and the optical black line preceding time 318 of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 330.

[0096] like Figure 3A As shown, the sensor can cycle multiple times to receive data for each pulse color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum). Each cycle can be timed. In one embodiment, the cycle can be timed to operate at intervals of 16.67 ms. In another embodiment, the cycle can be timed to operate at intervals of 8.3 ms. It should be understood that other timing intervals are contemplated and are intended to fall within the scope of this disclosure.

[0097] Figure 4AThe operation of an embodiment of the electromagnetic transmitter is illustrated graphically. The transmitter can be timed to correspond to the sensor's operating cycle, such that electromagnetic radiation is emitted during and / or a portion of the sensor's operating cycle. Figure 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In one embodiment, the transmitter may pulse during the readout portion 302 of a sensor operating cycle. In one embodiment, the transmitter may pulse during the blanking portion 316 of a sensor operating cycle. In one embodiment, the transmitter may pulse for a duration that spans two or more portions of a sensor operating cycle. In one embodiment, the transmitter may begin pulsed during the blanking portion 316 or during the optical black portion 320 of the readout portion 302, and end pulsed during the readout portion 302 or during the optical black portion 318 of the readout portion 302 in the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of this disclosure, provided that the transmitter's pulses correspond to the sensor's cycles.

[0098] Figure 4B The control of exposure is graphically represented by varying the duration and magnitude of the emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, and pulse 3 at 416). A transmitter with a fixed output magnitude can be combined with the above... Figure 3D and Figure 4A During any given cycle, pulses are applied at regular intervals to provide the required electromagnetic energy to the pixel array. A transmitter with a fixed output value can pulse over longer time intervals, thus providing more electromagnetic energy to the pixels, or it can pulse over shorter time intervals, thus providing less electromagnetic energy. Whether longer or shorter time intervals are required depends on the operating conditions.

[0099] Compared to adjusting the time interval of a fixed output pulse value from the transmitter, increasing the pulse value itself can provide more electromagnetic energy to the pixel. Similarly, decreasing the pulse value provides less electromagnetic energy to the pixel. It should be noted that, if needed, the system implementation can have the ability to adjust both the pulse value and duration simultaneously. Furthermore, the sensor's sensitivity and duration can be adjusted to meet the requirements of optimal image quality. Figure 4BThe diagram illustrates variations in the magnitude and duration of pulses. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. Additionally, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416, such that the electromagnetic energy provided by this pulse is represented by the area under the pulse shown in the diagram. In the illustration, pulse 2 at 414 has a relatively lower magnitude or intensity and a longer duration compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration compared to pulse 1 at 412 and pulse 2 at 414.

[0100] Figure 5 For the combination of the principles and teachings of this disclosure Figures 3A-3D and Figure 4A The operating cycle, electromagnetic emitter, and emitted electromagnetic pulses are graphically represented to indicate an embodiment of the imaging system disclosed herein during operation. As can be seen in the figures, the electromagnetic emitter pulses primarily during the sensor's blanking cycle 316, causing the pixels to be charged and ready for reading during the readout portion 302 of the sensor cycle. The dotted-line portion of the pulse (from...) Figure 4A This demonstrates the possibility or capability of emitting electromagnetic energy during the optical black portions 320 and 318 of the readout cycle (sensor cycle) 300 when additional time is needed or desired for pulsed electromagnetic energy.

[0101] An exposure frame includes data read from the pixel array of the image sensor during readout cycle 302. The exposure frame can be combined with an indication of the type of pulse emitted by the transmitter prior to readout cycle 302. This combination of exposure frames and pulse type indication can be referred to as a dataset. Multiple exposure frames can be combined to generate a black-and-white or RGB color image. Additionally, hyperspectral, fluorescence, and / or laser mapping imaging data can be overlaid on the black-and-white or RGB image.

[0102] In one embodiment, the exposure frame is data sensed by the pixel array during readout period 302, which occurs after blanking period 316. Electromagnetic radiation is emitted during blanking period 316. In one embodiment, a portion of the electromagnetic radiation emission overlaps with readout period 316. Blanking period 316 occurs while the optical black pixels of the pixel array are being read, and readout period 302 occurs while the effective pixels of the pixel array are being read. Blanking period 316 may overlap with readout period 302.

[0103] Figure 6A and Figure 6BThe process for recording image frames is illustrated. Multiple image frames can be chained together to generate a video stream. A single image frame can include data from multiple exposure frames, where the exposure frames are data sensed by the pixel array after electromagnetic radiation is emitted. Figure 6A The diagram illustrates a typical process that is usually implemented using a color image sensor with a color filter array (CFA) to filter out certain wavelengths of light for each pixel. Figure 6B This is the process disclosed in this paper, and it can be achieved using a monochrome "color-indeterminate" image sensor that receives electromagnetic radiation of all wavelengths.

[0104] Figure 6A The process shown occurs from time t(0) to time t(1). The process begins with the emission of white light at 602 and the sensing of white light at 604. At 606, an image is processed and displayed based on the sensing at 604.

[0105] Figure 6B The process shown occurs from time t(0) to time t(1). The process begins with the emission of green light 612, and after the emission of green light 612, reflected electromagnetic radiation 614 is sensed. The process continues with the emission of red light 616, and after the emission of red light 616, reflected electromagnetic radiation 618 is sensed. The process continues with the emission of blue light 620, and after the emission of blue light 620, reflected electromagnetic radiation 622 is sensed. The process continues with one or more emission schemes of laser scanning 624 pulses, and after each of these one or more emission schemes of laser scanning 624 pulses, reflected electromagnetic energy 626 is sensed. At 628, an image is processed and displayed based on each of the sensed reflected electromagnetic energy instances 614, 618, 622, and 626.

[0106] Figure 6B The process shown provides a high-resolution image and provides apparatus for generating an RGB image that also includes laser scan data. When using partitioned spectroscopy (such as...) Figure 6B As shown (in the diagram), this allows the sensor to be sensitive to electromagnetic energy across all wavelengths. Figure 6B In the process illustrated, the monochromatic pixel array is instructed to sense electromagnetic energy from predetermined partitions of the full electromagnetic energy spectrum in each cycle. Therefore, to form an image, the sensor only needs to cycle through multiple different partitions within the full spectrum. The final image is assembled based on these multiple cycles. Because the image from each color partition frame cycle has higher resolution (compared to a CFA pixel array), the resulting image when the partitioned light frames are combined also has higher resolution. In other words, because each pixel within the array (rather than at most every other pixel in a CFA sensor) senses the amplitude of the energy for a given pulse and a given scene only at intervals, a higher resolution image is produced for each scene.

[0107] As in Figure 6A and Figure 6B The implementation shown can be graphically illustrated between times t(0) and t(1), for Figure 6B Sensors in the partitioned spectral system Figure 6A Each cycle in the full-spectrum system is performed at least four times. In one implementation, the display device (LCD panel) operates at a rate of 50 to 60 frames per second. In such implementations, Figure 6B The localized optical system can operate at 200 to 240 frames per second to maintain the continuity and smoothness of the displayed video. In other implementations, different capture and display frame rates may exist. Furthermore, the average capture rate can be any multiple of the display rate.

[0108] In one implementation, it may be desirable that not all zones are represented equally within the system frame rate. In other words, not all light sources must pulse with the same regularity in order to emphasize and de-emphasize various aspects of the recorded scene as needed by the user. It should also be understood that invisible and visible zones of the electromagnetic spectrum can be pulsed together within the system, with their corresponding data values ​​stitched into the video output for display to the user.

[0109] The implementation scheme may include the following pulse cycle pattern:

[0110] i. Green pulse;

[0111] ii. Red pulse;

[0112] iii. Blue pulse;

[0113] iv. Green pulse;

[0114] v. Red pulse;

[0115] vi. Blue pulse;

[0116] vii. Laser scanning pulse scheme;

[0117] viii. (repeated)

[0118] As can be seen in this example, the laser scan partition can be pulsed at a rate different from the other partition pulses. This emphasizes a particular aspect of the scene, where the laser scan data overlaps only with other data in the video output to make the desired emphasis. It should be noted that adding a laser scan partition above the red, green, and blue partitions does not necessarily require the serialized system to operate at four times the rate of a full-spectrum non-serialized system, as each partition does not need to be represented equally in the pulse pattern. As seen in this implementation, adding fewer partition pulses represented in the pulse pattern (the laser scan in the example above) will result in an increase of less than 20% in the sensor's loop rate to accommodate irregular partition sampling.

[0119] In various implementations, the pulse cyclic pattern may also include any of the following wavelengths in any suitable order. Such wavelengths are particularly suitable for exciting fluorescent reagents to generate fluorescence imaging data by sensing the relaxation emission of the fluorescent reagent based on its relaxation emission:

[0120] i.770±20nm;

[0121] ii. 770±10nm;

[0122] iii. 770±5nm;

[0123] iv. 790±20nm;

[0124] v.790±10nm;

[0125] vi. 790±5nm;

[0126] vii.795±20nm;

[0127] viii. 795±10nm;

[0128] ix.795±5nm;

[0129] x.815±20nm;

[0130] xi.815±10nm;

[0131] xii.815±5nm;

[0132] xiii. 770nm to 790nm; and / or

[0133] xiv. 795nm to 815nm.

[0134] Partition loops can be divided to adapt to or approximate various imaging and video standards. In one implementation, a partition loop may include, as follows: Figures 7A-7DThe pulses of electromagnetic energy in the red, green, and blue spectra are best shown in the image. Figure 7A Different light intensities have been achieved by modulating the width or duration of the light pulse within the working range, indicated by the vertical gray dashed line. Figure 7B In this process, different light intensities are achieved by modulating the power of the optical power or the power of the electromagnetic transmitter (which can be a laser or an LED transmitter), while keeping the pulse width or duration constant. Figure 7C This illustrates a scenario where both optical power and pulse width are modulated for greater flexibility. Partition cycling can utilize cyan, magenta, yellow (CMY), infrared, ultraviolet, hyperspectral, and fluorescence, employing invisible pulse sources mixed with visible pulse sources, as well as any other color space required to generate the image or approximating currently known or yet-to-be-developed desired video standards. It should also be understood that the system is capable of switching between operating color spaces to provide the desired image output quality.

[0135] Using the color space green-blue-green-red (e.g.) Figure 7D In the implementation shown, it may be desirable to pulse the luminance component more frequently than the chromaticity component, because users are generally more sensitive to differences in light intensity than to differences in light color. Examples of such implementations include... Figure 7D The monochrome sensor shown utilizes this principle. Figure 7D In this configuration, green, which contains the most luminance information, can pulse more frequently or have greater intensity in the (GBGRGBGR…) scheme to obtain luminance data. This configuration will produce a video stream with noticeably more detail, without generating and transmitting imperceptible data.

[0136] In one implementation, pulses that replicate weaker regions can be used to generate an output already adjusted for the weaker pulses. For example, blue lasers are considered less sensitive relative to silicon-based pixels and harder to generate than red or green light; therefore, they can be pulsed more frequently during frame cycles to compensate for the weakness of the light. These additional pulses can be performed continuously over time or by pulsed simultaneously using multiple lasers to produce the desired compensation effect. It should be noted that by pulses during the blanking period (the time during which the sensor does not read out the pixel array), the sensor is insensitive to differences / mismatches between lasers of the same type and simply focuses the light for the desired output. In another implementation, the maximum range of light pulses may vary from frame to frame. This is in... Figure 7E As shown, the light pulses differ from frame to frame. The sensor can be configured to be programmed with different blanking periods in repeating patterns of two, three, four, or n frames. Figure 7EThe diagram illustrates four distinct optical pulses, with pulse 1 repeating, for example, after pulse 4, and a pattern of four frames with different blanking periods. This technique can be used to place the most powerful partition on the smallest blanking period, thus allowing the weakest partition to have a wider pulse on a subsequent frame without increasing readout speed. The reconstructed frame can still have a regular frame-to-frame pattern because it consists of many pulse frames.

[0137] Figure 8 A process flow 800 is illustrated, in which the sensor and / or transmitter are adjusted to compensate for differences in energy values ​​in the pulsed spectral partitions. In process flow 800, data is obtained from the histogram of the previous frame and analyzed at 802. The sensor is adjusted at 804, and the transmitter is adjusted at 806. At 808, an image is determined based on the adjusted sample time from the sensor, and / or at 810, an image is determined based on the adjusted (increased or decreased) emitted light.

[0138] Figure 9 This is process flow 900 for adjusting the sensor and recording frames based on readings from the adjusted sensor. In process flow 900, a histogram of the previous frame is obtained at 902 and the sensor is adjusted based on sensitivity at 904. At 906, a frame is recorded based on readings from the adjusted sensor. In one example, process flows 800 and 900 are implemented because red light is more easily detected by the sensor within the system than blue light. In this example, this is because the blue region has lower quantum efficiency relative to silicon (in...). Figure 9 (As best shown in the image), so the sensor is adjusted to be less sensitive during the red zone cycle and more sensitive during the blue zone cycle. Similarly, the transmitter can be adjusted to provide regulated zones (e.g., higher or lower intensity and duration).

[0139] Figures 10-11 This is a schematic diagram of a method for performing color correction on digital imaging. Common in digital imaging is the manipulation of values ​​within image data to correct the output, thereby meeting user expectations or highlighting certain aspects of the imaged object. In systems where light is user-controlled, it is advantageous to provide light emission that is known to the user and may be only a portion of the electromagnetic spectrum or multiple portions of the entire electromagnetic spectrum. Image correction is important for meeting user expectations and identifying faults within the system. One calibration method employs a table of expected values ​​for a given imaging condition that can be compared with data from an image sensor.

[0140] Figure 10This is a schematic diagram of a method 1000 for performing color correction on digital imaging in a low-light environment. Method 1000 includes sampling a neutral color scene or palette (see 1010) at startup by running a full loop of an electromagnetic spectral partition at 1002. A lookup table 1008 is generated based on the color neutral scene or palette 1010. The lookup table 1008 is used to determine a histogram of the frame loop at 1008. At 1006, the histogram is compared with known or expected values ​​based on the neutral color scene or palette 1010 and also based on the lookup table 1008. The method includes adjusting settings on the image signal processor (ISP) at 1012 and / or adjusting the transmitter at 1014. Adjustment of the transmitter at 1014 may include adjustment of any aspect of the emitted light, such as magnitude, duration (i.e., on-time), or range within the spectral partition.

[0141] It should be noted that because each region of the light spectrum can have different energy values, the sensor and / or light emitter can be adjusted to compensate for these differences in energy values. For example, in one embodiment, because the blue light spectrum has a lower quantum efficiency relative to the red light spectrum than the silicon-based imager, the sensor's responsiveness can subsequently be tuned to respond less during the red cycle and more during the blue cycle. Conversely, because blue light has a lower quantum efficiency than red light, the emitter can emit blue light at an intensity higher than that of red light to produce a correctly exposed image.

[0142] Figure 11 This is a schematic diagram of method 1100 for performing hierarchical adjustments to an image signal processor (ISP) and / or a transmitter to reduce the amount of noise and artifacts in the output image stream or video. Method 1100 includes a full cycle of transmitting and sensing an electromagnetic spectral region at 1102. At 1104, the result of the full cycle of the electromagnetic region is compared with the desired output. Based on this comparison, the image signal processor (ISP) is adjusted at 1106 and / or the transmitter at 1108. The adjustments made to the ISP at 1106 and / or the transmitter at 1108 between frame cycles may be limited at 1110. For example, the transmitter may be adjusted at any time between frames by a portion of its operating range. Similarly, the ISP may be adjusted at any time between frames by a portion of its operating range. In one embodiment, both the transmitter and the ISP may be limited such that only a portion of the respective operating range of the transmitter and sensor may be adjusted together at any time between frames. The results of these hierarchical adjustments are compared at 1112, and the adjustments are completed based on this comparison.

[0143] In an exemplary embodiment, graded adjustment of the ISP and / or transmitter can be performed at approximately 0.1 dB within the operating range of the component to correct exposure of previous frames. 0.1 dB is merely an example, and it should be noted that in other embodiments, the permissible adjustment of the component can be any portion of its respective operating range. Components of the system can be varied by intensity or duration adjustment, typically controlled by the bit depth (resolution) of the component output. Component resolution can typically range from approximately 10 bits to 24 bits, but should not be limited to this range, as it is intended to include resolutions of components yet to be developed beyond those currently available. For example, if, after the first frame, it is determined that the scene is too blue when observed, the transmitter can be adjusted by graded adjustment (such as approximately 0.1 dB) as described above to reduce the pulse magnitude or duration of blue light during the system's blue cycle.

[0144] In this exemplary embodiment, a range exceeding 10% may be required, but the system has limited itself to an operating range of 0.1 dB adjustment per system cycle. Therefore, the blue light can be adjusted again during the next system cycle if needed. The graded adjustment between cycles can have a damping effect on the output image, and when the transmitter and sensor are operated at their operating limits, graded adjustment will reduce noise and artifacts. Any graded amount that determines the operating adjustment range of a component can be used as a limiting factor, or certain embodiments of the system may include components that can be adjusted throughout their entire operating range.

[0145] Additionally, the optical black area of ​​any image sensor can be used to aid in image correction and noise reduction. In one implementation, values ​​read from the optical black area can be compared with those values ​​in the sensor's effective pixel area to establish reference points that will be used for image data processing. Figure 12 This illustrates the types of sensor correction methods that can be employed in color pulse systems. CMOS image sensors typically have several non-ideal factors, such as fixed pattern noise (FPN) and line noise. Full illumination control offers the advantage of periodically acquiring the entire dark data frame and using it to correct pixel and column offsets.

[0146] FPN is dispersion in sensing element offset, which is typically caused by pixel-to-pixel dispersion resulting from random variations in dark current from one photodiode to another. Column fixed pattern noise is caused by offsets associated with specific columns of pixels in the readout chain and can produce perceived vertical stripes within the image.

[0147] Line noise is the random temporal variation of pixel offset within each line. Because line noise is temporal, correction must be recalculated for each line and each frame. For this purpose, there are typically many light-blind (OB) pixels within each line of the array; these pixels must first be sampled to evaluate the line offset before sampling the light-sensitive pixels. The line offset is then subtracted during line noise correction.

[0148] exist Figure 12 In the example, there are additional corrections involving acquiring data in the correct order, monitoring and controlling voltage offsets in the analog domain (black clamp), and identifying / correcting individual defective pixels. Process flow 1200 includes, at 1202, having the sensor cycle through each of the electromagnetic partitions at a first emission intensity. Process flow 1200 includes, at 1204, having the sensor cycle through each of the electromagnetic partitions at a second emission intensity. Process flow 1200 includes, at 1206, combining the data from the electromagnetic partitions at the first and second emission intensities.

[0149] Figure 13 This is a schematic diagram of process flow 1300 for increasing the dynamic range of the resulting image. Process flow 1300 includes, at 1302, causing the sensor to cycle through each of the electromagnetic partitions at a first emission intensity. Process flow 1300 includes, at 1304, causing the sensor to cycle through each of the electromagnetic partitions at a second emission intensity. Process flow 1300 includes, at 1306, causing the sensor to cycle through "n" electromagnetic partitions at an emission intensity of "m", and repeating this process any suitable number of times. Process flow 1300 includes, at 1308, causing the sensor to cycle through "n+i" electromagnetic partitions at an emission intensity of "m+j". Process flow 1300 includes, at 1310, combining the data from each of the cyclic emission intensities.

[0150] In one implementation, the exposure input can be varied over time at different levels and combined to produce a greater dynamic range. A greater dynamic range may be particularly desirable due to the limited spatial environment in which the imaging device is used. In confined spaces with little or no light other than that provided by the light source, and when the light source is close to the light emitter, exposure has an exponential relationship with distance. For example, objects close to the light source and the optical opening of the imaging device tend to be overexposed, while objects further away tend to be severely underexposed due to the presence of very little (any) ambient light.

[0151] As in Figure 13As can be seen, the loop of a system emitting electromagnetic energy in multiple partitions can be continuously looped according to the partitions of the electromagnetic spectrum. For example, in an implementation in which the emitter emits lasers in different red partitions, different blue partitions, different green partitions, and different laser scanning partitions, the two loop datasets to be combined can be of the following form:

[0152] i. A red color with an intensity of one at 1302;

[0153] ii. A red color with an intensity of two at 1304;

[0154] iii. A blue color with an intensity of one at 1302;

[0155] iv. A blue color with an intensity of two at 1304;

[0156] v. A green color with an intensity of one at 1302;

[0157] vi. A green color with an intensity of two at 1304;

[0158] vii. Laser scanning at intensity 1 at 1302; and

[0159] viii. Laser scanning with an intensity of 2 at 1304.

[0160] Alternatively, the system can loop in the following form:

[0161] i. A red color with an intensity of one at 1302;

[0162] ii. A blue color with an intensity of one at 1302;

[0163] iii. Green with an intensity of one at 1302;

[0164] iv. Laser scanning at intensity 1 at 1302;

[0165] v. A red color with an intensity of two at 1304;

[0166] vi. A blue color with an intensity of two at 1304;

[0167] vii. Green with an intensity of two at 1304; and

[0168] viii. Laser scanning with an intensity of 2 at 1304.

[0169] In such implementations, the first image can be derived from an intensity-uniform value, and the second image can be derived from an intensity-binary value, and then combined or processed at 1310 rather than at its constituent parts to form a complete image dataset.

[0170] Within the scope of this invention, it is conceivable that any number of transmission partitions can be used in any order. For example... Figure 13 As shown, "n" is used as a variable to represent any number of electromagnetic zones, and "m" is used to represent the intensity of any level of the "n" zones. Such a system can be looped in the following form:

[0171] i. n with intensity m at 1306;

[0172] ii. n+1 with intensity m+1;

[0173] iii. n+2 with intensity m+2; and

[0174] iv. n+i with intensity m+j at 1308.

[0175] Therefore, any serialized cyclic pattern can be used to generate the desired image correction, where "i" and "j" are additional values ​​within the operating range of the imaging system.

[0176] Figure 14 A process flow 1400, to be implemented by a controller and / or a monochrome image signal processor (ISP), is shown for generating a video stream of RGB images overlaid with laser scanning data. An image signal processor (ISP) chain can be assembled for the purpose of generating a sequence of sRGB images from raw sensor data generated in the presence of a GRGB-laser scanning pulse scheme. In process flow 1400, the first stage involves performing correction (see...). Figure 14 The process involves receiving data from the sensor at 1402, reordering at 1404, and performing sensor correction at 1406 to account for any non-ideal factors in the sensor technology that are best suited to working in the raw data domain. In the next stage, multiple frames are buffered (e.g., green frame 1408a, red-blue frame 1408b, and laser scan frame 1408c), as each final frame derives data from multiple raw frames. Frame reconstruction at 1264 continues by sampling data from the current frame and the two buffered frames (see 1408a, 1408b, and / or 1408c). The reconstruction process produces a full-color frame in a linear RGB color space, including the laser scan data. In this example, a white balance coefficient at 1268 and a color correction matrix at 1420 are applied before conversion to YCbCr space at 1422 for subsequent edge enhancement at 1424. After edge enhancement at 1424, the image is converted back to linear RGB at 1426 for scaling at 1428 (if applicable). Finally, the γ transfer function is applied at 1430 to convert the data to the sRGB domain at 1432.

[0177] Figure 15This is an example of color blending hardware 1500. Color blending hardware 1500 uses a memory writer 1502 to store an RGBG-laser scan video data stream in memory 1504, and at 1505 converts the video data stream into a parallel RGB+laser scan video data stream. The bit width on the input side can be, for example, 12 bits per color. The output width of this example will be at least 36 bits per pixel. Other implementations can have different initial bit widths and numbers three times the output width. The memory writer 1502 block takes the RGBG-laser scan video stream as its input and writes each frame to its correct frame memory 1504 (the memory writer triggers the same pulse generator that powers the laser source (see 1510)). Memory 1504 stores the exposure frame data in a pattern such as the one shown: red, green 1, blue, green 2, laser scan, and then back again starting with red. Memory reader 1506 reads three frames at a time to construct RGB pixels. Each pixel is three times the bit width of a single color component. At 1510, the memory reader 1506 also triggers a laser pulse generator. In one embodiment, the memory reader 1506 waits until a red frame, a green 1 frame, and a blue frame have been written, then continues reading them in parallel while the writer continues writing green 2, the laser scan, and begins returning to red. When red is complete, the reader begins reading from blue, green 2, the laser scan, and red. This pattern continues indefinitely.

[0178] Figure 16 This is a schematic diagram of process flow 1600 for sensor calibration. Process flow 1600 can be used in color and laser scanning pulse systems as discussed herein. Process flow 1600 can be used to counteract non-ideal factors in CMOS image sensors, such as fixed pattern noise (FPN) and row noise. Fixed pattern noise is dispersion in the offset of the sensing element. Typically, most of the dispersion in FPN is inter-pixel dispersion, which originates from random variations in dark current from photodiode to photodiode, among other sources. The system disclosed herein maintains complete control over the illumination source, which makes it possible to acquire dark data and use it to correct pixel and column offsets. In the illustrated example, a single frame buffer can be used to perform a moving average of the entire frame in the absence of light using, for example, a simple exponential smoothing filter. During normal operation, this dark average frame can be subtracted from each illuminated frame. Row noise is the random temporal variation of pixel offset within each row. Because row noise is temporal, corrections are calculated for each row and each frame. For this purpose, there are typically many light-blind (OB) pixels within each row of the pixel array. The OB pixels must be sampled first to evaluate the line offset before the photosensitive pixels are sampled. Then, the row offset is subtracted during the row noise correction process.

[0179] Process flow 1600 includes performing top deserialization 1602 and bottom deserialization 1603. Process flow 1600 includes performing line reordering at the top port at 1604 and at the bottom port at 1605. During line reordering, information can be stored in separate databases 1632, 1634, or other memory devices. Process flow 1600 includes performing black clamping calculation on the top ADC at 1606 and on the bottom ADC at 1607. This information exits process flow 1600 on a first-in-first-out (FIFO) basis. Process flow 1600 includes performing line noise correction on the top ADC at 1608 and on the bottom ADC at 1609. Process flow 1600 includes performing full line reconstruction at 1610 and dark frame accumulation at 1612. This information can be stored in database 1630 or other memory devices before performing fixed pattern noise (FPN) correction. The process flow includes performing fixed pattern noise (FPN) correction at 1614 and pixel defect correction at 1616. Process flow 1600 includes performing programmable digital gain at 1618 before the video stream leaves process flow 1600 to be provided to the user.

[0180] Figure 17 This is a schematic diagram of the pattern reconstruction process. Figure 17 The exemplary pattern shown includes red, green, blue, and laser scan pulses, each lasting for a duration of T1. In various embodiments, the light pulses may have the same duration or different durations. The red, green, blue, and laser scan exposure frames are combined to generate an RGB image overlaid with laser scan data. A single image frame including the red, green, blue, and laser scan exposure frames requires a time period of 4*T1 to generate. Figure 17 The durations shown are merely exemplary and may vary for different specific implementations. In other implementations, different pulse schemes may be employed. For example, the implementation may be based on the timing of each color component or frame (T1), and the reconstructed frame may have a period twice that of the input color frame (2×T1). Different frames within the sequence may have different frame periods, and the average capture rate may be any multiple of the final frame rate.

[0181] In one implementation, the dynamic range of the system is increased by varying the pixel sensitivity of the pixels within the pixel array of the image sensor. Some pixels can sense reflected electromagnetic radiation at a first sensitivity level, while other pixels can sense reflected electromagnetic radiation at a second sensitivity level, and so on. Different pixel sensitivities can be combined to increase the dynamic range provided by the pixel configuration of the image sensor. In one implementation, adjacent pixels are set with different sensitivities such that each cycle includes data generated by pixels that are more sensitive and less sensitive relative to each other. The dynamic range increases when multiple sensitivities are recorded in a single cycle of the pixel array. In one implementation, a wide dynamic range can be achieved by having multiple global TX signals, each TX firing only on a different pixel group. For example, in global mode, a global TX1 signal is firing pixel group 1, a global TX2 signal is firing pixel group 2, a global TXn signal is firing pixel group n, and so on.

[0182] Figure 18A This illustrates a timing example of two different pixel sensitivities (dual-pixel sensitivity) in a pixel array. In this case, the global TX1 signal fires half of the pixels in the array, and the global TX2 signal fires the other half. Because global TX1 and global TX2 have different "on" to "off" edge positions, the integrated light is different between the TX1 and TX2 pixels.

[0183] Figure 18B Different implementations for timing in dual-pixel sensitivity are shown. In this case, the optical pulse is modulated twice (pulse duration and / or amplitude). Pixel TX1 is integrated with pulse P1, and pixel TX2 is integrated with pulses P1+P2. Separating the global TX signal can be done in several ways, including distinguishing TX rows from each row, and sending multiple TX rows per row, where each TX row addresses a different group of pixels.

[0184] Figures 19A-19C This illustrates the use of pulsed and / or synchronized or constant white light emission and the corresponding color sensor. (As shown in...) Figure 19A As can be seen, a white light emitter can be configured to emit a light beam during the blanking period of a corresponding sensor to provide a controlled light source in a controlled lighting environment. The light source can emit a beam of constant magnitude and vary the duration of the pulses, such as... Figure 19A As shown in the diagram, or by changing the pulse value while keeping the pulse constant, data for correct exposure can be achieved, such as... Figure 19B As shown. Figure 19C The diagram shows a graphical representation of a constant light source modulated by a changing current controlled and synchronized with a sensor.

[0185] In one implementation, white light or multispectral light is emitted as pulses to provide data used within the system (in...Figures 19A-19C (Best illustrated in the image). White light emission combined with partitioning of the electromagnetic spectrum can be used to emphasize or de-emphasize certain aspects of a scene. Such implementations can use pulse patterns as follows:

[0186] i. Green pulse;

[0187] ii. Red pulse;

[0188] iii. Blue pulse;

[0189] iv. Laser scanning pulse;

[0190] v. Green pulse;

[0191] vi. A red pulse;

[0192] vii. Blue pulse;

[0193] viii. Laser scanning pulse;

[0194] ix. White light (multispectral) pulse;

[0195] x. (repeated)

[0196] Any system that uses an image sensor to cycle at least twice as fast as white light is intended to fall within the scope of this disclosure. It should be understood that this document contemplates any combination of partitions of the electromagnetic spectrum, whether derived from the visible or invisible part of the full electromagnetic spectrum.

[0197] Figures 20A-20C Each illustrates a light source 2000 having multiple emitters. The emitters include a first emitter 2002, a second reflector 2004, and a third emitter 2006. Additional emitters may be included, as discussed further below. Emitters 2002, 2004, and 2006 may include one or more laser emitters that emit light with different wavelengths. For example, the first emitter 2002 may emit a wavelength consistent with blue laser light, the second emitter 2004 may emit a wavelength consistent with green laser light, and the third emitter 2006 may emit a wavelength consistent with red laser light. For example, the first emitter 2002 may include one or more blue lasers, the second emitter 2004 may include one or more green lasers, and the third emitter 2006 may include one or more red lasers. Lasers 2002, 2004, and 2006 emit laser beams toward a collection region 2008, which may be a waveguide, a mirror, or a device for collecting light and / or directing it toward a waveguide (such as...). Figure 2 The jumper waveguide 206 or cavity waveguide 210 provides the location of other optical components for light.

[0198] In specific embodiments where reagents or dyes that aid in the identification of certain tissues, structures, chemical reactions, biological processes, etc., have been administered to a patient, emitters 2002, 2004, and 2006 may emit wavelengths intended to fluoresce the reagents or dyes. Such wavelengths may be determined based on the reagents or dyes administered to the patient. In such embodiments, the emitter may need to be highly precise in order to emit the desired wavelengths to fluoresce or activate certain reagents or dyes.

[0199] In one specific implementation, transmitters 2002, 2004, and 2006 emit hyperspectral wavelengths of electromagnetic radiation. Certain hyperspectral wavelengths can penetrate tissue and allow physicians to "see through" foreground tissue to identify chemical processes, structures, compounds, biological processes, etc., located behind the foreground tissue. Hyperspectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc., known to have specific spectral responses.

[0200] In one implementation, transmitters 2002, 2004, and 2006 emit laser scanning patterns for mapping the topology of a scene and / or for calculating the dimensions and distances between objects in the scene. In one implementation, the endoscopic imaging system is used in conjunction with multiple tools such as scalpels, retractors, clamps, etc. In such implementations, each of transmitters 2002, 2004, and 2006 can emit a laser scanning pattern such that the laser scanning pattern is projected individually onto each tool. In such implementations, the laser scanning data of each tool can be analyzed to identify the distances between the tool and other objects in the scene.

[0201] exist Figure 20B In this implementation, transmitters 2002, 2004, and 2006 each deliver laser light to the collection region 2008 at different angles. The change in angle can cause a change in the position of electromagnetic energy within the output waveguide. For example, if light enters the fiber bundle (glass or plastic) immediately at the collection region 2008, the changing angle can cause different amounts of light to enter different fibers. For example, the angle can cause an intensity variation across the collection region 2008. Furthermore, light from different transmitters may not be uniformly mixed, so some fibers may receive different amounts of different colors of light. Variations in the color or intensity of light in different fibers can lead to suboptimal lighting of the scene. For example, variations in delivered light or light intensity can cause this in both the scene and the captured image.

[0202] In one embodiment, an intervening optics element may be placed between the fiber bundle and emitters 2002, 2004, 2006 to mix different colors (wavelengths) of light before they enter the fibers or other waveguides. Exemplary intervening optics elements include diffusers, mixing rods, one or more lenses, or other optical components for mixing light such that a given fiber receives the same amount of each color (wavelength). For example, each fiber in the fiber bundle may have the same color. This mixing may result in the same color in each fiber; however, in some embodiments, it may still result in different total brightness delivered to different fibers. In one embodiment, the intervening optics element may also propagate or uniformly distribute light over the collection area such that each fiber carries the same total amount of light (e.g., the light may be diffused in a top-hat profile). Diffusers or mixing rods may cause light loss.

[0203] Although the collection area in 2008 was Figure 20A The term 2008 is used to refer to physical components, but the collection region 2008 can simply be the region that delivers light from the transmitters 2002, 2004, and 2006. In some cases, the collection region 2008 may include optical components such as diffusers, mixing rods, lenses, or any other intermediary optical components located between the transmitters 2002, 2004, 2006 and the output waveguide.

[0204] Figure 20C An embodiment of a light source 2000 is shown, having emitters 2002, 2004, 2006 providing light to a collection area 2008 at the same or substantially the same angle. The light is provided at an angle substantially perpendicular to the collection area 2008. The light source 2000 includes a plurality of dichroic mirrors, including a first dichroic mirror 2010, a second dichroic mirror 2012, and a third dichroic mirror 2014. Dichroic mirrors 2010, 2012, and 2014 include mirrors that reflect light of a first wavelength but transmit (or are transparent to) light of a second wavelength. For example, the third dichroic mirror 2014 may reflect blue laser light provided by a third emitter, while being transparent to red and green light provided by the first emitter 2002 and the second emitter 2004, respectively. The second dichroic mirror 2012 may be transparent to red light from the first emitter 2002 but reflect green light from the second emitter 2004. If other colors or wavelengths are included, dichroic mirrors can be selected to reflect light corresponding to at least one emitter and be transparent to the other emitters. For example, a third dichroic mirror 2014 reflects light from a third emitter 2006 but is transparent to emitters "behind" it, such as a first emitter 2002 and a second emitter 2004. In embodiments where dozens or hundreds of emitters exist, each dichroic mirror can reflect light from its corresponding emitter and the emitters in front of it, while being transparent to the emitters behind it. This allows dozens or hundreds of emitters to transmit electromagnetic energy to the collection area 2008 at substantially the same angle.

[0205] Because these dichroic mirrors allow other wavelengths to transmit or pass through, each of these wavelengths can reach the collection area 2008 from the same angle and / or at the same center point or focal point. Providing light from the same angle and / or the same focal point / center point significantly improves reception and color mixing at the collection area 2008. For example, a particular fiber can receive different colors in proportions identical to the proportions of transmission / reflection by emitters 2002, 2004, 2006 and mirrors 2010, 2012, 2014. Figure 20B Compared to the previous implementation, this significantly improves light mixing at the collection region. In one implementation, any of the optical components discussed herein may be used at the collection region 2008 to collect light before it is supplied to the fiber or fiber bundle.

[0206] Figure 20C An embodiment of a light source 2000 is shown having emitters 2002, 2004, 2006 that also provide light to a collection region 2008 at the same or substantially the same angle. For example, the light incident on the collection region 2008 is deflected from the vertical. Angle 2016 indicates the angle of deflection from the vertical. In one embodiment, laser emitters 2002, 2004, 2006 may have a Gaussian cross-sectional intensity profile. As previously described, an improved distribution of optical energy between fibers can be achieved by forming a flatter or cap-shaped intensity profile. In one embodiment, as angle 2016 increases, the intensity across the collection region 2008 approaches a cap-shaped profile. For example, by increasing angle 2016 until the profile is sufficiently flat, a cap-shaped profile can even approximate a non-flat output beam. A cap-shaped profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intermediary optical components between emitters 2002, 2004, 2006 and the output waveguide, fiber, or fiber bundle.

[0207] Figure 21 This is a schematic diagram showing a single optical fiber 2102 output via a diffuser 2104 at the output point. In one embodiment, the optical fiber 2102 has a diameter of 500 micrometers, a numerical aperture of 0.65, and emits a light cone 2106 of approximately 70 or 80 degrees without the diffuser 2104. With the diffuser 2104, the light cone 2106 may have an angle of approximately 110 or 120 degrees. The light cone 2106 can be the majority of the area where all light arrives and is uniformly distributed. The diffuser 2104 allows for a more uniform distribution of electromagnetic energy in the scene observed by the image sensor.

[0208] In one embodiment, the cavity waveguide 210 includes a single plastic or glass optical fiber of approximately 500 micrometers. Plastic fibers are less expensive, but their width can be reduced through coupling, diffusing, or other methods to allow the fiber to carry a sufficient amount of light into the scene. For example, a smaller fiber may not be able to carry as much light or power as a larger fiber. The cavity waveguide 210 may include a single or multiple optical fibers. The cavity waveguide 210 may receive light directly from a light source or via a jumper waveguide. A diffuser may be used to widen the light output 206 to obtain the desired field of view for the image sensor 214 or other optical components.

[0209] Although Figures 20A-20C Three emitters are shown, but in some embodiments, the number of emitters can range from one to hundreds or more. The emitters may emit light of different wavelengths or spectra, and this light can be used to continuously cover desired portions of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectra). The emitters may be configured to emit visible light such as red, green, and blue light, and may also be configured to emit hyperspectral emission of electromagnetic radiation, fluorescence excitation wavelengths for fluorescing reagents, and / or laser mapping patterns for calculating parameters and distances between objects in a scene.

[0210] Figure 22 A portion of an electromagnetic spectrum 2200 divided into twenty distinct sub-spectrums is shown. The number of sub-spectrums is merely exemplary. In at least one embodiment, spectrum 2200 may be divided into hundreds of sub-spectrums, each having a wavelet band. The spectrum may extend from infrared spectrum 2202, through visible spectrum 2204, and into ultraviolet spectrum 2206. Each sub-spectrum has a wavelet band 2208 covering a portion of spectrum 2200. Each wavelet may be defined by an upper wavelength and a lower wavelength.

[0211] In one embodiment, at least one emitter (such as a laser emitter) is included in a light source (such as light source 202, 2000) to provide complete and continuous coverage of the entire spectrum 2200. For example, the light source used to provide coverage of the illustrated sub-spectrum may include at least 20 different emitters, with at least one emitter for each sub-spectrum. In one embodiment, each emitter covers a 40-nanometer band of the spectrum. For example, one emitter may emit light in a band from 500 nm to 540 nm, while another emitter may emit light in a band from 540 nm to 580 nm. In another embodiment, the emitters may cover bands of other sizes, depending on the type of emitter available or the imaging requirements. For example, multiple emitters may include a first emitter covering a band from 500 nm to 540 nm, a second emitter covering a band from 540 nm to 640 nm, and a third emitter covering a band from 640 nm to 650 nm. Each emitter may cover different segments of the electromagnetic spectrum ranging from far-infrared, mid-infrared, near-infrared, visible, near-ultraviolet, and / or far-ultraviolet. In some cases, multiple transmitters of the same type or wavelength may be included to provide sufficient output power for imaging. The number of transmitters required for a particular waveband may depend on the monochromatic sensor’s sensitivity to the waveband and / or the power output capability of the transmitters in that waveband.

[0212] The bandwidth and coverage provided by the emitter can be selected to provide any desired combination of spectra. For example, continuous coverage of the spectrum using a very small bandwidth (e.g., 10 nm or less) allows for highly selective hyperspectral and / or fluorescence imaging. This bandwidth allows for the selective emission of excitation wavelengths of one or more specific fluorescent reagents. Additionally, the bandwidth allows for the selective emission of portions of hyperspectral electromagnetic radiation for the identification of specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths originate from an emitter that can be selectively activated, extreme flexibility is achieved in fluorescing one or more specific fluorescent reagents during examination. Furthermore, extreme flexibility is achieved in identifying one or more objects or processes via hyperspectral imaging. Therefore, more fluorescence and / or hyperspectral information can be obtained in less time and in a single examination, which would otherwise require multiple examinations and be delayed due to dye application or staining.

[0213] Figure 23This is a schematic diagram illustrating the timing of emission and readout for generating an image. Solid lines represent the readout (peak 2302) and blanking period (valley) for capturing a series of exposure frames 2304-2314. The series of exposure frames 2304-2314 may include a series of repeated exposure frames that can be used to generate laser scanning, hyperspectral, and / or fluorescence data that can be overlaid on an RGB video stream. In one embodiment, a single image frame includes information from multiple exposure frames, one of which includes red image data, another includes green image data, and another includes blue image data. Additionally, a single image frame may include one or more of hyperspectral image data, fluorescence image data, and laser scanning data. These multiple exposure frames are combined to produce a single image frame. The single image frame is an RGB image with fluorescence imaging data. The series of exposure frames includes a first exposure frame 2304, a second exposure frame 2306, a third exposure frame 2308, a fourth exposure frame 2310, a fifth exposure frame 2312, and an Nth exposure frame 2326.

[0214] Furthermore, hyperspectral image data, fluorescence image data, and laser scan data can be combined to identify critical tissues or structures and to measure the dimensions of those critical tissues or structures. For example, hyperspectral image data can be provided to a corresponding system to identify certain critical structures in the body, such as nerves, ureters, blood vessels, and cancerous tissue. The location and identification of the critical structures can be received from the corresponding system and can also be used to generate the topology of the critical structures using laser scan data. For example, the corresponding system determines the location of a cancerous tumor based on hyperspectral imaging data. Because the location of the cancerous tumor is known based on the hyperspectral imaging data, the topology and distance of the cancerous tumor can be calculated based on the laser scan data. This example can also be applied when identifying cancerous tumors or other structures based on fluorescence imaging data.

[0215] In one implementation, each exposure frame is generated based on at least one pulse of electromagnetic energy. The pulse of electromagnetic energy is reflected and detected by an image sensor and subsequently read out in a subsequent readout (2302). Thus, each blanking period and readout results in an exposure frame for a specific electromagnetic energy spectrum. For example, a first exposure frame 2304 may be generated based on the spectrum of a first one or more pulses 2316, a second exposure frame 2306 may be generated based on the spectrum of a second one or more pulses 2318, a third exposure frame 2308 may be generated based on the spectrum of a third one or more pulses 2320, a fourth exposure frame 2310 may be generated based on the spectrum of a fourth one or more pulses 2322, a fifth exposure frame 2312 may be generated based on the spectrum of a fifth one or more pulses 2324, and an Nth exposure frame 2326 may be generated based on the spectrum of an Nth one or more pulses 2326.

[0216] Pulses 2316 to 2326 may include energy from a single emitter or a combination of two or more emitters. For example, the spectrum may be selected to be included within a single readout cycle or multiple exposure frames 2304-2314 for a desired examination or detection of a specific tissue or condition. According to one embodiment, one or more pulses may include visible spectral light for generating RGB or black-and-white images, while simultaneously emitting one or more additional pulses to fluoresce a fluorescent reagent. For example, pulse 2316 may include red light, pulse 2318 may include blue light, and pulse 2320 may include green light, while the remaining pulses 2322-2326 may include wavelengths and spectra for detecting a specific tissue type, fluorescing a reagent, and / or mapping the topology of the scene. Again, pulses in a single readout cycle may include spectra generated by multiple different emitters (e.g., different segments of the electromagnetic spectrum) that can be used to detect a specific tissue type. For example, if a combination of wavelengths results in a pixel having a value above or below a threshold, that pixel may be classified as corresponding to a specific type of tissue. Each frame can also be used to narrow down the type of tissue present at that pixel (e.g., and every pixel in the image) to provide a very specific classification of the tissue and / or the state (disease / health) of the tissue based on the spectral response of the tissue and / or the presence of fluorescent reagents at the tissue site.

[0217] Multiple frames 2304 to 2314 are shown as readout periods of different lengths and pulses of different lengths or intensities. The blanking period, pulse length, or intensity, etc., can be selected based on the monochromatic sensor's sensitivity to a specific wavelength, the transmitter's power output capability, and / or the waveguide's carrying capacity.

[0218] In one implementation, dual image sensors can be used to acquire three-dimensional images or video feeds. Three-dimensional inspection allows for a better understanding of the three-dimensional structure of the inspected area and the mapping of different tissue or material types within that area.

[0219] In one exemplary embodiment, a fluorescent reagent is provided to a patient, and the fluorescent reagent is configured to attach to cancer cells. The fluorescent reagent is known to fluoresce when irradiated by a specific zone of electromagnetic radiation. The relaxation wavelength of the fluorescent reagent is also known. In this exemplary embodiment, the patient is imaged using an endoscopic imaging system as discussed herein. The endoscopic imaging system pulses zones of light with red, green, and blue wavelengths to generate an RGB video stream of the patient's interior. Additionally, the endoscopic imaging system pulses the electromagnetic radiation excitation wavelength of the fluorescent reagent applied to the patient. In this example, the patient has cancer cells, and the fluorescent reagent is attached to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength of the fluorescent reagent, the fluorescent reagent fluoresces and emits a relaxation wavelength. If cancer cells are present in the scene imaged by the endoscopic imaging system, the fluorescent reagent will also be present in the scene and, due to the emission of the excitation wavelength, will emit its relaxation wavelength after fluorescing. The endoscopic imaging system senses the relaxation wavelength of the fluorescent reagent, thereby sensing the presence of the fluorescent reagent in the scene. Because the fluorescent reagent is known to attach to cancer cells, the presence of the fluorescent reagent also indicates the presence of cancer cells in the scene. Endoscopic imaging systems can thus identify the location of cancer cells within a scene. The system can also emit laser scanning pulses to generate the scene's topology and calculate the dimensions of objects within it. The location of cancer cells (as identified by fluorescence imaging data) can be combined with topological and dimensional information calculated based on the laser scanning data. Therefore, the precise location, size, dimensions, and topology of cancer cells can be identified. This information can be provided to a physician to aid in the removal of cancer cells. Additionally, this information can be provided to robotic surgical systems, enabling them to remove cancer cells.

[0220] In another exemplary embodiment, an endoscopic imaging system is used to image a patient to identify quantitative diagnostic information about the patient's histopathology. In this example, the patient is suspected of or known to have a disease that can be tracked using hyperspectral imaging to observe the progression of the disease in the patient's tissues. The endoscopic imaging system pulses partitions of red, green, and blue wavelengths of light to generate an RGB video stream inside the patient's body. Additionally, the endoscopic imaging system pulses one or more hyperspectral wavelengths of light, allowing the system to "see through" some tissue and generate images of the tissue affected by the disease. The endoscopic imaging system senses reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue, thereby identifying the location of the diseased tissue within the patient's body. The endoscopic imaging system may also emit a laser scanning pulse scheme to generate the topology of a scene and calculate the dimensions of objects within the scene. The location of the diseased tissue (identified by the hyperspectral imaging data) can be combined with topological and dimensional information calculated using the laser scanning data. Therefore, the precise location, size, dimensions, and topology of the diseased tissue can be identified. This information can be provided to a physician to assist in the removal, imaging, or study of the diseased tissue. Additionally, this information can be provided to a robotic surgical system so that the system can remove diseased tissue.

[0221] Figure 24 This is a schematic diagram of an imaging system 2400 with a single cutoff filter. System 2400 is particularly suitable for specific implementations of fluorescent reagent imaging. System 2400 includes an endoscope 2406 or other suitable imaging device with a light source 2408 for use in low-light environments. Endoscope 2406 includes an image sensor 2404 and a filter 2402 for filtering out unwanted wavelengths of light or other electromagnetic radiation before reaching the image sensor 2404. Light source 2408 transmits light that can illuminate a surface 2262 in low-light environments such as body cavities. Light 2260 is reflected from surface 2262 and passes through filter 2402 before striking image sensor 2404.

[0222] Filter 2402 can be used in embodiments where a fluorescent reagent or dye has been applied. In such embodiments, light source 2408 emits an excitation wavelength to cause the fluorescent reagent or dye to fluoresce. Typically, the relaxation wavelength emitted by the fluorescent reagent or dye will have a different wavelength than the excitation wavelength. Filter 2402 can be selected to filter out the excitation wavelength and allow only the relaxation wavelength to pass through the filter and be sensed by image sensor 2404.

[0223] In one embodiment, filter 2402 is configured to filter out electromagnetic radiation excitation wavelengths that cause fluorescence in a reagent or dye, such that only the expected relaxation wavelength of the fluorescent reagent or dye is allowed to pass through filter 2402 and reach image sensor 2404. In one embodiment, filter 2402 filters out fluorescent reagent excitation wavelengths at least between 770 nm and 790 nm. In one embodiment, filter 2402 filters out fluorescent reagent excitation wavelengths at least between 795 nm and 815 nm. In one embodiment, filter 2402 filters out fluorescent reagent excitation wavelengths at least between 770 nm and 790 nm and between 795 nm and 815 nm. In these embodiments, filter 2402 filters out the excitation wavelength of the reagent and allows only the relaxation wavelength of the fluorescent reagent to be read by image sensor 2404. Image sensor 2404 may be a wavelength-variable image sensor, and filter 2402 may be configured to allow image sensor 2404 to receive only the relaxation wavelength of the fluorescent reagent and not the emission excitation wavelength of the reagent. The data determined by the image sensor 2404 can then indicate the presence of key body structures, tissues, biological processes, or chemical processes identified by the location of the reagent or dye.

[0224] Filter 2402 can also be used in embodiments where no fluorescent reagent or dye has been applied. Filter 2402 can be selected to allow wavelengths corresponding to the desired spectral response to pass through and be read by image sensor 2404. Image sensor 2404 can be a monochrome image sensor, such that pixels in the captured image that are above or below a threshold can be characterized as corresponding to certain spectral responses or fluorescence emissions. The spectral response or fluorescence emission determined by the pixels captured by image sensor 2404 can indicate the presence of certain body tissues or structures, certain diseases, certain chemical processes, etc.

[0225] Figure 25 This is a schematic diagram of an imaging system 2500 with multiple cutoff filters. System 2400 is particularly suitable for specific implementations of fluorescent reagent imaging. System 2500 includes an endoscope 2506 or other suitable imaging device with a light source 2508 for dark environments. Endoscope 2506 includes an image sensor 2504 and two filters 2502a, 2502b. It should be understood that in alternative embodiments, system 2500 may include any number of filters, and the number and type of filters may be selected for certain purposes, such as for acquiring imaging information of specific body tissues, physical conditions, chemical processes, etc. Filters 2502a, 2502b are configured to prevent the image sensor 2504 from sensing light or other electromagnetic radiation of unwanted wavelengths. Filters 2502a, 2502b may be configured to filter out unwanted wavelengths from white light or other electromagnetic radiation that may be emitted by the light source 2508.

[0226] Compared to Figure 24 The invention also describes filters 2502a and 2502b that can be used in embodiments where fluorescent reagents or dyes have been applied. Filters 2502a and 2502b can be configured to block the emission excitation wavelength of the reagent or dye and allow the image sensor 2504 to read only the relaxation wavelength of the reagent or dye. Furthermore, filters 2502a and 2502b can be used in embodiments where fluorescent reagents or dyes have not been applied. In such embodiments, filters 2502a and 2502b can be selected to allow wavelengths corresponding to the desired spectral response to pass through and be read by the image sensor 2504.

[0227] Multiple filters 2502a, 2502b can each be configured to filter out wavelengths of different ranges in the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than the desired wavelength range, and an additional filter can be configured to filter out wavelengths shorter than the desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 2504.

[0228] In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 513 nm and 545 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 565 nm and 585 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 900 nm and 1000 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 425 nm and 475 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 520 nm and 545 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 625 nm and 645 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 760 nm and 795 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 795 nm and 815 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 370 nm and 420 nm contacts the image sensor 2504. In one embodiment, filters 2502a and 2502b are configured such that electromagnetic radiation between 600 nm and 670 nm contacts the image sensor 2504. In one embodiment, filters 2502a, 2502b are configured to allow only certain fluorescence relaxation emissions to pass through filters 2502a, 2502b and contact image sensor 2504.

[0229] In one embodiment, system 2500 includes a plurality of image sensors 2504, and may specifically include two image sensors for generating three-dimensional images. Image sensors 2504 may be color / wavelength indeterminate and are configured to be used to read electromagnetic radiation of any wavelength reflected from surface 2512. In one embodiment, image sensors 2504 are each color-dependent or wavelength-dependent and are configured to be used to read electromagnetic radiation of a specific wavelength reflected from surface 2512 and returning to image sensor 2504. Alternatively, image sensors 2504 may include a single image sensor having a plurality of different pixel sensors configured to be used to read light of different wavelengths or colors, such as a Bayer color filter array. Alternatively, image sensors 2504 may include one or more color-indeterminate image sensors that may be configured to be used according to pulse scheduling (such as... Figures 5-7E and Figures 15-16 Those shown) read electromagnetic radiation of different wavelengths.

[0230] Figure 26 This is a schematic diagram illustrating a system 2600 for mapping surfaces and / or tracking objects in a dark environment. In one embodiment, an endoscope 2602 pulses a grid array 2606 (which may be referred to as a laser mapping pattern) onto a surface 2604 in a dark environment. Figure 26 In one embodiment shown, the grid array 2606 includes a vertical hash 2608 and a horizontal hash 2610. It should be understood that the grid array 2606 may include any suitable array for mapping the surface 2604, including, for example, a raster grid of discrete points, an occupied grid mapping map, a point array, etc. Additionally, the endoscope 2602 may pulse multiple grid arrays 2606, and may pulse one or more individual grid arrays on each of multiple objects or structures, for example, in a light-deficient environment.

[0231] In one embodiment, system 2600 pulses a grid array 2606, which can be used to map the three-dimensional topology of a surface and / or track the location of objects such as tools or other devices in a light-deficient environment. In one embodiment, system 2600 provides data to a third-party system or computer algorithm for determining surface dimensions and configuration via LiDAR mapping. System 2600 can pulse light or electromagnetic radiation of any suitable wavelength in the grid array 2606, including, for example, ultraviolet, visible, and / or infrared or near-infrared light. Surface 2604 and / or objects within the environment can be mapped and tracked with very high resolution and with very high accuracy and precision.

[0232] In one embodiment, system 2600 includes an imaging device having a tube, one or more image sensors, and a lens assembly having optical elements corresponding to the one or more image sensors. System 2600 may include a light engine having an emitter and a cavity, the emitter generating one or more electromagnetic radiation pulses, the cavity transmitting the one or more electromagnetic radiation pulses to the distal end of an endoscope in a light-deficient environment, such as within a body cavity. In one embodiment, at least a portion of the one or more electromagnetic radiation pulses comprises a laser mapping pattern emitted onto a surface within the light-deficient environment (such as the surface of body tissue within a body cavity and / or the surface of a tool or other device). Endoscope 2602 may include a two-dimensional, three-dimensional, or n-dimensional camera for mapping and / or tracking surfaces, dimensions, and configurations within the light-deficient environment.

[0233] In one embodiment, system 2600 includes a processor for determining the distance between an endoscope or tool and an object, such as surface 2604. The processor may also determine the angle between the endoscope or tool and the object. The processor may also determine surface area information about the object, including, for example, the dimensions of surgical tools, the dimensions of structures, the dimensions of anatomical structures, location information, and other location data and measurements. System 2600 may include one or more image sensors that provide image data output to a control system for determining the distance between the endoscope or tool and the object, such as surface 2604. The image sensors may output information to the control system for determining the angle between the endoscope or tool and the object. Additionally, the image sensors may output information to the control system for determining surface area information about the object, the dimensions of surgical tools, the dimensions of structures, the dimensions of anatomical structures, location information, and other location data and measurements.

[0234] In one implementation, the grid array 2606 is pulsed at a sufficient speed by the transmitter of the endoscope 2602 so that the grid array 2606 is invisible to the user. In various specific implementations, seeing the grid array 2606 during endoscopic imaging procedures and / or endoscopic surgery may be distracting to the user. The grid array 2606 can be pulsed with a sufficiently short cycle so that it is undetectable to the human eye. In an alternative implementation, the endoscope 2602 pulses the grid array 2606 at a sufficient repetition frequency so that the grid array 2606 is viewable by the user. In such implementations, the grid array 2606 may be overlaid on an image of surface 2604 on a display. The grid array 2606 may be overlaid on a black-and-white or RGB image of surface 2604 so that the grid array 2606 is visible to the user during use of system 2600. The user of system 2600 may instruct whether the grid array 2606 should be overlaid on the image of surface 2604 and / or whether the grid array 2606 should be visible to the user. System 2600 may include a display that provides real-time measurement of the distance from endoscope 2602 to surface 2604 or another object in a dark environment. The display may also provide real-time surface area information regarding surface 2604 and / or any object, structure, or tool in the dark environment. The measurement accuracy can be less than one millimeter.

[0235] Endoscope 2602 can pulse electromagnetic radiation according to pulse scheduling (such as those shown herein). For example, this may also include pulsed grid array 2606, and the pulses are used to generate RGB images and also to generate red, green, and blue light for overlay on the RGB images and / or for mapping and tracking surfaces 2604 and objects in a dark environment. Grid array 2606 may additionally combine hyperspectral or fluorescence excitation wavelengths of the electromagnetic radiation for pulsed operation. Data from each of RGB imaging, laser scanning imaging, hyperspectral imaging, and fluorescence imaging can be combined to identify the location, size, and surface topology of key structures within the body.

[0236] In one embodiment, endoscope 2602 includes one or more color-indeterminate image sensors. In one embodiment, endoscope 2602 includes two color-indeterminate image sensors for generating a three-dimensional image or mapping of a dark environment. The image sensors can generate an RGB image of the dark environment according to pulse scheduling as disclosed herein. Additionally, the image sensors can determine data for mapping the dark environment and tracking one or more objects within the dark environment based on data determined during pulse grid array 2606. Furthermore, the image sensors can determine spectral or hyperspectral data and fluorescence imaging data according to pulse scheduling, which can be modified by the user to suit specific needs of the imaging procedure. In one embodiment, pulse scheduling includes red, green, and blue pulses, pulses of grid array 2606, and / or pulses for generating hyperspectral image data and / or fluorescence image data. In various specific embodiments, pulse scheduling can include any suitable combination of electromagnetic radiation pulses as needed by the user. The recurrence frequency of electromagnetic radiation at different wavelengths can be determined based on, for example, the energy of certain pulses, user needs, whether certain data (e.g., hyperspectral data and / or fluorescence imaging data) needs to be continuously updated or can be updated less frequently, etc.

[0237] Pulse scheduling can be modified in any suitable manner, and certain electromagnetic radiation pulses can be repeated at any suitable frequency, depending on the needs of the user or the computer-implemented program used in certain imaging procedures. For example, in an implementation where surface tracking data generated based on mesh array 2606 is provided to a computer-implemented program for, for example, robotic surgery, mesh array 2606 may pulse more frequently than in the case where surface tracking data is provided to a user visualizing the scene during the imaging procedure. In such implementations where surface tracking data is used for robotic surgery, the surface tracking data may need to be updated more frequently or may need to be extremely accurate so that the computer-implemented program can perform robotic surgery with precision and accuracy.

[0238] In one implementation, system 2600 is configured to generate an occupancy grid mapping map comprising an array of cells divided into a grid. System 2600 is configured to store the height value of each of the respective grid cells to determine the surface mapping of a three-dimensional environment in a dark environment.

[0239] Figure 27A and Figure 27BPerspective and side views, respectively, are shown of a specific embodiment of a monolithic sensor 2700 according to the teachings and principles of this disclosure, which has multiple pixel arrays for generating three-dimensional images. Such an embodiment may be desirable for three-dimensional image capture, wherein two pixel arrays 2702 and 2704 can be offset during use. In another embodiment, the first pixel array 2702 and the second pixel array 2704 may be dedicated to receiving electromagnetic radiation within a predetermined wavelength range, wherein the first pixel array is dedicated to electromagnetic radiation within a different wavelength range than the second pixel array.

[0240] Figure 28A and Figure 28B Perspective and side views of a specific embodiment of an imaging sensor 2800 constructed on multiple substrates are shown. As shown, multiple pixel columns 2804 forming the pixel array are located on a first substrate 2802, and multiple circuit columns 2808 are located on a second substrate 2806. The electrical connections and communications between a pixel column and its associated or corresponding circuit column are also shown. In one embodiment, the image sensor may have a pixel array separate from all or most of the supporting circuitry, and it may otherwise be manufactured such that its pixel array and supporting circuitry are on a single, monolithic substrate / chip. This disclosure may use at least two substrates / chips, which will be stacked together using a three-dimensional stacking technique. The first of the two substrates / chips 2802 may be fabricated using an image CMOS process. The first substrate / chip 2802 may consist solely of a pixel array, or it may consist of a pixel array surrounded by a limited circuitry. The second or subsequent substrate / chip 2806 may be fabricated using any process, and is not necessarily derived from an image CMOS process. The second substrate / chip 2806 can be, but is not limited to, a high-density digital process for integrating various and multiple functions into a very limited space or area on the substrate / chip, or a mixed-mode or analog process for integrating, for example, precise analog functions, or an RF process for enabling wireless capabilities, or a MEMS (Micro-Electro-Mechanical System) for integrating MEMS devices. The image CMOS substrate / chip 2802 can be stacked with the second or subsequent substrate / chip 2806 using any three-dimensional technology. The second substrate / chip 2806 can support the majority or most of the circuitry that may be additionally implemented as peripheral circuitry in the first image CMOS chip 2802 (if implemented on a monolithic substrate / chip), and thus increases the overall system area while keeping the pixel array size constant and optimized to the maximum extent possible. Electrical connections between the two substrates / chips can be accomplished via interconnects, which can be bonding leads, lugs, and / or TSVs (Through Silicon Vias).

[0241] Figure 29A and Figure 29BPerspective and side views of a specific embodiment of an imaging sensor 2900 having multiple pixel arrays for generating three-dimensional images are shown, respectively. The three-dimensional image sensor can be constructed on multiple substrates and may include multiple pixel arrays and other associated circuitry, wherein multiple pixel columns 2904a forming a first pixel array and multiple pixel columns 2904b forming a second pixel array are located on respective substrates 2902a and 2902b, and multiple circuit columns 2908a and 2908b are located on a separate substrate 2906. Electrical connections and communications between pixel columns and associated or corresponding circuit columns are also shown.

[0242] The multiple pixel arrays can simultaneously sense information and combine information from the multiple pixel arrays to generate a three-dimensional image. In one embodiment, the endoscopic imaging system includes two or more pixel arrays that can be deployed to generate a three-dimensional image. The endoscopic imaging system may include a transmitter for emitting electromagnetic radiation pulses during the blanking period of the pixel arrays. The pixel arrays can be synchronized such that optical black pixels are read out simultaneously for the two or more pixel arrays (i.e., a blanking period occurs). The transmitter can emit electromagnetic radiation pulses to charge each of the two or more pixel arrays. The two or more pixel arrays can simultaneously read out their respective charged pixels such that the readout periods of the two or more pixel arrays occur simultaneously or substantially simultaneously. In one embodiment, the endoscopic imaging system includes multiple transmitters, each transmitter being individually synchronized with one or more of the multiple pixel arrays. Information from the multiple pixel arrays can be combined to generate three-dimensional image frames and video streams.

[0243] It should be understood that the teachings and principles of this disclosure can be applied to reusable device platforms, limited-use device platforms, reconfigurable device platforms, or single-use / disposable device platforms without departing from the scope of this disclosure. It should be understood that in a reusable device platform, the end user is responsible for cleaning and sterilizing the device. In a limited-use device platform, the device can be used a predetermined number of times before becoming inoperable. Typically, new devices are sterilized before delivery and should be cleaned and sterilized by the end user before any other use if intended for other purposes. In a reconfigurable device platform, a third party can reprocess (e.g., clean, package, and sterilize) a single-use device for additional use at a lower cost than a new unit. In a single-use / disposable device platform, a sterile device is provided to the operating room and can only be used once before being disposed of.

[0244] Example

[0245] The following embodiments relate to preferred features of other implementations:

[0246] Example 1 is a system. The system includes a transmitter for emitting electromagnetic radiation pulses, and an image sensor including a pixel array for sensing reflected electromagnetic radiation. The system includes a controller comprising a processor electrically communicating with the image sensor and the transmitter, wherein the controller synchronizes the timing of the electromagnetic radiation pulses during a blanking period of the image sensor. The system such that at least a portion of the electromagnetic radiation pulses emitted by the transmitter includes a laser scanning pattern.

[0247] Example 2 is a system according to Example 1, wherein the transmitter pulses the laser scanning pattern with a duration and frequency such that the laser scanning pattern is invisible to the user of the system.

[0248] Example 3 is a system according to any one of Examples 1 to 2, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to an electromagnetic radiation pulse emitted by the transmitter, and generates a dataset corresponding in time to each electromagnetic radiation pulse to generate a plurality of datasets corresponding to the plurality of exposure frames.

[0249] Example 4 is a system according to any one of Examples 1 to 3, wherein the plurality of exposure frames and the plurality of datasets are combined to form an image frame.

[0250] Example 5 is a system according to any one of Examples 1 to 4, wherein the pixel array of the image sensor is a dual-sensitivity pixel array, the dual-sensitivity pixel array comprising a plurality of pixels sensitive to long exposure and a plurality of pixels sensitive to short exposure.

[0251] Example 6 is a system according to any one of Examples 1 to 5, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter includes a green zone of electromagnetic radiation, a red zone of electromagnetic radiation, and a blue zone of electromagnetic radiation.

[0252] Example 7 is a system according to any one of Examples 1 to 6, wherein the transmitter is configured to transmit a plurality of electromagnetic radiation subpulses having a sub-duration shorter than the pulse duration during the pulse duration.

[0253] Example 8 is a system according to any one of Examples 1 to 7, wherein one or more electromagnetic radiation pulses emitted by the transmitter include electromagnetic radiation emitted simultaneously as a single pulse or a single sub-pulse at two or more wavelengths.

[0254] Example 9 is a system according to any one of Examples 1 to 8, wherein at least one of the electromagnetic radiation pulses emitted by the transmitter generates an exposure frame generated by the image sensor, and wherein the system further includes a display for displaying two or more exposure frames as a stacked image.

[0255] Example 10 is a system according to any one of Examples 1 to 9, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter is an excitation wavelength for causing the reagent to fluoresce, and wherein pulsing the excitation wavelength causes the image sensor to generate a fluorescence exposure frame indicating the location of the reagent within the scene.

[0256] Example 11 is a system according to any one of Examples 1 to 10, wherein the controller is further configured to provide the fluorescence exposure frame to a corresponding system, the corresponding system determining the location of a critical tissue structure based on the fluorescence exposure frame.

[0257] Example 12 is a system according to any one of Examples 1 to 11, wherein the controller is further configured to: receive the location of the key tissue structure from the corresponding system; generate an overlay frame including the location of the key tissue structure within the scene; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the reagent within the scene.

[0258] Example 13 is a system according to any one of Examples 1 to 12, wherein the image sensor is configured to sense reflected electromagnetic radiation generated by the laser scanning pattern to generate a topology exposure frame, and wherein the controller is further configured to: provide the topology exposure frame to a corresponding system, the corresponding system determining the topology of the scene and / or the dimensions of one or more objects within the scene; provide the location of the key organizational structure to the corresponding system; and receive the topology and / or dimensions of the key organizational structure from the corresponding system.

[0259] Example 14 is a system according to any one of Examples 1 to 13, wherein the key tissue structure is one or more of nerves, ureters, blood vessels, arteries, blood flow, cancerous tissue or tumors.

[0260] Example 15 is a system according to any one of Examples 1 to 14, wherein the blanking period of the image sensor corresponds to the time between the readout of the last row of the pixel array and the start of the next readout cycle of the pixel array.

[0261] Example 16 is a system according to any one of Examples 1 to 15, wherein the controller is further configured to adjust the sequence of electromagnetic radiation pulses emitted by the transmitter based on a threshold, wherein the threshold determines appropriate illumination of the scene in a dark environment.

[0262] Example 17 is a system according to any one of Examples 1 to 16, further comprising a display for displaying a video stream captured by the image sensor, wherein the video stream is assigned 8-bit, 16-bit, or n-bit visible colors for the display.

[0263] Example 18 is a system according to any one of Examples 1 to 17, wherein the transmitter is configured to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising multiple image frames, wherein each image frame in the video stream comprises data from multiple exposure frames, each exposure frame corresponding to an electromagnetic radiation pulse.

[0264] Example 19 is a system according to any one of Examples 1 to 18, wherein the transmitter emits the electromagnetic radiation pulse during the blanking period of the image sensor, such that the pixel array is charged and ready to be read during the readout period of the image sensor.

[0265] Example 20 is a system according to any one of Examples 1 to 19, wherein the image sensor is configured to generate a topological exposure frame by sensing reflected electromagnetic radiation generated by the transmitter pulses the laser scanning pattern, wherein the topological exposure frame includes information for determining real-time measurements, the information including one or more of the following: the distance from the endoscope to the object; the angle between the endoscope and the object; or surface topological information about the object.

[0266] Example 21 is a system according to any one of Examples 1 to 20, wherein the topology exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 cm.

[0267] Example 22 is a system according to any one of Examples 1 to 21, wherein the topology exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

[0268] Example 23 is a system according to any one of Examples 1 to 22, further comprising a plurality of tools, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter includes a tool-specific laser scanning pattern for each of the plurality of tools.

[0269] Example 24 is a system according to any one of Examples 1 to 23, wherein the laser scanning pattern emitted by the transmitter includes a first output and a second output that are independent of each other, wherein the first output is used for illumination and the second output is used for tool tracking.

[0270] It should be understood that the various features disclosed herein offer significant advantages and advancements in the art. The following claims are examples of some of those features.

[0271] In the specific embodiments described above, for the purpose of simplification, the various features of this disclosure are concentrated in a single embodiment. The method of this disclosure should not be construed as implying an intention that the disclosure protected by the claims requires more features than expressly listed in each claim. Rather, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0272] It should be understood that any feature of the above-described arrangement, embodiments, and implementations may be combined in a single implementation that includes a combination of features obtained from any of the disclosed arrangement, embodiments, and implementations.

[0273] It should be understood that the above-described configuration is merely an exemplary application of the principles of this disclosure. Many modifications and alternative configurations can be devised by those skilled in the art without departing from the spirit and scope of this disclosure, and the appended claims are intended to cover such modifications and configurations.

[0274] Therefore, when this disclosure is illustrated and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications can be made without departing from the principles and ideas set forth herein, including but not limited to changes in size, material, shape, form, function and operation, assembly and use.

[0275] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to execute one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. Those skilled in the art will understand that components may have different names. This document is not intended to distinguish between components that are different in name rather than function.

[0276] For illustrative and descriptive purposes, the specific embodiments described above have been provided. These specific embodiments are not intended to be exhaustive or to limit this disclosure to the specific forms disclosed. Many modifications and changes can be made to this disclosure based on the foregoing teachings. Furthermore, it should be noted that any or all of the foregoing alternative embodiments can be used in any desired combination to form further hybrid embodiments of this disclosure.

[0277] Furthermore, while specific embodiments of this disclosure have been described and illustrated, this disclosure is not limited to the particular forms or arrangements of components as described and illustrated. The scope of this disclosure will be defined by the appended claims, any future claims filed herein and in different applications, and their equivalents.

Claims

1. A system comprising: A transmitter for emitting electromagnetic radiation pulses; An image sensor, the image sensor comprising a pixel array for sensing reflected electromagnetic radiation; and The controller includes a processor that is in electrical communication with the image sensor and the transmitter; The controller synchronizes the operation of the transmitter and the image sensor, such that the transmitter pulses electromagnetic radiation during the blanking period of the image sensor; The transmitter includes an electromagnetic radiation source configured to emit pulsed electromagnetic radiation in a laser scanning pattern. The image sensor is configured to generate a topological exposure frame by sensing electromagnetic radiation reflected from the laser scanning pattern.

2. The system according to claim 1, wherein, The transmitter pulses the laser scanning pattern with a duration and frequency such that the laser scanning pattern is invisible to the user of the system.

3. The system according to claim 1, wherein, The image sensor is configured to generate multiple exposure frames, each of which corresponds to an electromagnetic radiation pulse emitted by the transmitter, and to generate a dataset corresponding to each electromagnetic radiation pulse in time to generate multiple datasets corresponding to the multiple exposure frames.

4. The system according to claim 3, wherein, The multiple exposure frames are combined to form an image frame.

5. The system according to claim 1, wherein, The pixel array of the image sensor is a dual-sensitivity pixel array, which includes multiple pixels sensitive to long exposures and multiple pixels sensitive to short exposures.

6. The system according to claim 1, wherein, At least a portion of the electromagnetic radiation pulse emitted by the transmitter includes a green zone, a red zone, and a blue zone of electromagnetic radiation.

7. The system according to claim 1, wherein, The transmitter is configured to emit multiple electromagnetic radiation subpulses with sub-durations shorter than the pulse duration during the pulse duration.

8. The system according to claim 1, wherein, One or more of the electromagnetic radiation pulses emitted by the transmitter include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

9. The system according to claim 1, wherein, At least one of the electromagnetic radiation pulses emitted by the transmitter generates an exposure frame generated by the image sensor, wherein the system further includes a display for displaying two or more exposure frames as a stacked image.

10. The system according to claim 1, wherein, At least a portion of the electromagnetic radiation pulse emitted by the transmitter is an excitation wavelength for causing the reagent to fluoresce, and wherein pulsing the excitation wavelength causes the image sensor to generate a fluorescence exposure frame indicating the location of the reagent within the scene.

11. The system according to claim 10, wherein, The controller is also configured to provide the fluorescence exposure frame to a corresponding system, which determines the location of key tissue structures based on the fluorescence exposure frame.

12. The system according to claim 11, wherein, The controller is also configured to: Receive the location of the key organizational structure from the corresponding system; Generate an overlay frame that includes the location of the key organizational structure within the scene; as well as The overlay frame is combined with a color image frame depicting the scene to indicate the location of the reagent within the scene.

13. The system of claim 12, wherein the controller is further configured to: The topology exposure frame is provided to a corresponding system, which determines the topology of the scene and / or the dimensions of one or more objects within the scene; Provide the location of the key organizational structure to the corresponding system; and Receive the topology and / or dimensions of the key organizational structure from the corresponding system.

14. The system according to claim 13, wherein, The key tissue structures are one or more of the following: nerves, ureters, blood vessels, arteries, blood flow, cancerous tissue, or tumors.

15. The system according to claim 1, wherein, The blanking period corresponds to the time between the readout of the last row of valid pixels in the pixel array and the start of the next subsequent readout of valid pixels in the pixel array.

16. The system according to claim 1, wherein, The controller is also configured to adjust the sequence of electromagnetic radiation pulses emitted by the transmitter based on a threshold, wherein the threshold determines appropriate lighting for a scene in a dark environment.

17. The system of claim 1, further comprising a display for displaying a video stream captured by the image sensor, wherein the video stream is assigned 8-bit, 16-bit, or n-bit visible colors for the display.

18. The system according to claim 1, wherein, The transmitter is configured to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising multiple image frames, wherein each image frame in the video stream includes data from multiple exposure frames, each exposure frame corresponding to an electromagnetic radiation pulse.

19. The system according to claim 1, wherein, The transmitter emits the electromagnetic radiation pulse during the blanking period of the image sensor, thereby charging the pixel array and preparing it to be read during the readout period of the image sensor.

20. The system of claim 1, wherein the topology exposure frame includes information for determining real-time measurements, the information including one or more of the following: The distance from the endoscope to the object; The angle between the endoscope and the object; or Surface topology information about the object.

21. The system according to claim 20, wherein, The topology exposure frame includes information for determining the real-time measurement with an accuracy of less than 10 centimeters.

22. The system according to claim 20, wherein, The topology exposure frame includes information for determining the real-time measurement with an accuracy of less than one millimeter.

23. The system of claim 1 further comprises a plurality of tools, wherein at least a portion of the electromagnetic radiation pulse emitted by the transmitter includes a tool-specific laser scanning pattern for each of the plurality of tools.

24. The system according to claim 1, wherein, The laser scanning pattern emitted by the transmitter includes a first output and a second output that are independent of each other, wherein the first output is used for illumination and the second output is used for tool tracking.

Images