Efficient time division multiplexing in fluorescence and spectral imaging

Abstract

A visualization system (106) with customizable illumination cycles for addressing relative brightness differences between different illumination sources. A system includes an image sensor (124) including an array of pixels (125) and a transmitter (102) including a plurality of electromagnetic radiation sources (138), wherein the plurality of electromagnetic radiation sources includes a first source and a second source. The transmitter cycles at least a portion of the plurality of electromagnetic radiation sources according to a pulse period including a plurality of pulse emissions by the first source and a plurality of pulse emissions by the second source. When the image sensor reads out a data frame corresponding to the transmission of the first source, at least a portion of the plurality of pulsed transmissions of the second source overlaps a readout period of the image sensor.

Description

Technical Field

[0001] This disclosure relates to advanced visualization and digital imaging systems and methods, and more specifically, but not entirely, to efficient time-division multiplexing of different lighting sources. Background Technology

[0002] Endoscopic surgical instruments are often superior to traditional open surgical devices because smaller incisions tend to reduce postoperative recovery time and associated complications. In some cases of endoscopic visualization, it is desirable to utilize high-resolution color imaging and further utilize one or more advanced visualization techniques to view the space, providing additional information that is indistinguishable to the human eye. However, the spatially confined environment of endoscopy presents numerous technical challenges when attempting to capture advanced visualization data in low-light conditions.

[0003] Many endoscopic visualization systems exist that attempt to capture advanced visualization data, such as multispectral, fluorescence, and laser mapping data, while operating within the space-constrained environment of an endoscope. However, these endoscopic visualization systems do not address the inherent characteristics of image sensors, resulting in varying efficiency of pixel arrays in accumulating electromagnetic radiation across different bands of the electromagnetic spectrum.

[0004] Conventional fluorescence visualization systems implementing time-division multiplexing illumination can make white light and fluorescent excitation pulses last for equal durations. In a conventional example implementation, the system instructs a white light source to emit white light for a defined duration, and then instructs an image sensor to read out the pixel array. The system can then instruct a fluorescent light source to emit fluorescent excitation light for the same defined duration, and then instruct the image sensor to read out the pixel array. Therefore, in this conventional multiplexing visualization system, the white light source and the fluorescent light source are instructed to emit light at different times and for the same defined duration.

[0005] The aforementioned traditional visualization systems do not consider the differences in brightness or energy output from white and fluorescent light sources. They also fail to account for the inherently different sensitivities of pixel arrays to different wavelengths of light. In most cases, fluorescent light sources are relatively dim compared to white light sources, making it difficult for image sensors to accumulate sufficient energy when exposed to fluorescent excitation light. In traditional visualization systems, image frames output in response to fluorescent light emission are darker and have lower accumulated energy levels compared to image frames output in response to white light emission. While traditional visualization systems may attempt to correct for energy differences using digital post-processing of the image, they do not consider energy differences during image acquisition.

[0006] For example, co-owned U.S. Patent Application Publication No. 2019 / 0404131, filed October 24, 2019, entitled “HYPERSPECTRAL AND FLUORESCENCEIMAGING WITH TOPOLOGY LASERSCANNING IN ALIGHT DEFICIENT ENVIRONMENT,” describes an endoscopic visualization system for color and “dedicated” imaging, the entire contents of which are incorporated herein by reference. In this disclosure, the emitter is configured to emit red, green, blue, and dedicated emission light, wherein the dedicated emission may include hyperspectral, fluorescent, or laser mapping emission. However, this disclosure does not indicate that the duration of light emission can be optimized or adjusted based on the inherent sensitivity of the pixel array to electromagnetic radiation of different wavelengths. Furthermore, this disclosure does not indicate that multiple light sources can be emitted simultaneously to enable the image sensor to accumulate more energy from dimmer light sources.

[0007] For example, U.S. Patent No. 10,958,852, filed September 25, 2019, entitled “IMAGING APPARATUSAND CONTROL METHODHAVING A PLURALITY OF SYNCHRONIZEDLIGHT SOURCE DEVICES,” describes an imaging device with three image sensors that receive illumination from a white light source and an infrared laser. The entire contents of this patent are incorporated herein by reference. In this disclosure, the three image sensors are dedicated to sensing a combination of green, blue, or red light with infrared light. The white light source and the infrared laser can be actuated according to various modes, including in which the infrared laser is continuously displayed and the white light source is turned on and off. However, this disclosure does not indicate that the white light source and the infrared laser can be synchronized such that the pulses of the infrared laser overlap with the readout period of the color image frame but not with the emission of the white light source. Furthermore, this disclosure does not indicate that two of the image sensors can sense color and infrared visualization data for stereoscopic visualization.

[0008] Therefore, there is a significant need for visualization systems capable of capturing color (including high-resolution color, multispectral, fluorescence, and laser mapping imaging data) frame-by-frame with consistent exposure, regardless of energy differences between white light, multispectral, fluorescence, and laser mapping sources. Specifically, there is a need to extend the duration of image sensor exposure to weaker light sources without sacrificing the quality of the resulting image frames.

[0009] Based on the foregoing, this paper describes systems, methods, and apparatuses for fluorescence, multispectral, and laser mapping imaging in low-light environments, wherein a weaker light source can be emitted simultaneously with a stronger light source. Attached Figure Description

[0010] Non-limiting and incomplete embodiments of this disclosure are described with reference to the following accompanying drawings, wherein, unless otherwise specified, similar reference numerals in the various views indicate similar parts. The advantages of this disclosure will be better understood with reference to the following description and drawings, wherein:

[0011] Figure 1A This is a schematic diagram of an example system for endoscopic visualization using color imaging and advanced imaging.

[0012] Figure 1B This is a schematic diagram of an example image pickup section of a system for endoscopic visualization using color imaging and advanced imaging.

[0013] Figure 1C This is a schematic diagram of an example transmitter and controller for a system that utilizes color imaging and advanced imaging for endoscopic visualization;

[0014] Figure 2A This is a schematic block diagram of an example data flow used in a time series visualization system;

[0015] Figure 2B This is a schematic block diagram of an example data flow used in a time series visualization system;

[0016] Figure 2C It is a schematic flowchart of the data flow used to capture and read data from a time series visualization system;

[0017] Figure 3A This is a schematic block diagram of an example system for processing data output from an image sensor, which has a controller that communicates with a transmitter and an image sensor.

[0018] Figure 3B This is a schematic block diagram of an example system for processing data output from an image sensor to generate color imaging data and advanced imaging data;

[0019] Figure 3C This is a schematic block diagram of an example system for processing data through a memory buffer to provide data frames to an image signal processor at regular intervals;

[0020] Figure 4 This is a schematic diagram of a lighting system used to illuminate a dimly lit environment based on a variable pulse period;

[0021] Figure 5 This is a schematic diagram of the synchronous operation cycle of the transmitter and the image sensor;

[0022] Figure 6 This is a schematic diagram of the synchronous operation cycle of the transmitter and the image sensor;

[0023] Figure 7 This is a schematic diagram of the synchronous operation cycle of the transmitter and the image sensor;

[0024] Figure 8A It is a schematic diagram of an example mapping pattern including a grid array;

[0025] Figure 8B It is a schematic diagram of an example mapping pattern including a dot array;

[0026] Figure 9 This shows a portion of the electromagnetic spectrum divided into multiple different bands, which can be pulsed by the electromagnetic radiation source of the transmitter;

[0027] Figure 10 This is a schematic diagram illustrating the timing of transmission and readout for generating data frames in response to electromagnetic radiation pulses; and

[0028] Figure 11 This is a schematic block diagram of an example computing device. Detailed Implementation

[0029] This document discloses systems, methods, and apparatuses for digital visualization, primarily applicable to medical applications such as medical endoscopic imaging. One embodiment of this disclosure is an endoscopic system for color visualization and "advanced visualization" of a scene. Advanced visualization includes one or more of multispectral imaging, fluorescence imaging, or topographic mapping. Data retrieved from advanced visualization can be processed by one or more algorithms configured to determine the characteristics of the scene. Specifically, advanced visualization data can be used to identify tissue structures within the scene, generate three-dimensional topographic maps of the scene, calculate the dimensions of objects within the scene, identify edges and boundaries of different tissue types, etc.

[0030] The systems, methods, and apparatus described herein are specifically optimized to address the variation between “stronger” and “weaker” electromagnetic radiation (EMR) sources. In some cases, a stronger EMR source is considered “stronger” based on the inherent characteristics of the pixel array; for example, if the pixel array is inherently more sensitive to detecting EMR emitted by a stronger EMR source, then the stronger EMR source can be classified as “stronger” when compared to another EMR source. Conversely, if the pixel array is inherently less sensitive to detecting EMR emitted by a weaker EMR source, then the weaker EMR source can be classified as “weaker” when compared to another EMR source. Furthermore, a “stronger” EMR source may have a higher amplitude, greater brightness, or higher energy output compared to a “weaker” EMR source. This disclosure addresses the difference between stronger and weaker EMR sources by adjusting the pulse period of the transmitter to ensure that the pixel array has sufficient time to accumulate a adequate amount of EMR corresponding to each of the stronger and weaker EMR sources.

[0031] One embodiment of this disclosure is an endoscopic visualization system including a transmitter, an image sensor, and a controller. The transmitter includes multiple individual and independently actuable EMR sources that can be individually cyclically turned on and off to illuminate a scene with EMR pulses. The image sensor accumulates EMR and reads out data used to generate multiple data frames. The controller synchronizes the operation of the transmitter and the image sensor to output a desired visualization scheme based on user input. The visualization scheme may include one or more of color imaging, multispectral imaging, fluorescence imaging, morphological mapping, or anatomical measurement.

[0032] The controller instructs the transmitter and image sensor to operate in a synchronized sequence, thereby outputting a video stream comprising one or more types of visualization (i.e., color imaging, multispectral imaging, fluorescence imaging, morphological mapping, or anatomical measurement). The controller instructs the transmitter to actuate one or more of a plurality of EMR sources to pulse according to a variable pulse period. The controller instructs the image sensor to accumulate EMR and read out data according to a variable sensor period synchronized in time with the variable pulse period. The synchronized sequence of the transmitter and image sensor enables the image sensor to read out data corresponding to multiple different visualization types. For example, the image sensor can read out color frames in response to EMR pulses of white light or other visible light from the transmitter; the image sensor can read out multispectral frames in response to EMR pulses of multispectral bands from the transmitter; the image sensor can read out data for calculating 3D topographic maps in response to EMR pulses of patterned pulses from the transmitter, and so on.

[0033] The controller optimizes and adjusts the sensor cycle of the image sensor to output data frames at a sufficient rate for color imaging and / or advanced imaging, while ensuring that the pixel array accumulates sufficient EMR for each data frame. The controller can instruct the image sensor to perform pixel merging on a per-frame basis, allowing the image sensor to merge pixels from some data frames and read out all pixels from other data frames. In some cases, the controller instructs the image sensor to read out all pixels, thereby outputting high-resolution color data frames in response to the emitter pulse white light EMR. The controller can also instruct the image sensor to merge the pixel array and read out fewer pixels in response to the emitter pulse EMR for advanced visualization, such as multispectral imaging, fluorescence imaging, or topography.

[0034] The controller can further optimize and adjust the variable pulse period in real time based on user input, the full exposure of the resulting data frame, and the inherent characteristics of the corresponding pixel array. In some cases, the pixel array has different sensitivities to different bands of the EMR. In these cases, depending on the type of illumination pulse, the pixel array is illuminated with the EMR for a shorter or longer duration to ensure that the pixel array outputs data frames with a consistent exposure level. The controller adjusts the illumination time of the pixel array and the pulse duration of the transmitter in real time to compensate for the varying efficiency of the pixel array to different types of illumination.

[0035] Implement the systems and methods described herein for color visualization and advanced visualization. The advanced visualization techniques described herein can be used to identify certain tissues, visualize tissues in a foreground, calculate the 3D topography of a scene, and calculate the size and distance of objects within a scene. Specifically, the advanced visualization techniques described herein include multispectral visualization, fluorescence visualization, and laser mapping visualization.

[0036] Multispectral visualization

[0037] Spectral imaging uses multiple bands across the electromagnetic spectrum. This differs from conventional cameras that capture only light across three wavelengths (including red, green, and blue wavelengths) of the visible spectrum, which are distinguishable by the human eye, to generate an RGB image. Spectral imaging can use any band of wavelengths in the electromagnetic spectrum, including infrared wavelengths, the visible spectrum, the ultraviolet spectrum, X-ray wavelengths, or any suitable combination of various wavelength bands. Spectral imaging can overlay an image generated based on invisible light bands (such as infrared) onto an image based on visible light bands (such as a standard RGB image) to provide additional information that can be easily distinguished by humans or computer algorithms.

[0038] The multispectral imaging techniques discussed in this paper can be used to "see through" tissue layers in the foreground of a scene to identify specific tissue types and / or specific biological or chemical processes. Multispectral imaging can be used in medical settings to quantitatively track disease progression and determine histopathology. Additionally, multispectral imaging can be used to identify critical structures such as neural tissue, muscle tissue, cancer cells, and blood vessels. In one implementation, multispectral partitions of the EMR are pulsed, and in response to the EMR partitions, data on the spectral responses of different tissue types are collected. Data storage of the spectral responses can be generated and analyzed to assess the scene and predict which tissues are present within the scene based on the sensed spectral responses.

[0039] Multispectral imaging offers numerous advantages over conventional imaging. Information obtained through multispectral imaging allows physicians and / or computer-based programs to accurately identify certain tissues or conditions that might be undetectable using RGB imaging. Furthermore, multispectral imaging can be used during medical procedures to provide image-guided surgery, enabling physicians to, for example, view tissues hidden behind certain tissues or fluids, identify atypical cancer cells in contrast to typical healthy cells, identify specific tissues or conditions, and identify critical structures. Multispectral imaging provides specialized diagnostic information about tissue physiology, morphology, and composition that cannot be generated using conventional imaging.

[0040] Fluorescence visualization

[0041] Fluorescence occurs when orbital electrons in molecules, atoms, or nanostructures are excited by light or other EMRs and then relax to their ground state by emitting photons from the excited state. The specific frequency of the EMR, whether the orbital electrons are excited or emitted by photons during relaxation, depends on the specific atom, molecule, or nanostructure. In most cases, the light emitted by the material has a longer wavelength than the radiation absorbed by the material, and therefore has lower energy.

[0042] Fluorescence imaging is particularly useful in biochemistry and medicine as a non-destructive means of tracking or analyzing biomolecules. Biomolecules, including certain tissues or structures, are tracked by analyzing their fluorescence emission after excitation by EMR at certain wavelengths. However, relatively few cellular components are naturally fluorescent. In some implementations, it may be advantageous to visualize certain tissues, structures, chemical processes, or biological processes that are inherently non-fluorescent. In such implementations, dyes or reagents may be administered to the body, including molecules, proteins, or quantum dots with fluorescent properties. The reagents or dyes may then fluoresce after excitation by EMR at certain wavelengths. Different reagents or dyes may include different molecules, proteins, and / or quantum dots that will fluoresce at specific wavelengths of EMR. Therefore, it may be necessary to excite reagents or dyes with EMR at specific frequency bands to achieve fluorescence and identify desired tissues, structures, or processes in vivo.

[0043] The fluorescence imaging techniques described herein can be used to identify certain substances, tissues, components, or processes within body cavities or other light-deficient environments. Fluorescence imaging data can be provided to practicing physicians or computer-implemented algorithms to enable the identification of certain structures or tissues within the body. Such fluorescence imaging data can be overlaid on black-and-white or RGB images to provide additional information and context.

[0044] The fluorescence imaging techniques described herein can be implemented in conjunction with fluorescent reagents or dyes. Some reagents or dyes are known to adhere to certain types of tissues and fluoresce at specific wavelengths of the electromagnetic spectrum. In one embodiment, a reagent or dye is administered to a patient, and this reagent or dye is configured to fluoresce when activated by light of certain wavelengths. The visualization system disclosed herein is used to excite the reagents or dyes and cause them to fluoresce. The fluorescence of the reagents or dyes is detected by an imaging sensor to aid in the identification of tissues or structures in body cavities. In one embodiment, multiple reagents or dyes are administered to a patient, each configured to fluoresce at different wavelengths and / or provide indications of different structures, tissues, chemical reactions, biological processes, etc. In such embodiments, the visualization system described herein emits each of the applicable wavelengths to cause each of the applicable reagents or dyes to fluoresce. This eliminates the need to perform separate imaging procedures for each of the multiple reagents or dyes.

[0045] Laser mapping visualization

[0046] Laser mapping typically involves the controlled deflection of a laser beam. Laser mapping can be performed to generate one or more 3D topographic maps of a scene, calculate distances between objects within the scene, calculate the dimensions of objects within the scene, track the relative positions of tools within the scene, and so on.

[0047] Laser mapping combines the controlled directional aiming of a laser beam with a laser rangefinder. By measuring distance in each direction, the laser rangefinder can rapidly capture the surface shape of objects, tools, and landscapes. The construction of a full three-dimensional topography can include combining multiple surface models obtained from different viewpoints. Various measurement systems and methods exist in this field for applications in archaeology, geography, atmospheric physics, autonomous vehicles, and more. One such system includes Light Detection and Ranging (LIDAR), a three-dimensional mapping system. LIDAR has been used in navigation systems such as aircraft or satellites to determine the position and orientation of sensors combined with other systems and sensors. LIDAR uses active sensors to illuminate objects and detects the energy reflected from the objects and back to the sensors.

[0048] As described herein, the term "laser mapping" includes laser tracking. Laser tracking, or the use of lasers for tool tracking, measures objects by determining the position of an optical target relative to the object. Laser trackers can be accurate to the order of 0.025 mm over distances of several meters. The visualization system described herein, pulsed EMR, is used in conjunction with a laser tracking system to enable the tracking and measurement of the position of tools within a scene.

[0049] The endoscopic visualization system described herein enables laser mapping imaging to determine precise measurements and topographic contours of a scene. In one specific implementation, mapping data is used to determine precise measurements between, for example, structures or organs within a body cavity, devices or tools within a body cavity, and / or key structures within a body cavity. As described herein, the term "mapping" encompasses techniques referred to as laser mapping, laser scanning, topographic scanning, 3D scanning, laser tracking, tool tracking, etc. Mapped data frames, as discussed herein, include data for calculating one or more of the following: topographic map of the scene, dimensions of objects or structures within the scene, distances between objects or structures within the scene, relative positions of tools or other objects within the scene.

[0050] For the purpose of facilitating an understanding of the principles of this disclosure, reference will now be made to embodiments shown in the accompanying drawings, and these embodiments will be described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any changes and further modifications to the features of the invention shown herein, as well as any additional applications of the principles of this disclosure as shown herein (which will generally occur to those skilled in the art and those familiar with the contents of this disclosure), will be considered within the scope of the disclosure protected by the claims.

[0051] Before disclosing and describing the structures, systems, and methods, it should be understood that this disclosure is not limited to the specific structures, configurations, process steps, and materials disclosed herein, as such structures, configurations, process steps, and materials can vary to some extent. Furthermore, it should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the invention will be defined only by the appended claims and their equivalents.

[0052] In describing and claiming the subject matter of this disclosure, the following terms will be used in accordance with the following definitions.

[0053] It should be noted that the singular forms “a,” “an,” and “the” used in this specification and the appended claims include multiple referents unless the context clearly indicates otherwise.

[0054] As used herein, the terms “comprising,” “including,” “characterized by,” and their grammatical equivalents are non-exhaustive or open-ended terms that do not exclude additional, unmentioned elements or method steps.

[0055] As used herein, the phrase “consisting of” and its grammatical equivalents exclude any element or step not included in the claims.

[0056] As used herein, the phrase “consistent with…” and its grammatical equivalents limit the scope of the claims to the specified materials or steps and to materials or steps that do not substantially affect one or more of the essential and novel features of the claimed disclosure.

[0057] As used in this article, the term "proximal" broadly refers to the concept of the part closer to the starting point.

[0058] As used in this article, the term "far side" is generally used in contrast to the proximal side, and therefore, depending on the context, it refers to the part that is farther from the starting point or the farthest part.

[0059] As used herein, a color sensor is a known sensor with a color filter array (CFA) on which the incident EMR is filtered to its single component. In the visible range of the electromagnetic spectrum, such a CFA can be based on a Bayer template or a modified form thereof to separate the green, red, and blue spectral components in the visible light EMR.

[0060] As used in this article, a monochrome sensor refers to an unfiltered imaging sensor, which includes pixels that cannot distinguish colors.

[0061] Now refer to the attached diagram, Figures 1A-1C A schematic diagram of a system 100 for endoscopic visualization is shown. System 100 includes a transmitter 102, a controller 104, and an optical visualization system 106. System 100 includes one or more tools 108, which may include endoscopic tools such as forceps, brushes, scissors, cutters, drills, suturers, ligation devices, tissue suturers, suturing systems, etc. System 100 includes one or more endoscopes 110, such as arthroscopes, bronchoscopes, colonoscopes, colposcopes, cystoscopes, esophagoscopes, gastroscopes, laparoscopes, laryngoscopes, neuroendoscopes, rectoscopes, sigmoidoscopes, thoracoscopes, etc.

[0062] The optical visualization system 106 may be located at the distal end of the lumen of the endoscope 110. Alternatively, one or more components of the optical visualization system 106 may be located at the proximal end of the lumen of the endoscope 110 or in another region of the endoscope 110. The optical visualization system 106 may include one or more image sensors 124, each image sensor including a pixel array (see First as... Figure 2AThe pixel array 125 shown herein. The optical visualization system 106 may include one or more lenses 126 and filters 128, and may also include one or more prisms 132 for reflecting EMR onto the pixel array 125 of one or more image sensors 124. The system 100 may include a waveguide 130 configured to transmit EMR from the transmitter 102 to the distal end of the endoscope 110 to illuminate a dark environment for visualization. The system 100 may also include a waveguide 131 configured to transmit EMR from the transmitter 102 to a termination point on the tool 108, which may be specifically actuated for laser mapping imaging and tool tracking as described herein.

[0063] The optical visualization system 106 may specifically include two lenses 126 dedicated to each image sensor 124 to focus the EMR onto the rotating image sensor 124 and achieve a depth view. The filter 128 may include a notch filter configured to block unwanted reflected EMR. In certain applications, unwanted reflected EMR may include fluorescence excitation wavelengths pulsed by the emitter 102, where the system 100 intends to detect only fluorescence relaxation wavelengths emitted by fluorescent reagents or tissue.

[0064] Image sensor 124 includes one or more image sensors, and as... Figure 1A-Figure 1B The illustrated example embodiment shows an optical visualization system 106 comprising two image sensors 124. Image sensors 124 may include CMOS image sensors and may specifically include high-resolution image sensors configured to read out data according to a rolling readout scheme. Image sensors 124 may include multiple different image sensors tuned to collect EMR at different efficiencies in different bands. In one embodiment, image sensors 124 include separate image sensors optimized for color imaging, fluorescence imaging, multispectral imaging, and / or topography mapping.

[0065] Emitter 102 includes one or more EMR sources, which may include, for example, a laser, a laser beam, a light-emitting diode (LED), a discharge power source, an incandescent light source, an electroluminescent source, etc. In some embodiments, emitter 102 includes at least one white light EMR source 134 (which may be referred to herein as a white light source). Emitter 102 may additionally include one or more EMR sources 138, which are tuned to emit EMR in a specific band. Specifically, EMR source 138 may be tuned to emit EMR bands selected for multispectral visualization or fluorescence visualization. Emitter 102 may additionally include one or more mapping sources 142, which are configured to emit EMR in a mapping pattern, such as a grid array or dot array selected for capturing data for topographic mapping or anatomical measurements.

[0066] One or more white light EMR sources 134 emit EMR into a dichroic mirror 136, which feeds the white light EMR into a waveguide 130. Specifically, the white light EMR source 134 may feed into a first waveguide 130a dedicated to white light EMR. An EMR source 138 emits EMR into separate dichroic mirrors 140, each of which feeds EMR into waveguide 130 and may specifically feed into a second waveguide 130b. The first waveguide 130a and the second waveguide 130b are then combined into a waveguide 130 that transmits the EMR to the distal end of the endoscope 110 to illuminate the scene using the emission of EMR 144.

[0067] One or more EMR sources 138, tuned to emit an EMR in a specific band, may be specifically tuned to emit an EMR selected for multispectral visualization or fluorescence visualization. In some cases, the EMR source 138 is finely tuned to emit an EMR with a center wavelength having a tolerance threshold of no more than ±5 nm, ±4 nm, ±3 nm, ±2 nm, or ±1 nm. The EMR source 138 may include a laser or laser beam, which is individually and cyclically turned on and off by the transmitter 102 to pulse the emission of the EMR 144 and illuminate the scene through the finely tuned band of the EMR.

[0068] One or more mapping sources 142 are configured to pulse EMR with a mapping pattern, which may include a dot array, a grid array, a vertical hash, a horizontal hash, a pin grid array, etc. The mapping pattern used for laser mapping imaging is selected to determine one or more of the following: a three-dimensional topographic map of the scene, the distance between two or more objects within the scene, the size of objects within the scene, the position of tool 108 within the scene, etc. The EMR diffraction pulsed by the mapping source 142 will spread the energy wave according to the desired mapping pattern. The mapping source 142 may specifically include means for EMR beam splitting using a quantum dot array diffraction grating. The mapping source 142 may be configured to emit low-mode laser light.

[0069] Controller 104 (which may be referred to herein as a camera control unit or CCU) may include a field-programmable gate array (FGPA) 112 and a computer 113. FGPA 112 may be configured to perform overlay processing 114 and image processing 116. Computer 113 may be configured to generate pulse periods 118 for transmitter 102 and perform further image processing 120. FGPA 112 receives data from image sensor 124 and may combine data from two or more data frames via overlay processing 114 to output an overlay image frame. Computer 113 may provide data to transmitter 102 and image sensor 124. Specifically, computer 113 may calculate and adjust the variable pulse period to be transmitted by transmitter 102 in real time based on user input. Additionally, computer 113 may receive data frames from image sensor 124 and perform further image processing 120 on those data frames.

[0070] Controller 104 can communicate with networks such as the Internet and automatically upload data to the network for remote storage. MCU 122 and image sensor 124 can be exchanged and updated, and continue to communicate with the established controller 104. In some cases, controller 104 may be "obsolete" relative to MCU 122, but will still successfully communicate with MCU 122. This can improve data security in hospitals or other healthcare facilities, as the existing controller 104 can be configured to withstand a wide range of security protocols to protect patient data.

[0071] The controller 104 can reprogram the image sensor 124 for each data frame to set the blanking period duration and / or readout period duration required for subsequent frame cycles. A frame cycle includes a blanking period and a readout period. Generally, the pixel array 125 accumulates EMR during the blanking period and reads out pixel data during the readout period. It should be understood that the blanking period corresponds to the time between the readout of the last row of valid pixels in the pixel array of the image sensor and the start of the next subsequent readout of valid pixels in the pixel array. Additionally, the readout period corresponds to the duration during which valid pixels in the pixel array are being read. Furthermore, the controller 104 can write the correct registers to the image sensor 124 to adjust the duration of one or more of the blanking period or readout period for each frame cycle as needed within the sensor cycle.

[0072] Endoscope 110 includes a microcontroller unit (MCU) 122 disposed therein. MCU 122 may be specifically disposed within the handheld portion of endoscope 110 and communicates with electronic circuitry (such as image sensor 124) disposed within the distal end of the lumen of endoscope 110. MCU 122 receives instructions from controller 104, including indications of pulse period 118 provided to transmitter 102 and corresponding sensor period timings for image sensor 124. MCU 122 executes a general application programming interface (API). Controller 104 communicates with MCU 122, and MCU 122 performs a conversion function that translates instructions received from controller 104 into the correct format for each type of image sensor 124. In some cases, system 100 may include multiple different image sensors, each operating according to a different "language" or format, and MCU 122 is configured to translate instructions from controller 104 into each of the appropriate data format languages. The general-purpose API on MCU122 transmits information via scene, including parameters such as gain, exposure, white balance, and setpoint. Depending on the type of visualization, MCU122 runs a feedback algorithm to controller 104 for any number of parameters.

[0073] MCU 122 stores operational data and images captured by image sensor 124. In some cases, MCU 122 does not need to continuously push data up the data chain to controller 104. Data can be set once on microcontroller 122, and then only critical information can be pushed to controller 104 through the feedback loop. MCU 122 can be configured in several modes, including master mode (which may be referred to as "master" mode when master / slave communication protocols are involved). MCU 122 ensures that all downstream components (i.e., including remote components of image sensor 124, which may be referred to as "slave" in master / slave communication protocols) are notified of the configuration for the upcoming data frame. The upcoming configuration may include, for example, gain, exposure duration, readout duration, pixel binning configuration, etc.

[0074] MCU 122 includes internal logic for executing triggers to coordinate different devices, including, for example, multiple image sensors 124. MCU 122 provides instructions for an upcoming frame and executes triggers to ensure that each image sensor 124 begins capturing data simultaneously. In some cases, image sensors 124 can automatically advance to subsequent data frames without needing to receive a unique trigger from MCU 122.

[0075] In some cases, endoscope 110 includes two or more image sensors 124 that simultaneously detect EMR and output data frames. Simultaneous data frames can be used to output three-dimensional images and / or output images with increased sharpness and dynamic range. The pixel array of image sensor 124 may include effective pixels and optical black (“OB”) pixels or optically blind pixels. Optical black pixels can be read during the blanking cycle of the pixel array when the pixel array is “reset” or calibrated. After the optical black pixels have been read, effective pixels are read during the readout cycle of the pixel array. Effective pixels accumulate EMR pulsed by transmitter 102 during the blanking cycle of image sensor 124. Pixel array 125 may include monochrome or “colorless” pixels that do not include any filters for selectively receiving EMR in certain bands. The pixel array may include a color filter array (CFA), such as a Bayer patterned CFA, which selectively allows EMR in certain bands to pass through the filter and be accumulated by the pixel array.

[0076] Image sensor 124 is instructed by a combination of cooperating MCU 122 and controller 104. Ultimately, MCU 122 provides image sensor 124 with instructions on how to capture upcoming data frames. These instructions include, for example, indications of the upcoming data frame's gain, exposure, white balance, exposure duration, readout duration, pixel binning configuration, etc. While image sensor 124 is reading data from the current data frame, MCU 122 is rewriting the correct registers for the next data frame. MCU 122 and image sensor 124 operate in a round-trip data stream, where image sensor 124 provides data to MCU 122, and MCU 122 rewrites the correct registers to image sensor 124 for each upcoming data frame. In some configurations, MCU 122 and image sensor 124 can operate according to a "ping-pong buffer."

[0077] Image sensor 124, MCU 122, and controller 104 participate in a feedback loop to continuously adjust and optimize the configuration of upcoming data frames based on output data. MCU 122 continuously rewrites the correct registers to image sensor 124 based on the type of the upcoming data frame (i.e., color data frame, multispectral data frame, fluorescence data frame, topography mapping data frame, etc.), the configuration of previous output data frames, and user input. In an example implementation, image sensor 124 outputs multispectral data frames in response to EMR pulses in the multispectral band from transmitter 102. MCU 122 and / or controller 104 determine that the multispectral data frame is underexposed and cannot be successfully analyzed by the corresponding machine learning algorithm. MCU 122 and / or controller 104 then adjust the configuration for the upcoming multispectral data frames to ensure proper exposure in future multispectral data frames. MCU 122 and / or controller 104 may indicate that gain, exposure duration, pixel binning configuration, etc., must be adjusted for future multispectral data frames to ensure proper exposure. All image sensors 124 can be configured to adjust in real time based on previous output data processed through a feedback loop and further based on user input.

[0078] Waveguides 130 and 131 include one or more optical fibers. The optical fibers may be made of low-cost materials (such as plastic) to allow for the arrangement of one or more of waveguides 130 and 131. In some embodiments, one or more of waveguides 130 and 131 include a single glass fiber having a diameter of 500 micrometers. In some embodiments, one or more of waveguides 130 and 131 include multiple glass fibers.

[0079] Figure 2A and Figure 2B Each diagram illustrates a data stream 200 for time-series visualization in low-light environments. For example... Figures 2A-2B The data stream 200 shown can be generated by, for example Figures 1A-1C The system 100 shown is used for endoscopic visualization. Figure 2A A general implementation of EMR that can be applied to any type of lighting or wavelength is shown. Figure 2B An example implementation of a visible light EMR source, a multispectral EMR source, a fluorescence EMR source, and a mapping EMR source is shown, wherein the emitter 102 actuates the visible light EMR source, the multispectral EMR source, the fluorescence EMR source, and the mapping EMR source.

[0080] Data stream 200 includes a transmitter 102, a pixel array 125 (not shown) of an image sensor 124, and an image signal processor 140. The image signal processor 140 may include, for example... Figure 1A and Figure 1C One or more of the image processing modules 116 and 120 shown. Transmitter 102 includes multiple separate and independently actuable EMR sources (see, for example, ...). Figure 1A and Figure 1C (As shown in 134, 138). Each EMR source can be cyclically turned on and off to emit EMR pulses with defined duration and amplitude. The pixel array 125 of the image sensor 124 may include a color filter array (CFA) or an unfiltered array comprising pixels that cannot distinguish colors. The transmitter 102 and the pixel array 125 are each connected to the controller 104 (not in...). Figures 2A-2B (As shown in the diagram) communication, the controller instructs the transmitter 102 and pixel array 125 to operate synchronously to generate multiple data frames according to the desired visualization scheme.

[0081] Controller 104 instructs transmitter 102 to cycle through multiple EMR sources according to a variable pulse period. Controller 104 calculates the variable pulse period based at least in part on user input indicating a desired visualization scheme. For example, the desired visualization scheme might indicate that the user wants to view the scene using only color imaging. In this case, the variable pulse period might include only pulses of white light EMR. In an alternative example, the desired visualization scheme might indicate that the user wants to visualize the scene when neural tissue can be identified in the scene and / or when an tool in the scene is within a threshold distance from neural tissue. In this example, the variable pulse period might include pulses of white light EMR and might also include pulses of one or more multispectral bands of EMR that elicit a spectral response from neural tissue and / or “see through” non-neural tissue by penetrating it. Additionally, the variable pulse period might include pulses of EMR in a mapping pattern configured for laser mapping imaging to determine when an tool is within a threshold distance from neural tissue. Controller 104 can reconfigure the variable pulse period in real time in response to receiving a revised desired visualization scheme from the user.

[0082] Figure 2A and Figure 2B Each diagram illustrates a data stream 200 for time-series visualization in low-light environments. For example... Figures 2A-2B The data stream 200 shown can be generated by, for example Figures 1A-1C The system 100 shown is used for endoscopic visualization. Figure 2A A general implementation of EMR that can be applied to any type of lighting or wavelength is shown. Figure 2B An example implementation of a visible light EMR source, a multispectral EMR source, a fluorescence EMR source, and a mapping EMR source is shown, wherein the emitter 102 actuates the visible light EMR source, the multispectral EMR source, the fluorescence EMR source, and the mapping EMR source.

[0083] Data stream 200 includes a transmitter 102, a pixel array 125 (not shown) of an image sensor 124, and an image signal processor 140. The image signal processor 140 may include, for example... Figure 1A and Figure 1COne or more of the image processing modules 116 and 120 shown. Transmitter 102 includes multiple separate and independently actuable EMR sources (see, for example, ...). Figure 1A and Figure 1C (As shown in 134, 138). Each EMR source can be cyclically turned on and off to emit EMR pulses with defined duration and amplitude. The pixel array 125 of the image sensor 124 may include a color filter array (CFA) or an unfiltered array comprising pixels that cannot distinguish colors. The transmitter 102 and the pixel array 125 are each connected to the controller 104 (not in...). Figures 2A-2B (As shown in the diagram) communication, the controller instructs the transmitter 102 and pixel array 125 to operate synchronously to generate multiple data frames according to the desired visualization scheme.

[0084] Controller 104 instructs transmitter 102 to cycle through multiple EMR sources according to a variable pulse period. Controller 104 calculates the variable pulse period based at least in part on user input indicating a desired visualization scheme. For example, the desired visualization scheme might indicate that the user wants to view the scene using only color imaging. In this case, the variable pulse period might include only pulses of broad-spectrum visible light EMR. In an alternative example, the desired visualization scheme might indicate that the user wants to be notified when neural tissue can be identified in the scene and / or when an tool in the scene is within a threshold distance from neural tissue. In this example, the variable pulse period might include white light EMR pulses and may also include pulses of one or more multispectral bands of EMR that elicit a spectral response from neural tissue and / or “see through” non-neural tissue by penetrating it. Additionally, the variable pulse period might include pulsed EMR with a mapping pattern configured for laser mapping imaging to determine when an tool is within a threshold distance from neural tissue. Controller 104 can reconfigure the variable pulse period in real time in response to receiving a revised desired visualization scheme from the user.

[0085] Figure 2A The diagram illustrates how a transmitter cyclically turns one or more EMR sources on and off to emit EMR pulses during each of a plurality of individual blanking cycles of the pixel array 125. Specifically, transmitter 102 emits pulsed EMR during each of the T1, T2, T3, and T4 blanking cycles of the pixel array 125. The pixel array 125 accumulates EMR during its blanking cycles and reads out data during its readout cycles.

[0086] Specifically, pixel array 125 accumulates EMR during the T1 blanking period and reads out T1 data frames during the T1 readout period following the T1 blanking period. Similarly, pixel array 125 accumulates EMR during the T2 blanking period and reads out T2 data frames during the T2 readout period following the T2 blanking period. Pixel array 125 accumulates EMR during the T3 blanking period and reads out T3 data frames during the T3 readout period following the T3 blanking period. Pixel array 125 accumulates EMR during the T4 blanking period and reads out T4 data frames during the T4 readout period following the T4 blanking period. Each of the T1, T2, T3, and T4 data frames is provided to image signal processor 140.

[0087] The content of each of the T1-T4 data frames depends on the type of EMR pulsed by transmitter 102 during the previous blanking cycle. For example, if transmitter 102 pulses white light during the previous blanking cycle, the resulting data frame may include a color data frame (if pixel array 125 includes a color filter array for outputting red, green, and blue image data). Also, for example, if transmitter 102 pulses EMR in a multispectral band during the previous blanking cycle, the resulting data frame is a multispectral data frame that includes information for identifying the spectral response of one or more objects within the scene and / or information for “seeing through” one or more structures within the scene. Also, for example, if transmitter 102 pulses EMR in a fluorescence excitation band during the previous blanking cycle, the resulting data frame is a fluorescence data frame that includes information for identifying the fluorescent reagent or autofluorescence response of tissues within the scene. For example, if transmitter 102 pulses a mapped pattern EMR during the previous blanking period, the resulting data frame is a mapped data frame that includes one or more of the following information for calculating the three-dimensional topography of the scene, the dimensions of one or more objects in the scene, the distance between two or more objects in the scene, etc.

[0088] Some “machine vision” or “computer vision” data frames, including multispectral data frames, fluorescence data frames, and mapping data frames, can be provided to corresponding algorithms or neural networks configured to evaluate the information contained therein. Multispectral algorithms can be configured to identify one or more tissue structures within a scene based on how those structures respond to the selection of one or more different bands of EMR for multispectral imaging. Fluorescence algorithms can be configured to identify the location of fluorescent reagents or autofluorescent tissue structures within a scene. Mapping algorithms can be configured to calculate one or more of the following based on mapping data frames: a 3D topographic map of the scene, a depth map, the size of one or more objects within the scene, and / or the distance between two or more objects within the scene.

[0089] Figure 2BAn example is shown in which emitter 102 cycles through separate visible light EMR sources, multispectral EMR sources, fluorescence EMR sources, and mapping EMR sources to emit pulsed visible light 204, pulsed multispectral light 206, pulsed fluorescence light 208, and pulsed EMR emitted in a mapping pattern 210. It should be understood that... Figure 2B This is merely illustrative, and launches 204, 206, 208, and 210 can be launched in any order, as shown in... Figure 2B It can be launched during a single visualization scene as shown, and can also be launched during separate visualization scenes.

[0090] Pixel array 125 reads out color data frames 205 in response to pulsed visible light 204 EMR by emitter 102. Pulsed visible light 204 EMR may specifically include white light pulses. Pixel array 125 reads out multispectral data frames 207 in response to pulsed multispectral 206 EMR by emitter 102. Pulsed multispectral 206 EMR may specifically include one or more EMRs in the bands of approximately 513 nm–545 nm, 565 nm–585 nm, 770 nm–790 nm, and / or 900 nm–1000 nm. It should be understood that pulsed multispectral 206 EMR may include various other bands for inducing a spectral response. Pixel array 125 reads out fluorescence data frames 209 in response to pulsed fluorescence 208 EMR by emitter 102. The pulsed fluorescence 208 band EMR can specifically include one or more EMRs in the 770nm-795nm and / or 790nm-815nm bands. Pixel array 125 reads out mapped data frame 211 in response to a pulse from transmitter 102 to map the EMR of pattern 210. Pulse mapping pattern 210 can include one or more of vertical hash, horizontal hash, pin grid array, dot array, discrete point grating grid, etc. Each of color data frame 205, multispectral data frame 207, fluorescence data frame 209, and mapped data frame 211 is provided to image signal processor 140.

[0091] In one embodiment, transmitter 102 pulses red, green, and blue visible light EMRs, respectively. In this embodiment, pixel array 125 may include a monochrome (colorless) array of pixels. Pixel array 125 can read out red, green, and blue data frames individually in response to individual pulses of the red, green, and blue visible light EMRs.

[0092] In one embodiment, transmitter 102 pulses a band of visible light EMR separately, the band of which is selected to capture luminance (“Y”) imaging data, red chromaticity (“Cr”) imaging data, and blue chromaticity (“Cb”) imaging data. In this embodiment, pixel array 125 can read out luminance data frames (including only luminance imaging information), red chromaticity data frames, and blue chromaticity data frames, respectively.

[0093] Figure 2C A schematic flowchart illustrating the processing flow for the operation of the synchronous transmitter 102 and the pixel array 125 is shown. This processing flow corresponds to... Figure 2A The schematic diagram is shown. The processing flow includes a controller 104 that instructs the transmitter 102 to pulse EMR during the T1 blanking period of the pixel array 125, and then instructs the pixel array 125 to read data during the T1 readout period following the T1 blanking period. Similarly, the controller 104 instructs the transmitter to pulse EMR during each of the T2, T3, and T4 blanking periods. The controller 104 also instructs the transmitter to read data during each of the T2, T3, and T4 readout periods following the corresponding blanking period. Each of the output data frames is provided to the image signal processor 140.

[0094] The transmitter 102 pulses according to a variable pulse cycle comprising one or more types of EMR. The variable pulse cycle may include visible light EMR, which may include white light emission, red light emission, green light emission, blue light emission, or some other bands of visible light EMR. White light emission may be pulsed using white light-emitting diodes (LEDs) or other light sources, and may alternatively be pulsed using a combination of red, green, and blue light sources. The variable pulse cycle may include one or more bands of EMR selected for multispectral imaging or fluorescence imaging. The variable pulse cycle may include one or more EMRs emitted in a mapping pattern selected for three-dimensional topographic mapping or calculating dimensions within a scene. In some cases, several types of EMR are presented in a variable pulse cycle with a regularity different from other types of EMR. This can be implemented to emphasize and de-emphasize multiple aspects of the recorded scene according to the user's needs.

[0095] The controller 104 adjusts the variable pulse period in real time based on the visualization target. This system allows the user to input one or more visualization targets and change those targets while using the system. For example, a visualization target might indicate that the user wants to view only color imaging data, in which case the variable pulse period could include pulses or constant emission of white light (or other visible light EMR). A visualization target might indicate that the user wants to be notified when the scene includes one or more types of tissue or condition that can be identified using one or more of color imaging, multispectral imaging, or fluorescence imaging. A visualization target might indicate that a fluorescent reagent or dye has been administered to the patient and that fluorescence imaging should continue while the reagent or dye remains active. A visualization target might indicate that the user wants to view a three-dimensional topographic map of the scene, receive information about distances or dimensions within the scene, receive an alarm when the tool enters within a critical distance of a tissue structure, and so on.

[0096] The variable pulse period may include one or more finely tuned sections of the electromagnetic spectrum, which are selected to elicit a fluorescence response from a reagent, dye, or autofluorescent tissue. The fluorescence excitation bands of EMR include one or more of the following: 400±50nm, 450±50nm, 500±50nm, 550±50nm, 600±50nm, 650±50nm, 700±50nm, 710±50nm, 720±50nm, 730±50nm, 740±50nm, 750±50nm, 760±50nm, 770±50nm, 780±50nm, 790±50nm, 800±50nm, 810±50nm, 820±50nm, 830±50nm, 840±50nm, 850±50nm, 860±50nm, 870±50nm, 880±50nm, 890±50nm, or 900±50nm. The aforementioned wavelength bands can be finely tuned, allowing the transmitter to pulse center wavelengths with tolerance thresholds of ±100nm, ±90nm, ±80nm, ±70nm, ±60nm, ±50nm, ±40nm, ±30nm, ±20nm, ±10nm, ±8nm, ±6nm, ±5nm, ±4nm, ±3nm, ±2nm, and ±1nm. In some cases, the transmitter comprises multiple laser beams, each configured to pulse an EMR at a specific wavelength with a tolerance threshold of no more than ±5nm, ±4nm, ±3nm, or ±2nm.

[0097] Variable pulse cycling may include one or more bands of EMR tuned for multispectral imaging. These bands of the EMR are selected to elicit a spectral response from a specific tissue or to penetrate a specific tissue (so that material set behind that tissue can be visualized). The multispectral bands of the EMR include one or more of the following: 100±50 nm, 150±50 nm, 200±50 nm, 250±50 nm, 300±50 nm, 350±50 nm, 400±50 nm, 450±50 nm, 400±50 nm, 410±50 nm, 420±50 nm, 430±50 nm, 440±50 nm, 450±50 nm, 460±50 nm, 470±50 nm, 480... ±50nm, 490±50nm, 500±50nm, 510±50nm, 520±50nm, 530±50nm, 540±50nm, 550±50nm, 560±50nm, 570±50nm, 580±50nm, 590±50nm, 600±50nm, 610±50nm, 620±50nm, 630±50nm, 640±50nm, 650±50nm, 660±50nm, 6 70±50nm, 680±50nm, 690±50nm, 700±50nm, 710±50nm, 720±50nm, 730±50nm, 740±50nm, 750±50nm, 760±50nm, 770±50nm, 780±50nm, 790±50nm, 800±50nm, 810±50nm, 820±50nm, 830±50nm, 840±50nm, 850±50nm 860±50nm, 870±50nm, 880±50nm, 890±50nm, 900±50nm, 910±50nm, 920±50nm, 930±50nm, 940±50nm, 950±50nm, 960±50nm, 970±50nm, 980±50nm, 990±50nm, 1000±50nm, 900±100nm, 950±100nm, or 1000±100nm. The aforementioned wavelength bands can be finely tuned, allowing the transmitter to pulse center wavelengths with tolerance thresholds of ±100nm, ±90nm, ±80nm, ±70nm, ±60nm, ±50nm, ±40nm, ±30nm, ±20nm, ±10nm, ±8nm, ±6nm, ±5nm, ±4nm, ±3nm, ±2nm, and ±1nm. In some cases, the transmitter comprises multiple laser beams, each configured to pulse an EMR at a specific wavelength with a tolerance threshold of no more than ±5nm, ±4nm, ±3nm, or ±2nm.

[0098] Certain multispectral wavelengths penetrate tissue and allow physicians to "see through" foreground tissue to identify chemical processes, structures, compounds, biological processes, etc., located behind the foreground tissue. Multispectral wavelengths can be specifically selected to identify specific diseases, tissue conditions, biological processes, chemical processes, tissue types, etc., known to have certain spectral responses.

[0099] The variable pulse period may include one or more EMR emissions optimized for mapping imaging, which includes, for example, 3D topography mapping, depth map generation, calculating distances between objects within a scene, calculating the size of objects within a scene, determining whether a tool or other object is close to a threshold distance from another object, etc. Pulses used for laser mapping imaging include EMRs formed with a mapping pattern, which may include one or more of vertical hashes, horizontal hashes, dot arrays, etc.

[0100] Controller 104 optimizes the variable pulse period to adapt to various imaging and video standards. In most applications, the system output includes a video stream of at least 30 frames per second (fps). Controller 104 synchronizes the operation of the transmitter and image sensor to output data at a sufficient frame rate for scene visualization and further for processing the scene using one or more advanced visualization techniques. Users can request a real-time color video stream of the scene and can further request information based on one or more of multispectral imaging, fluorescence imaging, or laser mapping imaging (which may include topographic mapping, calculation of size and distance, etc.). Controller 104 enables the image sensor to sense color data frames, multispectral data frames, fluorescence data frames, and mapping data frames separately based on the transmitter's variable pulse period.

[0101] In some cases, the user requests more data types than the system can accommodate while maintaining a smooth video frame rate. The system is constrained by the image sensor's ability to accumulate sufficient electromagnetic energy during each blanking cycle to output data frames with adequate exposure. In some cases, the image sensor outputs data at a rate of 60fps-120fps, and specifically at 60fps. In these cases, for example, controller 104 can use 24fps-30fps for color visualization, and can use additional frames per second for one or more advanced visualization techniques.

[0102] The controller 104 calculates and adjusts the variable pulse period of the transmitter 102 in real time, at least in part, based on the known capabilities of the pixel array 125. The controller 104 has access to data stored in memory indicating how long the pixel array 125 must be exposed to EMR in a specific band to accumulate sufficient EMR for outputting data frames with adequate exposure. In most cases, the pixel array 125 inherently has different sensitivities to different bands of EMR. Therefore, the pixel array 125 may require longer or shorter blanking period durations for some bands of EMR to ensure that all data frames output by the image sensor 124 have sufficient exposure levels.

[0103] Controller 104 determines various advanced visualization algorithms (see, for example) Figure 3B The described algorithms (346, 348, 350) have specific data input requirements. For example, controller 104 may determine that certain advanced visualization algorithms do not require the same regularity of data input as a 30fps color video stream output. In these cases, controller 104 may optimize the variable pulse period to include white light pulses at a rate more frequent than those used for advanced visualization (such as multispectral, fluorescence, or laser mapping imaging). Additionally, controller 104 determines whether certain algorithms can be operated with lower-resolution data frames read out by the image sensor using a pixel-binning configuration. In some cases, controller 104 ensures that all color frames provided to the user are read out at high resolution (without pixel binning). However, some advanced visualization algorithms (see, for example, 346, 348, 350) can be executed with lower-resolution data frames.

[0104] System 100 may include multiple image sensors 124, which may have different or identical pixel array configurations. For example, one image sensor 124 may include a monochrome or "colorless" pixel array without filters, another image sensor 124 may include a pixel array with a Bayer pattern CFA, and yet another image sensor 124 may include a pixel array with different CFAs. Multiple image sensors 124 can be assigned to detect EMR for a specific imaging modality, such as color imaging, multispectral imaging, fluorescence imaging, or laser mapping imaging. Furthermore, each of the image sensors 124 may be configured to simultaneously accumulate EMR and output data frames, enabling all image sensors to sense data from all imaging modalities.

[0105] The controller 104 prioritizes certain advanced visualization techniques based on the user's end goal. In some cases, the controller 104 prioritizes other advanced visualization techniques over other advanced visualization techniques to output a smooth and high-definition color video stream to the user. In other cases, the controller 104 prioritizes one or more advanced visualization techniques over color visualization, and in these cases, the output color video stream may appear discontinuous to the human eye because the system outputs color imaging data at less than 30fps.

[0106] For example, a user can indicate that a fluorescent reagent has been administered to a patient. If the fluorescent reagent is time-sensitive, the controller 104 can ensure a sufficient frame rate is dedicated to fluorescence imaging to ensure the user receives enough fluorescence imaging data while the reagent remains active. In another example, the user requests a notification whenever their tool enters within a threshold distance of a tissue (such as a blood vessel, nerve fiber, cancerous tissue, etc.). In this example, the controller 104 can prioritize laser mapping visualization to continuously determine the distance between the user's tool and surrounding structures, and can further prioritize multispectral or fluorescence imaging that enables the system to identify certain tissues. The controller 104 can also prioritize color visualization to ensure the user continues to view a color video stream of the scene.

[0107] Figures 3A-3C A schematic diagram of a system 300 for processing data output from an image sensor 124 including a pixel array 125 is shown. System 300 includes a controller 104 communicating with each of a transmitter 102 and the image sensor 124 including the pixel array 125. The transmitter 102 includes one or more visible light sources 304, a multispectral source 306, a fluorescence source 308, and a mapping pattern source 310 for EMR. The pixel array data readout 342 of the image sensor 124 includes, as in combination with... Figure 2B One or more of the color data frame 205, multispectral data frame 207, fluorescence data frame 209, and mapping data frame 211 discussed.

[0108] like Figure 3B As shown, all data read from the pixel array can undergo frame correction 344 processing by the image signal processor 140. In various specific embodiments, one or more of the color data frame 205, multispectral data frame 207, fluorescence data frame 209, and mapping data frame 211 undergo the frame correction 344 process. Frame correction 344 includes one or more of sensor correction, white balance, color correction, or edge enhancement.

[0109] The multispectral data frame 207 may undergo spectral processing 346 performed by the image signal processor 140 and / or another processor outside the system 300. Spectral processing 346 may include machine learning algorithms and may be performed by a neural network configured to process the multispectral data frame 207 to identify tissue structures based on whether one or more tissue structures in the scene emit spectral responses.

[0110] The fluorescence data frame 209 may undergo fluorescence processing 348 performed by an image signal processor 140 and / or another processor external to system 300. Fluorescence processing 348 may include machine learning algorithms and may be performed by a neural network configured to process the fluorescence data frame 209 and identify intensity maps, wherein fluorescence relaxation wavelengths are detected by the pixel array.

[0111] The mapped data frame 211 may undergo topography processing 350 performed by the image signal processor 140 and / or another processor external to the system 300. Topography processing 350 may include machine learning algorithms and may be performed by a neural network configured to evaluate time-of-flight information to compute a depth map representing the scene. Topography processing 350 includes computing one or more of the following: a three-dimensional topography map of the scene; the dimensions of one or more objects within the scene; the distance between two or more objects within the scene; the distance between an tool and an organizational structure within the scene.

[0112] Figure 3C A schematic diagram of a system 300 and processing flow for managing data output at irregular rates is shown. An image sensor 124 operates according to a sensor cycle that includes a blanking cycle and a readout cycle. At the end of each readout cycle, the image sensor 124 outputs a data frame that includes an indication of the amount of EMR accumulated by the pixel array during the previous accumulation or blanking cycle.

[0113] Each frame period in the sensor cycle can be adjusted frame-by-frame to optimize the image sensor output and compensate for the pixel array 125, which has varying degrees of sensitivity to EMR at different bands. The duration of each blanking period can be shortened or lengthened to customize the amount of EMR that the pixel array 125 can accumulate. Additionally, the duration of each readout period can be shortened or lengthened by implementing a pixel-merging configuration or by having the image sensor read out each pixel within the pixel array 125. Therefore, since the sensor cycle includes a variable frame rate, the image sensor 124 can output data frames at an irregular rate. System 300 includes a memory buffer 352 that receives data frames from the image sensor 124. The memory buffer 352 stores the data frames and then outputs each data frame to the image signal processor 140 at a regular rate. This allows the image signal processor 140 to process each data frame sequentially at a regular rate.

[0114] Figure 4 This is a schematic diagram of an illumination system 400 for illuminating a dark environment 406 (such as inside a body cavity). In most cases, the transmitter 102 is the only illumination source within the dark environment 406, causing the pixel array of the image sensor to not detect any ambient light. The transmitter 102 includes multiple individual and independently actuated EMR sources, which may include a visible light source 304, a multispectral source 306, a fluorescence source 308, and a mapped pattern source 310. The transmitter can cyclically turn selected sources on and off to pulse them according to a variable pulse period received from the controller 104. Each of the EMR sources is fed into a collection region 404 of the transmitter 102. The collection region 404 can then be fed into a waveguide (see, for example...). Figure 1A In 130), the waveguide transmits pulsed EMR to the distal end of the endoscope within the light-deficient environment 406.

[0115] The variable pulse period is customizable and can be adjusted in real time based on user input. Transmitter 102 can instruct the individual EMR sources to pulse in any order. Additionally, transmitter 102 can adjust one or more of the duration or intensity of each pulse of the EMR. The variable pulse period can be optimized to adequately illuminate the low-light environment 406, ensuring that the resulting data frames read from pixel array 125 are within the desired exposure range (i.e., the frames are neither underexposed nor overexposed). The desired exposure range can be determined based on user input, the requirements of image signal processor 140, and / or the requirements of a specific image processing algorithm (see [link to image processing algorithm]). Figure 3B (344, 346, 348, and 350 in the diagram). Sufficient illumination of the light-deficient environment 406 depends on the energy output of each EMR source and also on the efficiency of the pixel array 125 used for sensing the different bands of the EMR.

[0116] Figures 5-7 This is a schematic diagram illustrating the synchronized operation cycles of 500, 600, and 700 for the image sensor and transmitter. Operation cycles 500, 600, and 700 can be specifically implemented when a user simultaneously requests a color video stream and one or more types of advanced visualization. The user can request the display of a real-time color video stream, which includes additional information obtained through one or more of multispectral visualization, fluorescence visualization, morphological mapping, or anatomical measurement scans. Advanced visualization data can be processed by one or more algorithms, such as combining... Figure 3B The algorithms discussed are 346, 348, and 350.

[0117] Figures 5-7 Operation cycles 500, 600, and 700, implemented to acquire color data frames and "advanced" data frames, are shown, wherein the advanced data frames include any of the multispectral data frame 207, fluorescence data frame 209, or mapping data frame 211 as described herein. Figures 5-7 In each of these, the rolling readout sequence of the image sensor 124 is represented by diagonals 508, 608, and 708. Additionally, the illumination period of the transmitter 102 is represented by gray areas for the visible light EMRs 502, 602, and 702, and by diagonal patterns for the advanced EMRs 504, 604, and 704. Similarly, the resulting data frames are represented such that color data frames are represented by gray boxes 503, 603, and 703, and advanced data frames include diagonal patterns 505, 605, and 705.

[0118] Color data frames 503, 603, and 703 are sensed by image sensor 124 in response to pulsed visible light EMRs 502, 602, and 702 (which may specifically include white light). Advanced data frames 505, 605, and 705 are sensed by image sensor 124 in response to the emission of “advanced” EMRs 504, 604, and 704 by transmitter 102. These advanced data frames may specifically include any of the multispectral 206, fluorescence 208, or mapping 210 emissions as described herein. Advanced EMRs 504, 604, and 704 may include EMRs in specific bands for multispectral or fluorescence visualization, or may include EMRs diffracted into mapping patterns for one or more of the following: topography, anatomical measurement, tool tracking, etc.

[0119] In some advanced visualization schemes described herein, advanced imaging signals are superimposed on a color image. To achieve this, system 100 acquires both a color image and an advanced image. Image sensor 124 reads out a color data frame 205 in response to accumulated reflected white light. If the advanced image is a multispectral image, image sensor 124 reads out a multispectral data frame 207 in response to accumulated spectral response emitted by tissue and / or accumulated reflected EMR within multispectral bands. If the advanced image is a fluorescence image, image sensor 124 reads out a fluorescence data frame 209 in response to accumulated fluorescence relaxation wavelength emitted by tissue or reagent and / or accumulated reflected EMR within fluorescence excitation bands. In some specific implementations, the EMR accumulated for the advanced image (i.e., spectral response, reflected multispectral EMR, fluorescence relaxation emission, or reflected fluorescence excitation EMR) is relatively dark compared to the white light emission used for the color data frame. A controller (which may include controller 104 or a component of MCU 122) may adjust the pulse period of emitter 102 to compensate for the relative darkness of the EMR for advanced visualization.

[0120] The controller (which may include a component of controller 104 or MCU 122) selects the operating cycles of 500, 600, and 700 based on the relative "intensity" of the visible light EMR and the advanced EMR. In some cases, the visible light EMR source is considered "stronger" based on the inherent characteristics of the pixel array. For example, if the pixel array is inherently more sensitive to detecting EMR emitted by the visible light EMR source, then the visible light EMR source can be classified as "stronger" when compared to the advanced EMR source. Conversely, if the pixel array is inherently less sensitive to detecting EMR emitted by the advanced EMR source, then the advanced EMR source can be classified as "weaker" when compared to the visible light EMR source. It should be understood that different image sensors manufactured by different manufacturers or produced to different specifications may have different inherent sensitivities to different regions of the electromagnetic spectrum. Therefore, the relative "weaker" and "stronger" classification of different EMR sources may vary depending on the specifications of the image sensor used. In addition, a “stronger” EMR source may have a higher amplitude, greater brightness, or higher energy output compared to a “weaker” EMR source.

[0121] The controller can adjust the pulse period of the emitter 102 in real time based on the relative intensities of different EMR sources and further based on the exposure levels of the resulting data frames. In an example implementation, the user requests color imaging and fluorescence imaging. The controller determines that the fluorescence source is relatively weak compared to a white light source. The fluorescence source may be relatively weak due to the inherent characteristics of the pixel array. For example, the pixel array may be insensitive to detecting fluorescence excitation wavelengths emitted by the emitter or fluorescence relaxation wavelengths emitted by fluorescent tissues or reagents within the scene. Additionally, the fluorescence source may be relatively weak because fluorescence emission has lower amplitude, lower brightness, and / or lower energy output compared to a white light source. In response to determining that the fluorescence source is "weak" compared to a white light source, the controller can select one of an operating cycle 600 or 700 to extend the illumination time of the fluorescence source. The controller can adjust and optimize the operating cycles 600 and 700 in real time to ensure that all resulting data frames are adequately exposed.

[0122] Image sensor 124 typically employs a rolling shutter, where different rows on pixel array 125 are exposed sequentially, rather than simultaneously. In this configuration, when image sensor 124 reads out the last pixel of the first data frame, some pixels in pixel array 125 have already begun accumulating EMR for the second data frame. In some implementations, transmitter 102 does not illuminate the scene during the readout period of the pixel array and only illuminates the scene during the blanking period of the image sensor, which may be referred to in some cases as a "co-exposure" period. The blanking period of image sensor 124 (during which pixel array 125 accumulates EMR) can have a short duration. In some cases, pixel array 125 may not accumulate sufficient EMR for advanced visualization during the standard blanking period.

[0123] Figure 5 This is a schematic diagram of a synchronized operation cycle 500 that can be implemented when the visible light EMR 502 and the advanced EMR 504 have similar brightness levels, or when the pixel array 125 is able to accumulate sufficient visible light EMR 502 and advanced EMR 504 during the standard blanking cycle 506 of the image sensor. In an example embodiment, operation cycle 500 is selected when the advanced EMR 504 comprises an EMR that is divided into a mapping pattern for topographic mapping, anatomical measurement, tool tracking, etc. In another example embodiment, operation cycle 500 is selected when the advanced EMR 504 comprises an EMR band that has a similar brightness level to the visible light EMR 502, selected for multispectral or fluorescence visualization.

[0124] During operation cycle 500, emitter 102 pulses visible light EMR 502, and then image sensor 124 reads out and outputs color data frame 503 according to the rolling readout 508 scheme (see diagonal 508 representing the rolling readout of pixel array 125). Emitter 102 pulses advanced EMR 504 (which may include any of pulsed multispectral 206, pulsed fluorescence 208, or pulsed mapping 210 as described herein), and then image sensor 124 reads out and outputs advanced data frame 505 according to the rolling readout 508 scheme. Advanced data frame 505 may include any of multispectral data frame 207, fluorescence data frame 209, or mapping data frame 211 as described herein. Operation cycle 500 limits energy accumulation to the blanking cycle 506 of image sensor 124 such that pixel array 125 does not accumulate EMR while image sensor is performing the rolling readout 506 scheme.

[0125] Operation cycle 500 produces the “cleanest” data frames, where all pixels within pixel array 125 are exposed to EMR for the same duration because pixel array 125 accumulates EMR only during blanking cycle 506. The resulting data frames 503, 505 may be free of image artifacts caused by some pixels accumulating EMR for a longer duration than others. However, operation cycle 500 may be insufficient if pixel array 125 cannot accumulate sufficient advanced EMR 504 during blanking cycle 506. Operation cycle 500 leaves a significant portion of the image sensor's available exposure time unused.

[0126] Figure 6 This is a schematic diagram of a synchronous operating cycle 600 that can be implemented when the visible light EMR 602 and the advanced EMR 604 have different brightness levels, or when the pixel array 125 does not accumulate sufficient advanced EMR 604 during a single blanking cycle 606. In an example implementation, operating cycle 600 is selected when the advanced EMR 604 includes an EMR band selected for fluorescence visualization, which is relatively darker compared to the visible light EMR 602. Operating cycle 600 can be specifically implemented when the advanced EMR 604 includes a near-infrared (NIR) band of the electromagnetic spectrum.

[0127] During operation cycle 600, transmitter 102 pulses the advanced EMR 604 for a longer duration, allowing pixel array 125 to accumulate the advanced EMR 604 over a longer period. This attempts to compensate for the brightness difference between the visible light EMR 602 and the advanced EMR 604 to ensure that each of the color data frame 603 and the advanced data frame 605 includes sufficient data. Figure 6 In the high-level EMR 604, the transmission is represented by an area including a diagonal pattern, and the resulting high-level data frame 605 is represented by the same pattern. For example... Figure 6 As shown, transmitter 102 continuously pulses the advanced EMR 604 during the entire rolling readout 608 of color data frame 603, the blanking period 606 between color data frame 603 and advanced data frame 605, and also during the rolling readout 608 of advanced data frame 605. When pulsed visible light EMR 602, advanced EMR 604 is turned off only during blanking period 606.

[0128] The operating cycle 600 enables the pixel array 125 to accumulate advanced EMR 604 over a longer period of time. This may be important if the pixel array 125 is inherently inefficient in accumulating advanced EMR 604 and / or if advanced EMR 604 is relatively dark compared to visible light EMR 602.

[0129] During operation cycle 600, color data frame 603 is partially exposed to the advanced EMR 604 at the point where data frames are superimposed. Therefore, the resulting color data frame 603 includes the sum of accumulated energy from some of the visible light EMR 602 and the advanced EMR 604. However, for several reasons, this "overflow" from the advanced EMR 604 to the color data frame 603 does not affect the integrity of the final superposition. That is, the advanced EMR 604 signal is typically very weak compared to the visible light EMR 602, and therefore its contribution to the color data frame 603 is negligible in most cases. Furthermore, when the advanced EMR 604 signal is high or comparable to the visible light EMR 602 signal, the effect is masked by the superimposed display when color data frame 603 and advanced data frame 605 are combined. Additionally, further image processing can be used to separate the two components (i.e., the reflected signal from the visible light EMR 602 and the signal from the advanced EMR 604).

[0130] Figure 7 This is a schematic diagram of a synchronous operating cycle 700 that can be implemented when the visible light EMR 702 and the advanced EMR 704 have different brightness levels, or when the pixel array 125 does not accumulate sufficient advanced EMR 704 during a single blanking cycle 706. In an example specific implementation, the operating cycle 700 is selected when the advanced EMR 704 includes an EMR band selected for fluorescence visualization, which is relatively dark compared to the visible light EMR 702. The operating cycle 700 can be specifically implemented when the advanced EMR 704 includes a near-infrared (NIR) band of the electromagnetic spectrum.

[0131] During operation cycle 700, transmitter 102 continuously transmits Advanced EMR 704 instead of cyclically turning Advanced EMR 704 on and off. Transmitter 102 transmits Advanced EMR 702 during the rolling readout 708 of color data frame 703, during all blanking cycles 706, and during the rolling readout 708 of Advanced data frame 705.

[0132] Operating cycles of 600 and 700 allow for significantly longer exposure times for advanced data frames. This greatly increases the sensitivity of advanced data frames and effectively utilizes the image sensor's exposure, as the entire rolling readout shutter cycle is used to accumulate EMR, rather than just during the blanking cycle.

[0133] Data frames obtained from any of operation cycles 500, 600, and 700 can be overlaid to output a color image combined with advanced visualization data. In some cases, multispectral or fluorescence visualization is used to identify the location of tissues or regions within a scene, and in these cases, the identified tissues or regions can be displayed in pseudocolor as an overlay on a color image depicting the scene. In an example embodiment, a color video stream of the scene is presented to the user, and desired tissue structures (e.g., cancer cells, nerve tissue, blood, etc.) are highlighted in a monochrome that can be selected by the user. In another embodiment, a color video stream of the scene is presented to the user, and desired tissue structures are highlighted using an intensity map indicating the relative probability of their location. The intensity map is similar to a temperature-based thermogram but will instead indicate the relative intensity of spectral or fluorescence signals accumulated by the pixel array 125.

[0134] When the advanced EMR 604 and 704 are fluorescence excitation wavelengths in the infrared or near-infrared range of the electromagnetic spectrum, operating cycles of 600 and 700 can be specifically implemented. Operating cycles of 600 and 700 can also be deployed when the pulses are in the visible and long-red bands of the electromagnetic spectrum. Infrared and near-infrared EMRs are darker than white light, therefore image sensors may struggle to accumulate sufficient energy to read out fluorescence data frames in response to exposure to fluorescence excitation wavelengths.

[0135] The controller optimizes the pulse period of transmitter 102 and the sensor period of image sensor 124 based on user input, pulse type (e.g., white light, multispectral band, fluorescence band, or EMR divided into mapping patterns), desired exposure of the resulting data frame, desired frame rate, and acceptable resolution of the resulting data frame. The systems, methods, and apparatuses described herein interweave color video data streams with one or more advanced visualization data streams. Each of these data streams can be sensed by a single image sensor 124, multiple image sensors 124 acting in concert, or multiple image sensors 124 acting sequentially.

[0136] The controller ensures that each resulting data frame includes exposure (brightness) levels within a threshold range. This threshold exposure range can vary based on the frame type. For example, a color data frame might have a different threshold exposure range than a high-level data frame because the color data frame will be viewed by the user on the monitor, while the high-level data frame might be evaluated by an algorithm (see, for example, combining...). Figure 3B The algorithms discussed are 346, 348, and 350. Various algorithms 346, 348, and 350 may have different exposure requirements to successfully make a determination based on the data frame. Therefore, the controller may need to extend the illumination time of the advanced data frame to compensate for the relative weakness of the advanced EMR source and further ensure that the resulting data frame can be evaluated by the appropriate algorithm.

[0137] The controller can adjust the duration of the readout and blanking cycles in real time based on user input and the resulting data frames processed through the feedback loop. For example, if the controller determines that a previous multispectral data frame cannot be evaluated using the Spectral Processing 344 algorithm due to underexposure of the multispectral data frame, the controller can extend the multispectral illumination time of subsequent multispectral frames. The controller can switch between operating cycles of 500, 600, and 700 based on the EMR source used, user feedback, and the resulting data frames. The controller can generate a customized operating cycle that includes components of any of the operating cycles 500, 600, and 700.

[0138] Figure 8A and Figure 8B A schematic diagram of a system 800 for emitting EMR in a mapping pattern 210 is shown. Emitter 102 can pulse mapping pattern 210 using a low-mode laser that is diffracted to generate a suitable mapping pattern 210. Mapping pattern 210 reflects from the tissue in a manner dependent on the wavelength of the EMR and the specific anatomical features of the tissue.

[0139] like Figure 8A The example mapping pattern 210 shown is a grid array including vertical hash 802 and horizontal hash 804. Figure 8B The example mapping pattern 210 shown is a dot array. Mapping pattern 210 may include any suitable array for mapping a surface, including, for example, a raster grid of discrete points, an occupied grid mapping map, a dot array, a vertical hash, a horizontal hash, etc. Mapping pattern 210 is emitted by an illumination source 808, which may originate from an EMR source within transmitter 102 and terminate at endoscope 110 or tool 108.

[0140] Such as combination Figures 1A-1C As discussed, the distal end of endoscope 110 may include one or more waveguides 130 that emit EMR originating from transmitter 102. Mapping pattern 210 may be emitted from these waveguides 130 and projected onto the surface of tissue. Additionally, one or more tools 108 within the scene may include waveguides 131 terminating at the distal end of tool 108 and / or on one side of tool 108, such as... Figure 1A As shown. The waveguide 131 can also project the mapping pattern 210 onto the tissue surface within the scene. In some cases, the tool 108 and the endoscope 110 jointly project the mapping pattern 210, and the resulting data frames captured by the image sensor 124 include data for tracking the position of the tool 108 within the scene relative to the endoscope 110 or other tools 108.

[0141] The transmitter 102 can map EMR pulses of any suitable wavelength, including, for example, ultraviolet, visible, and / or infrared or near-infrared light. It can map and track surfaces and / or objects in the environment with very high resolution and with very high accuracy and precision.

[0142] A mapping pattern 210 is selected for the desired anatomical measurement scheme, such as 3D topographic mapping, measuring distances and dimensions within the scene, tracking the relative position of tool 108 within the scene, etc. Image sensor 124 detects reflected EMR and outputs a mapping data frame 211 in response to transmitter 102 pulse mapping pattern 210. The resulting mapping data frame 211 is provided to the topographic processing algorithm 350, which is trained to calculate one or more of the following: a 3D topographic map of the scene, the distance between two or more objects within the scene, the dimensions of objects within the scene, the relative distance between a tool and another object within the scene, etc.

[0143] The transmitter 102 can pulse the mapping pattern 210 at a sufficient speed to render it invisible to the user. In various implementations, seeing the mapping pattern 210 during endoscopic visualization procedures can be distracting for the user. In some cases, the rendering of the mapping pattern 210 can be overlaid on a color video stream to provide more context to the user of the visualized scene. Users can also request to view real-time measurements and proximity alerts of objects within the scene as the tool approaches critical structures such as blood vessels, nerve fibers, or cancerous tissue. The accuracy of the measurements can be less than one millimeter.

[0144] Figure 9 A portion of an electromagnetic spectrum 900 divided into twenty different bands is shown. The number of bands is merely exemplary. In at least one embodiment, the spectrum 900 may be divided into hundreds of bands. The spectrum 900 may extend from the infrared spectrum 902, through the visible spectrum 904, and into the ultraviolet spectrum 906. Each band may be defined by an upper wavelength and a lower wavelength.

[0145] Multispectral imaging includes imaging information from across the electromagnetic spectrum 900. The EMR multispectral pulse may include multiple sub-pulses across one or more portions of the electromagnetic spectrum 900 or the entire electromagnetic spectrum 900. The EMR multispectral pulse may include a single partition of the EMR wavelength. The resulting multispectral data frame includes information sensed by the pixel array following the EMR multispectral pulse. Therefore, a multispectral data frame may include data from any suitable partition of the electromagnetic spectrum 900, and may include multiple data frames from multiple partitions of the electromagnetic spectrum 900.

[0146] Depending on the specific implementation, transmitter 102 may include any number of multispectral EMR sources as needed. In one embodiment, each multispectral EMR source covers a spectrum spanning 40 nanometers. For example, one multispectral EMR source may emit EMR in the 500 nm to 540 nm band, while another may emit EMR in the 540 nm to 580 nm band. In another embodiment, multispectral EMR sources may cover bands of other sizes, depending on the type of EMR sources available or the imaging requirements. Each multispectral EMR source may cover different segments of the electromagnetic spectrum 900, ranging from far-infrared, mid-infrared, near-infrared, visible, near-ultraviolet, and / or far-ultraviolet. In some cases, multiple multispectral EMR sources of the same type or wavelength may be included to provide sufficient output power for imaging. The number of multispectral EMR sources required for a particular band may depend on the sensitivity of pixel array 125 to the band and / or the power output capability of the EMR sources in that band.

[0147] The bandwidth and coverage provided by the EMR source can be selected to provide any desired combination of spectra. For example, continuous coverage of the spectrum 900 using a very small bandwidth (e.g., 10 nm or less) allows for highly selective multispectral and / or fluorescence imaging. This bandwidth allows for the selective emission of excitation wavelengths of one or more specific fluorescent reagents. Additionally, this bandwidth allows for the selective emission of certain zones of the multispectral EMR for the identification of specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths originate from a selectively activatable EMR source, extreme flexibility is achieved in fluorescing one or more specific fluorescent reagents during examination. Furthermore, extreme flexibility is achieved in identifying one or more objects or processes through multispectral imaging. Therefore, more fluorescence and / or multispectral information can be obtained in less time and in a single examination, which would otherwise require multiple examinations and be delayed due to the application of dyes or staining agents.

[0148] Figure 10 This is a schematic diagram illustrating the timing of emission and readout for generating an image. Solid lines represent the readout (peak 1002) and blanking periods (valleys) for capturing a series of exposure frames 1004-1014. The series of data frames 1004-1014 may include a series of repeating data frames that can be used to generate calibration, multispectral, and / or fluorescence data that can be superimposed on an RGB video stream. This series of data frames includes a first data frame 1004, a second data frame 1006, a third data frame 1008, a fourth data frame 1010, a fifth data frame 1012, and an Nth data frame 1026.

[0149] In one implementation, each data frame is generated based on at least one pulse of EMR. The EMR pulse is reflected and detected by the pixel array 125 and subsequently read out in a subsequent readout (1002). Thus, each blanking period and readout produces a data frame for a specific EMR band. For example, a first data frame 1004 may be generated based on the band of a first one or more pulses 1016, a second data frame 1006 may be generated based on the band of a second one or more pulses 1018, a third data frame 1008 may be generated based on the band of a third one or more pulses 1020, a fourth data frame 1010 may be generated based on the band of a fourth one or more pulses 1022, a fifth data frame 1012 may be generated based on the band of a fifth one or more pulses 1024, and an Nth data frame 1026 may be generated based on the band of an Nth one or more pulses 1026.

[0150] Pulses 1016-1026 may include energy from a single EMR source or a combination of two or more EMR sources. For example, bands may be selected to be included in a single readout cycle or within multiple data frames 1004-1014 for desired examination or detection of a specific tissue or condition. According to one embodiment, one or more pulses may include visible spectral light for generating RGB or black-and-white images, while one or more additional pulses are emitted to sense the spectral response of the EMR for multispectral wavelengths.

[0151] Pulses 1016-1026 are emitted according to a variable pulse period determined by controller 104. For example, pulse 1016 may include white light, pulse 1018 may include a multispectral band, pulse 1020 may include white light, pulse 1022 may include a fluorescence band, pulse 1024 may include white light, and so on.

[0152] Multiple frames 1004-1014 are shown as readout periods of different lengths and pulses of different lengths or intensities. The blanking period, pulse length, or intensity, etc., can be selected based on the sensitivity of the monochromatic sensor to a specific wavelength, the power output capability of the EMR source, and / or the carrying capacity of the waveguide.

[0153] In one implementation, dual image sensors can be used to acquire three-dimensional images or video feeds. Three-dimensional inspection allows for a better understanding of the three-dimensional structure of the inspected area and the mapping of different tissue or material types within that area.

[0154] In one example implementation, an endoscopic imaging system is used to image a patient to identify quantitative diagnostic information about the patient's histopathology. In this example, the patient is suspected of or known to have a disease that can be tracked using multispectral imaging to observe the progression of the disease within the patient's tissues. The endoscopic imaging system pulses white light to generate an RGB video stream inside the patient's body. Additionally, one or more multispectral bands of the pulsed light from the endoscopic imaging system allow the system to "see through" some tissue and generate images of the tissue affected by the disease. The endoscopic imaging system senses reflected multispectral EMR to generate multispectral imaging data of the diseased tissue, thereby identifying the location of the diseased tissue within the patient's body. The endoscopic imaging system can also emit a mapping pulse scheme to generate a three-dimensional topographic map of the scene and calculate the dimensions of objects within the scene. The location of the diseased tissue (as identified by the multispectral imaging data) can be combined with the topographic map and size information calculated using the mapping data. Therefore, the precise location, size, dimensions, and morphology of the diseased tissue can be identified. This information can be provided to a physician to assist in the removal, imaging, or study of the diseased tissue. Additionally, this information can be provided to robotic surgical systems so that they can remove diseased tissue.

[0155] Figure 11 A schematic block diagram of an example computing device 1100 is shown. The computing device 1100 can be used to execute various programs, such as those discussed herein. The computing device 1100 can perform various monitoring functions discussed herein, and can execute one or more application programs, such as the applications or functions described herein. The computing device 1100 can be any of a variety of computing devices, such as a desktop computer, an internal computer, a vehicle control system, a laptop computer, a server computer, a handheld computer, a camera controller, a tablet computer, etc.

[0156] The computing device 1100 includes one or more processors 1104, one or more memory devices 1104, one or more interfaces 1106, one or more mass storage devices 1108, one or more input / output (I / O) devices 1110, and a display device 1130, all of which are coupled to a bus 1112. The processor 1104 includes one or more processors or controllers that execute instructions stored in the memory devices 1104 and / or mass storage devices 1108. The processor 1104 may also include several types of computer-readable media, such as cache memory.

[0157] The memory device 1104 includes various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 1114) and / or non-volatile memory (e.g., read-only memory (ROM) 1116). The memory device 1104 may also include rewritable ROM, such as flash memory.

[0158] Mass storage devices 1108 include various computer-readable media, such as magnetic tape, magnetic disks, optical disks, solid-state storage (e.g., flash memory), etc. Figure 11 As shown, a specific mass storage device 1108 is a hard disk drive 1124. Various drives may also be included in the mass storage device 1108 to enable reading and / or writing from various computer-readable media. The mass storage device 1108 includes removable media 1126 and / or non-removable media.

[0159] I / O device 1110 includes various means that allow input to or retrieval of data and / or other information to or from computing device 1100. Example I / O device 1110 includes cursor control device, keyboard, keypad, microphone, monitor or other display device, speaker, printer, network interface card, modem, etc.

[0160] Display device 1130 includes any type of device capable of displaying information to one or more users of computing device 1100. Examples of display device 1130 include monitors, display terminals, video projection devices, etc.

[0161] Interface 1106 includes various interfaces that allow computing device 1100 to interact with other systems, devices, or computing environments. Example interface 1106 may include any number of different network interfaces 1120, such as interfaces for connecting to a local area network (LAN), wide area network (WAN), wireless network, and the Internet. Other interfaces include user interface 1118 and peripheral device interface 1122. Interface 1106 may also include one or more user interface elements 1118. Interface 1106 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mouse, touchpad, or any suitable user interface now known or hereafter discovered by a person skilled in the art), keyboards, etc.

[0162] Bus 1112 enables processor 1104, memory device 1104, interface 1106, mass storage device 1108, and I / O device 1110 to communicate with each other and with other devices or components coupled to bus 1112. Bus 1112 represents one or more of several types of bus architectures (such as system bus, PCI bus, IEEE bus, USB bus, etc.).

[0163] For illustrative purposes, the programs and other executable program components shown herein are discrete blocks such as, for example, block 302; however, it should be understood that such programs and components may reside at various times in different storage units of the computing device 1100 and be executed by the processor 1102. Alternatively, the systems and programs (including programs or other executable program components) described herein may be implemented in hardware, or by a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) may be programmed to execute one or more systems and programs described herein.

[0164] Example

[0165] The following embodiments relate to preferred features of other implementations:

[0166] Example 1 is a system. The system includes an image sensor comprising a pixel array. The system includes a transmitter comprising a plurality of electromagnetic radiation (EMR) sources, including a stronger EMR source and a weaker EMR source. The pixel array is relatively inefficient in accumulating EMR emitted by the weaker EMR source compared to EMR emitted by the stronger EMR source. The transmitter cycles at least a portion of the plurality of EMR sources according to a pulse period comprising multiple pulse emissions from the stronger EMR source and at least one emission from the weaker EMR source. When the image sensor reads out a data frame corresponding to an emission from the stronger EMR source, at least one emission from the weaker EMR source overlaps with the readout period of the image sensor.

[0167] Example 2 is the system according to Example 1, wherein the weaker EMR source emits EMR in one or more of the following bands: the near-infrared band of the electromagnetic spectrum; or the infrared band of the electromagnetic spectrum.

[0168] Example 3 is a system according to any one of Examples 1-2, wherein the stronger EMR source emits one or more of the following: white light; or a partition of EMR from the visible band of the electromagnetic spectrum.

[0169] Example 4 is a system according to any one of Examples 1-3, wherein the EMR emitted by the weaker EMR source has a shorter amplitude compared to the EMR emitted by the stronger EMR source.

[0170] Example 5 is a system according to any one of Examples 1-4, wherein the EMR emitted by the weaker EMR source has less energy compared to the EMR emitted by the stronger EMR source.

[0171] Example 6 is a system according to any one of Examples 1-5, wherein the EMR emitted by the weaker EMR source is darker than the EMR emitted by the stronger EMR source.

[0172] Example 7 is a system according to any one of Examples 1-6, wherein the stronger EMR source and the weaker EMR source emit different bands of the electromagnetic spectrum, and wherein the pixel array is inherently less efficient in accumulating the bands of EMR emitted by the weaker EMR source compared to the bands of EMR emitted by the stronger EMR source.

[0173] Example 8 is a system according to any one of Examples 1-7, wherein the stronger EMR source is a white light source, and wherein the data frame corresponding to the emission of the stronger EMR source is a color data frame.

[0174] Example 9 is a system according to any one of Examples 1-8, wherein the weaker EMR source is a fluorescence source, the fluorescence source is configured to emit an EMR at a fluorescence excitation wavelength, and wherein: the fluorescence excitation wavelength of the EMR is selected to cause one or more of the reagents or tissues to fluoresce; the pixel array accumulates the fluorescence relaxation wavelength of the EMR emitted by one or more of the reagents or tissues; and the image sensor reads out fluorescence data frames during a rolling readout sequence assigned to the weaker EMR source.

[0175] Example 10 is a system according to any one of Examples 1-9, wherein the stronger EMR source is cyclically turned off during the blanking period of the image sensor immediately preceding the rolling readout sequence assigned to the weaker EMR source.

[0176] Example 11 is a system according to any one of Examples 1-10, wherein the image sensor reads out color data frames and advanced data frames, and wherein: the pixel array accumulates EMR generated by the emission of the stronger EMR source to output the color data frames; and the pixel array accumulates EMR generated by the emission of the weaker EMR source to output the advanced data frames.

[0177] Example 12 is a system according to any one of Examples 1-11, wherein the advanced data frame includes one or more of the following: a fluorescence data frame corresponding to the pixel array, wherein the fluorescence data frame accumulates one or more of the fluorescence excitation wavelength or fluorescence relaxation wavelength of the EMR; or a multispectral data frame corresponding to the pixel array, wherein the multispectral data frame accumulates one or more of the multispectral wavelength or spectral response emission of the EMR.

[0178] Example 13 is a system according to any one of Examples 1-12, wherein the pulse period causes the stronger EMR source to be cyclically turned on during a blanking period immediately preceding the readout of the color data frame; and wherein the stronger EMR source is cyclically turned off during a blanking period immediately preceding the readout of the high-level data frame.

[0179] Example 14 is a system according to any one of Examples 1-13, wherein the pulse cycle causes the weaker EMR source to be cyclically turned on during each of the following periods: the blanking period immediately preceding the readout of the high-level data frame; and the rolling readout sequence when the image sensor is reading out the color data frame.

[0180] Example 15 is a system according to any one of Examples 1-14, wherein the pulse cycle causes the weaker EMR source to be further cycled on during the rolling readout sequence while the image sensor is reading out the high-level data frame.

[0181] Example 16 is a system according to any one of Examples 1-15, wherein the pulse period causes the weaker EMR source to be cyclically turned on during each of the following periods: a blanking period immediately preceding the readout of the color data frame; and a blanking period immediately preceding the readout of the high-level data frame.

[0182] Example 17 is a system according to any one of Examples 1-16, wherein the pulse period causes: the stronger EMR source to be cyclically turned on during a blanking period immediately preceding the readout of the color data frame; the stronger EMR source to be cyclically turned off during a rolling readout sequence while the image sensor is reading out the color data frame; the stronger EMR source to be cyclically turned off during a blanking period immediately preceding the readout of the advanced data frame; and the stronger EMR source to be cyclically turned off during a rolling readout sequence while the image sensor is reading out the advanced data frame.

[0183] Example 18 is a system according to any one of Examples 1-17, wherein the pulse period causes the weaker EMR source to be cyclically turned off during the blanking period immediately preceding the readout of the color data frame; the weaker EMR source to be cyclically turned on during the rolling readout sequence while the image sensor is reading out the color data frame; the weaker EMR source to be cyclically turned on during the blanking period immediately preceding the readout of the advanced data frame; and the weaker EMR source to be cyclically turned on during the rolling readout sequence while the image sensor is reading out the advanced data frame.

[0184] Example 19 is a system according to any one of Examples 1-18, wherein the pulse period causes the weaker EMR source to be continuously and cyclically turned on.

[0185] Example 20 is a system according to any one of Examples 1-19, wherein information from the color data frame and the advanced data frame is combined to generate an overlay frame, wherein the overlay frame includes a color image depicting a scene and also includes pseudo-color overlay depiction information determined based on the advanced data frame.

[0186] Example 21 is a system. The system includes an image sensor comprising a pixel array. The system includes a transmitter having a plurality of electromagnetic radiation sources, wherein the plurality of electromagnetic radiation sources includes a first source and a second source. The transmitter cycles through at least a portion of the plurality of electromagnetic radiation sources according to a pulse period, the pulse period including a plurality of pulse emissions by the first source and a plurality of pulse emissions by the second source. The system is such that when the image sensor reads out a data frame corresponding to an emission from the first source, at least a portion of the plurality of pulse emissions from the second source overlaps with the readout period of the image sensor. The system is such that the second source is cyclically turned off during at least a portion of the plurality of pulse emissions from the first source.

[0187] Example 22 is a system according to Example 21, wherein the first source is a visible light source that pulses electromagnetic radiation in the visible band of the electromagnetic spectrum. The data frame corresponding to the emission of the first source is a color data frame; wherein, when the transmitter pulses the visible light source, the pixel array accumulates reflected electromagnetic radiation during the blanking period; and wherein, when the transmitter pulses the visible light source, the image sensor reads out the color data frame during a readout period immediately following the blanking period.

[0188] Example 23 is a system according to any one of Examples 21-22, wherein the second source is an excitation source configured to emit electromagnetic radiation of a fluorescence excitation wavelength; wherein the fluorescence excitation wavelength of the electromagnetic radiation is selected to cause one or more of the reagents or tissues to fluoresce; wherein the pixel array accumulates the fluorescence relaxation wavelength of the electromagnetic radiation emitted by one or more of the reagents or tissues; and wherein the image sensor reads out fluorescence data frames during a readout period corresponding to the second source.

[0189] Example 24 is a system according to any one of Examples 21-23, wherein the second source is cyclically activated during each of the following periods: the readout period when the image sensor reads out the color data frame; and the blanking period immediately following the readout of the color data frame by the image sensor.

[0190] Example 25 is a system according to any one of Examples 21-24, wherein the second source is further cyclically activated during a readout period corresponding to the second source, wherein the image sensor reads out the fluorescence data frame; and wherein the second source is continuously cyclically activated throughout the following periods: the readout period when the image sensor reads out the color data frame; the blanking period immediately following the image sensor reading out the color data frame; and the readout period in which the image sensor reads out the fluorescence data frame.

[0191] Example 26 is a system according to any one of Examples 21-25, wherein the transmitter cyclically shuts off the first source during each of the following periods: the readout period when the image sensor reads out the color data frame; the blanking period immediately following the image sensor reading out the color data frame; and the readout period in which the image sensor reads out the fluorescence data frame.

[0192] Example 27 is a system according to any one of Examples 21-26, wherein the transmitter cyclically turns the first source and the second source on and off according to the pulse period, such that the plurality of pulse emissions of the first source do not overlap with the plurality of pulse emissions of the second source.

[0193] Example 28 is a system according to any one of Examples 21-27, further comprising: an endoscope, wherein the image sensor is disposed in the distal region of the endoscope; and a prism disposed in the distal region of the endoscope, the prism being configured to reflect electromagnetic radiation onto the image sensor; wherein the planar side of the pixel array is oriented parallel to the longitudinal axis of the endoscope.

[0194] Example 29 is a system according to any one of Examples 21-28, wherein the pixel array is relatively inefficient in accumulating electromagnetic radiation emitted by the second source compared to electromagnetic radiation emitted by the first source, and wherein the second source emits electromagnetic radiation in one or more of the following bands: the near-infrared band of the electromagnetic spectrum; or the infrared band of the electromagnetic spectrum.

[0195] Example 30 is a system according to any one of Examples 21-29, wherein the pixel array is relatively inefficient in accumulating electromagnetic radiation emitted by the second source compared to electromagnetic radiation emitted by the first source, and wherein the first source emits in one or more of the following bands: white light; or a partition of electromagnetic radiation from the visible band of the electromagnetic spectrum.

[0196] Example 31 is a system according to any one of Examples 21-30, wherein the electromagnetic radiation emitted by the second source has a shorter amplitude compared to the electromagnetic radiation emitted by the first source.

[0197] Example 32 is a system according to any one of Examples 21-31, wherein the electromagnetic radiation emitted by the second source has less energy compared to the electromagnetic radiation emitted by the first source.

[0198] Example 33 is a system according to any one of Examples 21-32, wherein the electromagnetic radiation emitted by the second source is dimmer than the electromagnetic radiation emitted by the first source.

[0199] Example 34 is a system according to any one of Examples 21-33, wherein the first source and the second source emit different bands of the electromagnetic spectrum, and wherein the pixel array is inherently less efficient in accumulating the bands of electromagnetic radiation emitted by the second source compared to the bands of electromagnetic radiation emitted by the first source.

[0200] Example 35 is a system according to any one of Examples 21-34, wherein the image sensor reads out color data frames and advanced data frames, and wherein: the pixel array accumulates electromagnetic radiation generated by the emission of the first source to output the color data frames; and the pixel array accumulates electromagnetic radiation generated by the emission of the second source to output the advanced data frames.

[0201] Example 36 is a system according to any one of Examples 21-35, wherein the advanced data frame includes one or more of the following: a fluorescence data frame corresponding to the pixel array, wherein the fluorescence data frame accumulates one or more of the fluorescence excitation wavelength or the fluorescence relaxation wavelength of electromagnetic radiation; or a multispectral data frame corresponding to the pixel array, wherein the multispectral data frame accumulates one or more of the multispectral wavelength or the spectral response emission of electromagnetic radiation.

[0202] Example 37 is a system according to any one of Examples 21-36, wherein the pulse period causes the first source to be cyclically turned on during a blanking period immediately preceding the readout of the color data frame; the first source is cyclically turned off during a rolling readout sequence while the image sensor is reading out the color data frame; the first source is cyclically turned off during a blanking period immediately preceding the readout of the high-level data frame; and the first source is cyclically turned off during a rolling readout sequence while the image sensor is reading out the high-level data frame.

[0203] Example 38 is a system according to any one of Examples 21-37, wherein the pulse period causes: the second source to be cyclically turned off during the blanking period immediately preceding the readout of the color data frame; the second source to be cyclically turned on during the rolling readout sequence while the image sensor is reading out the color data frame; the second source to be cyclically turned on during the blanking period immediately preceding the readout of the high-level data frame; and the second source to be cyclically turned on during the rolling readout sequence while the image sensor is reading out the high-level data frame.

[0204] Example 39 is a system according to any one of Examples 21-38, wherein information from the color data frame and the advanced data frame is combined to generate an overlay frame, wherein the overlay frame includes a color image depicting a scene and also includes pseudo-color overlay depiction information determined based on the advanced data frame.

[0205] Example 40 is a system according to any one of Examples 21-39, wherein the color data frame further includes advanced visualization data generated by the transmitter cyclically activating the second source during a readout period when the image sensor reads out the color data frame; and wherein the system further includes a processor configured to subtract the advanced visualization data from the color data frame. Noise from the advanced data frame can be removed from the color frame using an algorithm configured to estimate the signal collected from each pixel of the advanced data frame during the readout period and then subtract that signal from the same pixel of the color data frame. For example, if the frame time consists of 60% blanking period and 40% readout period, 40% of the signal collected by each pixel during the advanced data frame can be subtracted from the corresponding pixel of the color data frame.

[0206] It should be understood that the various features disclosed herein offer significant advantages and advancements in the art. The following claims are examples of some of those features.

[0207] In the specific embodiments described above, for the purpose of simplification, the various features of this disclosure are concentrated in a single embodiment. The method of this disclosure should not be construed as implying an intention that the disclosure protected by the claims requires more features than expressly listed in each claim. Rather, the innovative aspects fail to embody all the features of the single embodiment disclosed above.

[0208] It should be understood that any feature of the above-described arrangement, embodiments, and implementations may be combined in a single implementation that includes a combination of features obtained from any of the disclosed arrangement, embodiments, and implementations.

[0209] It should be understood that the above-described configuration is merely an exemplary application of the principles of this disclosure. Many modifications and alternative configurations can be devised by those skilled in the art without departing from the spirit and scope of this disclosure, and the appended claims are intended to cover such modifications and configurations.

[0210] Therefore, when this disclosure is illustrated and described above with particularity and detail, it will be apparent to those skilled in the art that numerous modifications can be made without departing from the principles and ideas set forth herein, including but not limited to changes in size, material, shape, form, function and operation, assembly and use.

[0211] Additionally, where appropriate, the functions described herein may be performed by one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) may be programmed to execute one or more systems and programs described herein. Certain terms used throughout the following description and claims refer to specific system components. Those skilled in the art will understand that components may have different names. This document is not intended to distinguish between components that are different in name rather than function.

[0212] For illustrative and descriptive purposes, the specific embodiments described above have been provided. These specific embodiments are not intended to be exhaustive or to limit this disclosure to the specific forms disclosed. Many modifications and changes can be made to this disclosure based on the foregoing teachings. Furthermore, it should be noted that any or all of the foregoing alternative embodiments can be used in any desired combination to form further hybrid embodiments of this disclosure.

[0213] Furthermore, while specific embodiments of this disclosure have been described and illustrated, this disclosure is not limited to the particular forms or arrangements of components as described and illustrated. The scope of this disclosure will be defined by the appended claims, any future claims filed herein and in different applications, and their equivalents.

Claims

1. A system comprising: Image sensors including pixel arrays; as well as A transmitter comprising multiple electromagnetic radiation sources, wherein the multiple electromagnetic radiation sources include a first source and a second source; The transmitter cyclically transmits at least a portion of the plurality of electromagnetic radiation sources according to a pulse period, wherein the pulse period includes: Multiple pulses emitted by the first source; and Multiple pulses emitted by the second source; Wherein, when the image sensor reads out a data frame corresponding to the emission of the first source, at least a portion of the multiple pulse emissions from the second source overlaps with the readout period of the image sensor; and The second source is cyclically turned off during at least a portion of the multiple pulses emitted by the first source.

2. The system according to claim 1, wherein, The first source is a visible light source of pulsed electromagnetic radiation in the visible band of the electromagnetic spectrum; Among them, the data frame corresponding to the transmission of the first source is a color data frame; When the emitter pulses the visible light source, the pixel array accumulates reflected electromagnetic radiation during the blanking period; and When the transmitter pulses the visible light source, the image sensor reads out the color data frame during the readout period immediately following the blanking period.

3. The system according to claim 2, wherein, The second source is an excitation source configured to emit electromagnetic radiation at a fluorescence excitation wavelength; The fluorescence excitation wavelength of the electromagnetic radiation is selected to make one or more of the reagents or tissues fluoresce; The pixel array accumulates the fluorescence relaxation wavelengths of electromagnetic radiation emitted by one or more of the reagents or the tissue; and The image sensor reads out fluorescence data frames during the readout period corresponding to the second source.

4. The system according to claim 3, wherein, The second source is cyclically turned on during each of the following periods: The readout period when the image sensor reads out the color data frame; and The blanking period immediately following the image sensor reading out the color data frame.

5. The system according to claim 4, wherein, The second source is further cyclically activated during the readout cycle corresponding to the second source, wherein the image sensor reads out the fluorescence data frame; and The second source is continuously and cyclically activated throughout each of the following cycles: The readout period when the image sensor reads out the color data frame; The blanking period immediately following the image sensor reading out the color data frame; and The image sensor reads out the readout period of the fluorescence data frame.

6. The system according to claim 5, wherein, The transmitter cyclically shuts down the first source during each of the following cycles: The readout period when the image sensor reads out the color data frame; The blanking period immediately following the image sensor reading out the color data frame; and The image sensor reads out the readout period of the fluorescence data frame.

7. The system according to claim 1, wherein, The transmitter cyclically turns the first source and the second source on and off according to the pulse period, so that the plurality of pulses emitted by the first source do not overlap with the plurality of pulses emitted by the second source.

8. The system according to claim 1, further comprising: An endoscope, wherein the image sensor is disposed in the distal region of the endoscope; and A prism disposed in the distal region of the endoscope, the prism being configured to reflect electromagnetic radiation onto the image sensor; The planar side of the pixel array is oriented parallel to the longitudinal axis of the endoscope.

9. The system according to claim 1, wherein, The pixel array is relatively inefficient in accumulating electromagnetic radiation emitted by the second source compared to electromagnetic radiation emitted by the first source, wherein the second source emits electromagnetic radiation in one or more of the following wavelength bands: The near-infrared band of the electromagnetic spectrum; or The infrared band of the electromagnetic spectrum.

10. The system according to claim 1, wherein, The pixel array is relatively inefficient in accumulating electromagnetic radiation emitted by the second source compared to electromagnetic radiation emitted by the first source, wherein the first source emits in one or more of the following wavelength bands: White light; or The partitioning of electromagnetic radiation from the visible band of the electromagnetic spectrum.

11. The system according to claim 1, wherein, The electromagnetic radiation emitted by the second source has a shorter amplitude compared to the electromagnetic radiation emitted by the first source.

12. The system according to claim 1, wherein, The electromagnetic radiation emitted by the second source has less energy compared to the electromagnetic radiation emitted by the first source.

13. The system according to claim 1, wherein, The electromagnetic radiation emitted by the second source is dimmer than the electromagnetic radiation emitted by the first source.

14. The system according to claim 1, wherein, The first source and the second source emit different bands of the electromagnetic spectrum, and wherein the pixel array is inherently less efficient in accumulating the bands of electromagnetic radiation emitted by the second source compared to the bands of electromagnetic radiation emitted by the first source.

15. The system according to claim 1, wherein, The image sensor reads out color data frames and high-level data frames, wherein: The pixel array accumulates the electromagnetic radiation emitted by the first source to output the color data frame; and The pixel array accumulates electromagnetic radiation generated by the emission of the second source to output the advanced data frame.

16. The system according to claim 15, wherein, The advanced data frame includes one or more of the following: A fluorescence data frame corresponding to the pixel array, wherein the fluorescence data frame accumulates one or more of the fluorescence excitation wavelength or the fluorescence relaxation wavelength of the electromagnetic radiation; or A multispectral data frame corresponding to the pixel array, wherein the multispectral data frame accumulates one or more of the multispectral wavelengths of electromagnetic radiation or the spectral response emission of electromagnetic radiation.

17. The system according to claim 16, wherein, The pulse period causes: During the blanking period immediately preceding the readout of the color data frame, the first source loop is activated; While the image sensor is reading out the color data frame, the first source is cyclically turned off during the rolling readout sequence; During the blanking period immediately preceding the reading of the high-level data frame, the first source loop is turned off; as well as The first source is cyclically turned off during the rolling readout sequence while the image sensor is reading out the advanced data frame.

18. The system according to claim 17, wherein, The pulse period causes: During the blanking period immediately preceding the readout of the color data frame, the second source loop is turned off; The second source is cyclically activated during the rolling readout sequence while the image sensor is reading out the color data frame; During the blanking period immediately preceding the reading of the high-level data frame, the second source loop is enabled; as well as The second source is cyclically activated during the rolling readout sequence while the image sensor is reading out the advanced data frame.

19. The system according to claim 16, wherein, Information from the color data frame and the advanced data frame is combined to generate an overlay frame, wherein the overlay frame includes a color image depicting the scene and also includes pseudo-color overlay depiction information determined based on the advanced data frame.

20. The system according to claim 16, wherein, The color data frame also includes advanced visualization data generated by the transmitter cyclically activating the second source during the readout cycle when the image sensor reads the color data frame; and The system also includes a processor configured to subtract the advanced visualization data from the color data frame.

Images