Systems and methods for medical imaging using rolling shutter imagers

Abstract

To effectively generate a global shutter effect in a rolling shutter imager.SOLUTION: A method for imaging tissue of a subject using an electronic rolling shutter imager includes sequentially resetting rows of pixels of the rolling shutter imager from a first row 506 to a last row 508, sequentially reading charge accumulated at the rows of pixels from the first row to the last row, wherein the first row is read after resetting the last row, illuminating the tissue of the subject with illumination light for an illumination period 520, 528 that lasts longer than a vertical blanking period 517, wherein the vertical blanking period is the period from the resetting of the last row to the reading of the first row, and generating an image frame from read values of charge accumulated at the rows of pixels, wherein at least one read value of charge accumulated at a row of pixels is removed or replaced to generate the image frame.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 793,790, filed January 17, 2019, which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to medical imaging, and more particularly to producing medical images. [Background technology]

[0003] Medical systems, instruments, or tools are utilized for various purposes before, during, or after surgery. Some of these medical tools may be used in what are commonly referred to as endoscopic procedures. For example, endoscopy in the medical field allows for viewing of internal features of a patient's body without the use of traditional fully invasive surgery. Endoscopic imaging and diagnostic systems incorporate endoscopes to allow surgeons to view the surgical site, and endoscopic tools enable non-invasive surgery of that site. Endoscopes may be used with camera systems for processing images received by the endoscope. Endoscopic camera systems typically include a camera head connected to a camera control unit (CCU), which processes input image data received from the camera's image sensor and outputs the image data for display. The CCU may control an illumination device that generates illumination light provided to the imaged scene.

[0004] Various image sensors, including charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors, can be used in endoscopic imaging systems. The construction of CCDs is generally more complex than that of CMOS sensors, which can be integrated into the same high-volume wafer fabrication facilities used for related technologies such as microprocessors and chipsets. As a result, for similar performance, CMOS sensors are often less expensive than CCDs. In addition to low cost, the common fabrication processes used to create CMOS imagers allow CMOS pixel arrays to be integrated onto a single circuit with other electronic devices, such as clock drivers, digital logic, analog-to-digital converters, and other appropriate electronics. The compact construction possible with CMOS imagers also reduces space requirements and may reduce power consumption.

[0005] CMOS-based imagers may utilize an electronic rolling shutter to expose pixels in a sensor array. In an electronic rolling shutter, rows of pixels are cleared, exposed, and read out in a sequential order. During integration, the rows of pixels are exposed to light energy, and each pixel generates a charge corresponding to the amount and wavelength of light striking the pixel. Because the rows are driven and read out in sequence, there is a lapse of time between the integration of the first row and the integration of the last row. Due to the lapse of time between the integration of the first row and the integration of subsequent rows, a CMOS imager with an electronic rolling shutter may capture video images that have blur or other rolling shutter effects. Summary of the Invention

[0006] In some embodiments, imaging systems and methods use an electronic rolling shutter imager to create a global shutter effect using coordinated illumination provided to a target object, such as a target tissue in a subject, in an electronic rolling shutter imager that can offer the cost and performance advantages of a rolling shutter imager without the rolling shutter effect. In some embodiments, illumination is provided during the period between the reset of the last row of the imaging sensor in a frame period and the readout of the first row of the imaging sensor in that frame period, resulting in a global shutter-type effect. The illumination period is extended beyond this period to increase the sensor's exposure time for increased sensitivity. Readouts from rows that may include the rolling shutter effect can be removed or replaced when generating an image frame. The illumination light can be pulsed according to the exposure needs of the imager.

[0007] In some embodiments, an extended global shutter period is created by skipping every readout / reset time so that each row integrates for two frames at the nominal frame rate. This may reduce the frame rate of the generated image, but may increase sensitivity and allow for lower illumination levels. The illumination light may be pulsed depending on the exposure needs of the imager.

[0008] According to some embodiments, a method of imaging tissue of a subject using an electronic rolling shutter imager includes sequentially resetting pixel rows of the rolling shutter imager from a first row to a last row; sequentially reading out charge accumulated in the pixel rows from the first row to the last row, where the first row is read out after resetting the last row; illuminating the tissue of the subject with illumination light during an illumination period lasting longer than a vertical blanking period, where the vertical blanking period is the period from the resetting of the last row to the readout of the first row; and generating an image frame from the readouts of the charge accumulated in the pixel rows, where at least one readout of the charge accumulated in the pixel rows is removed or replaced to generate the image frame.

[0009] According to some embodiments, a method of operating an electronic rolling shutter imager, such as for imaging tissue of a subject, includes sequentially resetting pixel rows of the rolling shutter imager from a first row to a last row; sequentially reading out charge accumulated in the pixel rows from the first row to the last row, the first row being read out after resetting the last row; illuminating the tissue of the subject with illumination light during an illumination period lasting longer than a vertical blanking period, the vertical blanking period being the period between the resetting of the last row and the readout of the first row; and generating an image frame from the readouts of the charge accumulated in the pixel rows, wherein at least one readout of the charge accumulated in the pixel rows is removed or replaced to generate the image frame.

[0010] In any of these embodiments, the illumination period may begin before the reset of the last row.

[0011] In any of these embodiments, the illumination period may end after the readout of the first row.

[0012] In any of these embodiments, the illumination period may begin when at least the vertical blanking period begins.

[0013] In any of these embodiments, the illumination period may end when at least the vertical blanking period ends.

[0014] In any of these embodiments, a readout of the charge stored in at least the first row of pixels may be removed or replaced to generate the image frame.

[0015] In any of these embodiments, the readout of the charge accumulated in at least the last pixel row may be removed or replaced to generate the image frame.

[0016] In any of these embodiments, at least one readout of the charge accumulated in a row of pixels may be replaced by at least one predetermined value to generate an image frame.

[0017] In any of these embodiments, at least one readout of the charge stored in a row of pixels may be removed by cropping to generate an image frame.

[0018] In any of these embodiments, the subject's tissue may be illuminated with visible light, and the method may further include illuminating the subject's tissue with fluorescence excitation light during at least a subsequent vertical blanking period, and generating a fluorescence image frame based on light emitted from the subject's tissue in response to the fluorescence excitation light.

[0019] In any of these embodiments, the method may further include administering a fluorescent imaging agent to the subject prior to generating the fluorescent image frames. Thus, the fluorescent imaging agent may be administered prior to performing the method.

[0020] In any of these embodiments, the method may include illuminating tissue of the subject with fluorescent excitation light and visible illumination light simultaneously during an illumination period.

[0021] In any of these embodiments, illuminating the tissue of the subject with illumination light can include pulsing the illumination light.

[0022] In any of these embodiments, the method may include controlling the pulse width of the pulsed illumination light based on a readout of the charge accumulated in the row of pixels during a previous frame.

[0023] In any of these embodiments, the illumination light may be generated by at least one LED.

[0024] In any of these embodiments, the rolling shutter imager may be part of an endoscopic imager, and the method of imaging tissue of a subject or operating an electronic rolling shutter may exclude inserting such an endoscopic imager into a lumen.

[0025] In any of these embodiments, the rolling shutter imager may include a mechanical shutter that may remain at least partially open from before the illumination period begins until after the illumination period ends.

[0026] In any of these embodiments, the method may further include reducing the amount of light received at the rolling shutter imager by operating a mechanical shutter.

[0027] In any of these embodiments, the method may further include adjusting the gain of the rolling shutter imager based on a readout of the charge accumulated in the row of pixels during the previous frame.

[0028] According to some embodiments, a system for imaging tissue of a subject may include an illumination source and an imaging device including an electronic rolling shutter imager, wherein the imaging device is configured to sequentially reset pixel rows of the rolling shutter imager from the first row to the last row, sequentially read out charges accumulated in the pixel rows from the first row to the last row, wherein the first row is read out after the last row is reset, control the illumination source to illuminate the tissue of the subject during an illumination period lasting longer than a vertical blanking period, wherein the vertical blanking period is the period from the resetting of the last row to the readout of the first row, and generate an image frame from the readouts of the charges accumulated in the pixel rows, wherein at least one readout of the charges accumulated in the pixel rows is removed or replaced to generate the image frame.

[0029] In any of these embodiments, the imaging device may include a camera control unit connected to an imaging head that includes a rolling shutter imager.

[0030] In any of these embodiments, the illumination source may be configured for pulse width modulated illumination, and the camera control unit generates a pulse width modulated waveform for controlling the illumination source.

[0031] In any of these embodiments, the imager may be configured to control the illumination source so that the illumination period begins before the reset of the last row.

[0032] In any of these embodiments, the imager may be configured to control the illumination source so that the illumination period ends after the readout of the first row.

[0033] In any of these embodiments, the imager may be configured to control the illumination source such that the illumination period begins when at least the vertical blanking period begins.

[0034] In any of these embodiments, the imager may be configured to control the illumination source such that the illumination period ends when at least the vertical blanking period ends.

[0035] In any of these embodiments, the imager may be configured to generate an image frame in which readouts of the charge accumulated in at least a first row of pixels are removed or replaced to generate the image frame.

[0036] In any of these embodiments, the imaging device may be configured to generate an image frame such that readings of the charge accumulated in at least the last row of pixels are removed or replaced to generate the image frame.

[0037] In any of these embodiments, the imaging device may be configured to generate an image frame, wherein at least one readout of the charge accumulated in the pixel row is replaced by at least one predetermined value to generate the image frame.

[0038] In any of these embodiments, the imager may be configured to generate an image frame, where at least one readout of the charge stored in the row of pixels is removed by cropping to generate the image frame.

[0039] In any of these embodiments, the imaging device may be configured to control the illumination source to illuminate the subject's tissue with visible light during a vertical blanking interval, illuminate the subject's tissue with fluorescence excitation light during at least a subsequent vertical blanking interval, and generate fluorescence image frames based on light emitted from the subject's tissue in response to the fluorescence excitation light.

[0040] In any of these embodiments, the imaging device can be configured to control the illumination source to illuminate the subject's tissue with fluorescent excitation light and visible illumination light simultaneously during an illumination period.

[0041] In any of these embodiments, the illumination source may be configured to pulse the illumination light.

[0042] In any of these embodiments, the imager may be configured to control the illumination source to control the pulse width of the pulsed illumination light based on a readout of the charge accumulated in the pixel row during the previous frame.

[0043] In any of these embodiments, the illumination source may include at least one LED.

[0044] In any of these embodiments, the imaging device may include an endoscopic imager.

[0045] In any of these embodiments, the rolling shutter imager may include a mechanical shutter, and the image capture device may be configured to control the mechanical shutter so that the mechanical shutter remains at least partially open from before the illumination period begins until after the illumination period ends.

[0046] In any of these embodiments, the image capture device may be configured to reduce the amount of light received at the rolling shutter imager by operating a mechanical shutter.

[0047] In any of these embodiments, the imager may be configured to adjust the gain of the rolling shutter imager based on a readout of the charge accumulated in the pixel row during the previous frame.

[0048] According to some embodiments, a method of imaging tissue of a subject using an electronic rolling shutter imager includes sequentially resetting pixel rows of the rolling shutter imager from a first row to a last row, the sequential resetting from the first row to the last row lasting a first amount of time; illuminating the tissue of the subject with illumination light for an illumination period beginning when the last row of the rolling shutter imager is reset and lasting at least the first amount of time; accumulating charge in the pixel rows for at least the illumination period based on light received from the subject's tissue while the tissue of the subject is illuminated with the illumination light; sequentially reading out the charge accumulated in the pixel rows from the first row to the last row when the illumination period ends; and generating an image frame from the readouts of the charge accumulated in the pixel rows.

[0049] According to some embodiments, a method of operating an electronic rolling shutter imager, such as for imaging tissue of a subject, includes sequentially resetting pixel rows of the rolling shutter imager from a first row to a last row, the sequential resetting from the first row to the last row lasting a first amount of time; illuminating the tissue of the subject with illumination light for an illumination period beginning when the last row of the rolling shutter imager is reset and lasting at least the first amount of time; accumulating charge in the pixel rows for at least the illumination period based on light received from the subject's tissue while the tissue of the subject is illuminated with the illumination light; sequentially reading out the charge accumulated in the pixel rows from the first row to the last row when the illumination period ends; and generating an image frame from the readouts of the charge accumulated in the pixel rows.

[0050] In any of these embodiments, illuminating the tissue of the subject with illumination light can include pulsing the illumination light.

[0051] In any of these embodiments, the method may further include controlling a pulse width of the pulsed illumination light based on a readout of the charge accumulated in the pixel row during a previous frame.

[0052] In either of these embodiments, each row of pixels may accumulate charge over an illumination period.

[0053] In any of these embodiments, each row of pixels may accumulate charge for at least twice the first amount of time.

[0054] In any of these embodiments, the illumination light may be generated by at least one LED.

[0055] In any of these embodiments, the rolling shutter imager may be part of an endoscopic imager, and the method of imaging tissue of a subject or operating an electronic rolling shutter may exclude inserting such an endoscopic imager into a lumen.

[0056] In any of these embodiments, the rolling shutter imager may include a mechanical shutter that may remain at least partially open from before illumination begins until after the illumination period ends.

[0057] In any of these embodiments, the method may further include reducing the amount of light received at the rolling shutter imager by operating a mechanical shutter.

[0058] In any of these embodiments, the method may further include adjusting the gain of the rolling shutter imager based on a readout of the charge accumulated in the row of pixels during the previous frame.

[0059] According to some embodiments, a system for imaging tissue of a subject includes an illumination source and an imaging device including an electronic rolling shutter imager, wherein the imaging device is configured to: sequentially reset pixel rows of the electronic rolling shutter imager from a first row to a last row, wherein the sequential resetting from the first row to the last row lasts a first amount of time; control the illumination source to illuminate the tissue of the subject with illumination light for an illumination period beginning when the last row of the rolling shutter imager is reset and lasting at least the first amount of time; accumulate charge in the pixel rows for at least the illumination period based on light received from the subject's tissue while the tissue of the subject is illuminated with the illumination light; and sequentially read out the charge accumulated in the pixel rows from the first row to the last row when the illumination period ends; and generate an image frame from the readout values ​​of the charge accumulated in the pixel rows.

[0060] In any of these embodiments, the imaging device may include a camera control unit connected to an imaging head that includes a rolling shutter imager.

[0061] In any of these embodiments, the illumination source may be configured for pulse width modulated illumination, and the camera control unit generates a pulse width modulated waveform for controlling the illumination source.

[0062] In any of these embodiments, the imaging device can be configured to control the illumination source to illuminate the subject's tissue with pulsed illumination light.

[0063] In any of these embodiments, the imager may be configured to control the illumination source to control the pulse width of the pulsed illumination light based on a readout of the charge accumulated in the pixel row during the previous frame.

[0064] In any of these embodiments, the imager may be configured so that each row of pixels accumulates charge over an illumination period.

[0065] In any of these embodiments, the imager may be configured so that each row of pixels accumulates charge for at least twice the first amount of time.

[0066] In any of these embodiments, the illumination source may include at least one LED.

[0067] In any of these embodiments, the imaging device may include an endoscopic imager.

[0068] In any of these embodiments, the rolling shutter imager may include a mechanical shutter, and the imager may be configured to control the mechanical shutter to remain at least partially open from before illumination begins until after the illumination period ends.

[0069] In any of these embodiments, the image capture device may be configured to reduce the amount of light received at the rolling shutter imager by operating a mechanical shutter.

[0070] In any of these embodiments, the imager may be configured to adjust the gain of the rolling shutter imager based on a readout of the charge accumulated in the pixel row during the previous frame.

[0071] According to some embodiments, a non-transitory tangible computer-readable medium may have computer-executable program code embedded therein for performing any of the methods described above.

[0072] According to some embodiments, a computer program product may include computer implementable instructions that, when implemented by a programmable computer, cause the computer to perform any of the methods described above.

[0073] According to some embodiments, a kit for processing time series images of tissue of a subject includes any of the above-described systems or the above-described non-transitory tangible computer-readable medium and a contrast agent.

[0074] In any of these embodiments, the imaging agent may be a fluorescent imaging agent.

[0075] According to some embodiments, the fluorescent imaging agent is for use in any of the above-described systems, any of the above-described methods, or any of the above-described kits for imaging tissue.

[0076] In any of these embodiments, tissue imaging may include imaging of the tissue during blood flow imaging, tissue perfusion imaging, lymphatic imaging, or a combination thereof.

[0077] In any of these embodiments, the blood flow imaging, tissue perfusion imaging, and / or lymphatic imaging may include blood flow imaging, tissue perfusion imaging, and / or lymphatic imaging during an invasive medical procedure, a minimally invasive medical procedure, or a non-invasive medical procedure.

[0078] In any of these embodiments, the invasive medical procedure may include a cardiac-related or reconstructive medical procedure.

[0079] In any of these embodiments, the cardiac-related medical procedure may include a coronary artery bypass graft (CABG) procedure.

[0080] In any of these embodiments, the CABG procedure can be on-pump or off-pump.

[0081] In any of these embodiments, the non-invasive medical procedure may include a wound care procedure.

[0082] In any of these embodiments, lymphatic imaging may include lymph node identification, lymph node drainage, lymphatic system mapping, or a combination thereof.

[0083] In any of these embodiments, the lymphatic imaging may relate to the female reproductive system.

[0084] According to some embodiments, any of the above-described systems, any of the above-described methods, any of the above-described kits, or any of the above-described fluorescent agents are used for lymphatic imaging.

[0085] According to some embodiments, any of the above-described systems, any of the above-described methods, any of the above-described kits, or any of the above-described fluorescent agents are used for blood flow imaging, tissue perfusion imaging, or a combination thereof. [Brief explanation of the drawings]

[0086] The present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0087] [Figure 1] FIG. 1 is a diagram of an endoscopic camera system according to some embodiments.

[0088] [Figure 2] FIG. 2 is a diagram of a portion of the endoscopic camera system of FIG. 1 and a target object for imaging, according to some embodiments.

[0089] [Figure 3] FIG. 3 is a block diagram of an endoscopic camera system according to some embodiments.

[0090] [Figure 4] FIG. 4 is a flow diagram of a method for generating alternating white light and fluorescent image frames, according to one embodiment.

[0091] [Figure 5] FIG. 5 is a diagram illustrating an exemplary relative timing of resetting and readout of pixel rows of an imaging sensor and illumination of an illumination device to produce alternating white light and fluorescent image frames, according to one embodiment.

[0092] [Figure 6]FIG. 6 is a flow diagram of a method for generating image frames with increased sensitivity by increasing integration time, according to one embodiment.

[0093] [Figure 7] FIG. 7 is a diagram illustrating an exemplary relative timing of resetting and readout of pixel rows of an imaging sensor and illumination of an illumination device to produce an image frame with increased sensitivity, according to one embodiment.

[0094] [Figure 8] FIG. 8 is a block diagram illustrating components of an exemplary lighting device that may be used to generate pulse-width modulated illumination light, according to some embodiments.

[0095] [Figure 9] FIG. 9 is a schematic diagram of an imager having three imaging sensors, according to some embodiments.

[0096] [Figure 10] FIG. 10 is an illustrative depiction of an exemplary fluorescence imaging system, according to some embodiments.

[0097] [Figure 11] FIG. 11 is an illustrative depiction of an exemplary illumination module of a fluorescence imaging system, according to some embodiments.

[0098] [Figure 12] FIG. 12 is an exemplary camera module of a fluorescence imaging system, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0099] Reference will now be made in detail to implementations and embodiments of various aspects and variations of the invention, examples of which are illustrated in the accompanying drawings. Various devices, systems, methods, processors, kits, and imaging agents are described herein. While at least two variations of the devices, systems, methods, processors, kits, and imaging agents are described, other variations may include aspects of the devices, systems, methods, processors, kits, and imaging agents described herein combined in any suitable manner having all or some combination of the described aspects. Examples are described more fully below with reference to the accompanying drawings; however, they may be embodied in different forms and should not be construed as limited to the embodiments described herein. Rather, these examples are provided so that this disclosure will be thorough and complete, and will fully convey exemplary implementations to those skilled in the art.

[0100] Described herein are exemplary devices, systems, and methods for medical imaging using a rolling shutter imager and controlled illumination to generate a time series of images with limited rolling shutter effects. More generally, exemplary devices, systems, and methods for operating a rolling shutter imager and controlled illumination for medical imaging, etc., to generate a time series of images with limited rolling shutter effects are described. The systems, devices, and methods can be used to image tissue of a subject, such as in an endoscopic imaging procedure. Imaging can be performed pre-operatively, intra-operatively, post-operatively, and during diagnostic imaging sessions and procedures. The imaging method itself may exclude insertion of an endoscopic imager into a lumen within the body. The imaging method itself may exclude any invasive surgical steps.

[0101] The imaging system may include a mode in which illumination of a target scene (e.g., a target tissue of a patient) is controlled to create a global shutter-type effect with a long integration time for the pixels of the imaging sensor. The imaging sensor may be driven according to a given frame rate, but instead of reading out rows of pixels of the sensor every possible frame period, rows are read out every other frame period, allowing the pixels of the sensor to integrate over a longer period. This results in a relatively long integration period during which substantially all rows can be simultaneously exposed to the scene. Illumination light is provided during this "global shutter" period, so that the resulting image frame is substantially free of the rolling shutter effect.

[0102] The imaging system may include a mode for generating alternating visible light and fluorescent image frames in which illumination light is provided during an extended vertical blanking period. This mode may be used, for example, to generate white light images with a fluorescent overlay. As known in the art, the vertical blanking period in the operation of a rolling shutter imager is the period from when the last row of the sensor is reset to when the first row of the sensor is read out. During the vertical blanking period, all rows are integrated simultaneously. Various embodiments achieve a global shutter-type effect by providing illumination light during the vertical blanking period. However, because the vertical blanking period is relatively short, in some embodiments, illumination light is provided for a period longer than the vertical blanking period to increase the sensitivity of the imager. This may result in a rolling shutter effect for some of the rows of the sensor; the visibility of these effects to a user may be minimized by removing or replacing one or more affected rows.

[0103] In various examples such as those described above, the illumination light may be adjusted using pulse width modulation to provide the correct amount of illumination for the scene. The imaging system may control the amount of light to optimally expose the imaging sensor, or may control it based on the intensity at the sensor in one or more previous frames.

[0104] As used in the following description, it should be understood that the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that, as used herein, the term "and / or" refers to and encompasses any combination of one or more of the associated listed items. It should be further understood that, as used herein, the terms "includes," "including," "comprises," and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, components, and / or units, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, units, and / or groups thereof.

[0105] Certain aspects of the present disclosure include process steps and instructions described herein in the form of algorithms. It should be noted that the process steps and instructions of the present disclosure may be embodied in software, firmware, or hardware, and, if embodied in software, may be downloaded to reside and operate on different platforms for use by various operating systems. As will become apparent from the discussion that follows, throughout the description, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and "generating" will be understood to refer to the operations and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in information storage, transmission, and display devices, such as memory and registers, of the computer system.

[0106] The present disclosure in some embodiments also relates to devices for performing the operations herein. This device may be specially constructed for the required purposes, or it may include a general-purpose computer selectively driven or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a non-transitory, computer-readable storage medium, such as a floppy disk, a USB flash drive, an external hard drive, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC), or any other type of disk, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus. Furthermore, the computers referred to herein may include a single processor or may be architectures employing multiple processor designs to increase computing power.

[0107] The methods, devices, and systems described herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the following. In addition, the present invention is not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0108] FIG. 1 shows an example of an endoscopic imaging system 10 including a scope assembly 11 that may be utilized in an endoscopic procedure. The scope assembly 11 includes an endoscope or scope 12 coupled to a camera head 16 by a coupler 13 located at the distal end of the camera head 16. Light is supplied to the scope by a light source 14 via a light guide 26, such as a fiber optic cable. The camera head 16 is coupled to a camera control unit (CCU) 18 by an electrical cable 15. The CCU 18 is connected to and communicates with the light source 14. The operation of the camera 16 is controlled in part by the CCU 18. The cable 15 may carry video and / or still image data from the camera head 16 to the CCU 18 and may also carry various control signals bidirectionally between the camera head 16 and the CCU 18.

[0109] A control or switch arrangement 17 may be provided on the camera head 16 to allow a user to manually control various functions of the system 10, which may include switching from one imaging mode to another, as described below. Voice commands may be entered into a microphone 25 attached to a headset 27 worn by the practitioner and coupled to the voice control unit 23. A handheld control device 29, such as a tablet with a touchscreen user interface or a PDA, may be coupled to the voice control unit 23 as an additional control interface. In the illustrated embodiment, a recorder 31 and a printer 33 are also coupled to the CCU 18. Additional devices, such as image capture and archiving devices, may be included in the system 10 and may be coupled to the CCU 18. Video image data acquired by the camera head 16 and processed by the CCU 18 is converted into an image that may be displayed on the monitor 20, recorded by the recorder 31, and / or used to generate a still image, a hard copy of which may be generated by the printer 33.

[0110] 2 shows an example of a portion of an endoscopic system 10 used to illuminate and receive light from an object 1, such as a surgical site on a patient. The object 1 may include a fluorescent marker 2, for example, as a result of administering a fluorescent contrast agent to the patient. The fluorescent marker 2 may include, for example, indocyanine green (ICG).

[0111] The light source 14 can generate visible illumination light (such as any combination of red, green, and blue light) to generate a visible (e.g., white light) image of the target object 1, and can also generate fluorescent excitation illumination light to excite fluorescent markers 2 in the target object to generate a fluorescent image. The illumination light is transmitted through an optical lens system 22 that focuses the light onto a light pipe 24. The light pipe 24 can generate homogenous light, which is then transmitted to a fiber optic light guide 26. The light guide 26 can include multiple optical fibers and is connected to a light post 28 that is part of the endoscope 12. The endoscope 12 includes an illumination path 12' and an optical channel path 12".

[0112] The endoscope 12 may include a notch filter 131 that passes some or all (preferably at least 80%) of the fluorescent emission light (e.g., in the wavelength range of 830 nm to 870 nm) emitted by the fluorescent markers 2 in the target object 1 and some or all (preferably at least 80%) of the visible light (e.g., in the wavelength range of 400 nm to 700 nm) such as the visible illumination light reflected by the target object 1, but blocks substantially all of the fluorescent excitation light (e.g., infrared light having a wavelength of 808 nm) used to excite the fluorescent emission from the fluorescent markers 2 in the target object 1. The notch filter 131 may have an optical density of OD5 or greater. In some embodiments, the notch filter 131 may be located in the coupler 13.

[0113] FIG. 3 schematically illustrates an exemplary imaging system 300 using an electronic rolling shutter imager 302 to generate images (e.g., still images and / or video) of a target object, such as a target tissue in a patient, in accordance with some embodiments. The system 300 may be used, for example, in the endoscopic imaging system 10 of FIG. 1. The imager 302 includes a CMOS sensor 304 having an array of pixels 305 arranged in pixel rows 308 and pixel columns 310. The imager 302 may include a control component 306 that controls signals generated by the CMOS sensor 304. Examples of control components include a gain circuit for generating a multi-bit signal indicative of light incident on each pixel of the sensor 304, one or more analog-to-digital converters, one or more line drivers that act as buffers and provide drive power to the sensor 304, a row circuit, and a timing circuit. The timing circuit may include components such as a bias circuit, a clock / timing generation circuit, and / or an oscillator. The row circuitry may enable one or more processing and / or operational tasks, such as addressing pixel rows 308, addressing pixel columns 310, resetting charge on pixel rows 308, enabling exposure of pixels 305, decoding signals, amplifying signals, analog-to-digital signal conversion, applying timing, applying readout and reset signals, and other suitable processes or tasks. The imager 302 may also include a mechanical shutter 312, which may be used, for example, to control exposure of the image sensor 304 and / or to control the amount of light received by the image sensor 304.

[0114] The one or more control components may be integrated into the same integrated circuit as the sensor 304, or may be separate components. The imager 302 may be incorporated into an imaging head, such as camera head 16 of the system 10.

[0115] One or more control components 306, such as row and timing circuits, may be electrically connected to an imaging controller 320, such as the camera control unit 18 of the system 10. The imaging controller 320 may include one or more processors 322 and memory 324. The imaging controller 320 may receive the imager row readouts and control other imager operations, including readout timing and mechanical shutter operation. The imaging controller 320 may generate image frames, such as video frames, from the row and / or column readouts from the imager 302. The generated frames may be provided to a display 350 for display to a user, such as a surgeon.

[0116] The system 300 in this example includes a light source 330 for illuminating a target scene. The light source 330 is controlled by an imaging controller 320. The imaging controller 320 may determine the type of illumination provided by the light source 330 (e.g., white light, fluorescent excitation light, or both), the intensity of the illumination provided by the light source 330, and / or the on / off times of the illumination synchronized with the rolling shutter operation. The light source 330 may include a first light generator 332 for generating light at a first wavelength and a second light generator 334 for generating light at a second wavelength. For example, in some embodiments, the first light generator 332 is a white light generator that may include multiple individual light generating components (e.g., multiple LEDs of different colors), and the second light generator 334 is a fluorescent excitation light generator, such as a laser diode.

[0117] The light source 330 includes a controller 336 that controls the light output of the light generators. The controller 336 can be configured to provide pulse-width modulation of the light generators to adjust the intensity of the light provided by the light source 330, which can be used to manage overexposure and underexposure. In some embodiments, the nominal current and / or voltage of each light generator remains constant, and the light intensity is adjusted by switching the light generators (e.g., LEDs) on and off according to a pulse-width control signal. In some embodiments, a PWM control signal is provided by the imaging controller 336. This control signal can be a waveform that corresponds to the desired pulse-width modulation operation of the light generators.

[0118] The imaging controller 320 may be configured to determine the illumination intensity required from the light source 330 and may generate a PWM signal that is communicated to the light source 330. In some embodiments, depending on the amount of light received at the sensor 304 and the integration time, the light source may be pulsed at different rates to vary the illumination light intensity at the target scene. The imaging controller 320 may determine the illumination light intensity required for a subsequent frame based on the amount of light received at the sensor 304 in the current frame and / or one or more previous frames. In some embodiments, the imaging controller 320 may control the intensity of a pixel via PWM of the light source 330 (to increase / decrease the amount of light at the pixel), via operation of the mechanical shutter 312 (to increase / decrease the amount of light at the pixel), and / or via a change in gain (to increase / decrease the pixel's sensitivity to received light). In some embodiments, the imaging controller 320 primarily uses PWM of the illumination source to control pixel intensity while keeping the shutter open (or at least not operating the shutter) and maintaining a gain level. The controller 320 may operate the shutter 312 and / or modify the gain when the light intensity is at a maximum or minimum and further adjustment is needed.

[0119] Method for generating visible and fluorescent light images - Patent Application 20070122997 Figure 4 shows an exemplary method 400 for generating a time series of white-light and fluorescent-light image frames, according to one embodiment. Method 400 may be performed by an imaging system, such as system 300 of Figure 3. In step 402, pixel rows of an electronic rolling shutter imager, such as imager 302 of Figure 3, are reset sequentially from the first row (e.g., the top or bottom row) to the last row (e.g., the bottom or top row), according to well-known rolling shutter imager operation.

[0120] In step 404, the charge accumulated in each pixel row since the reset in step 402 is read out in order from the first row to the last row. The readout of the first row does not begin until a period of time has elapsed since the reset of the last row in step 402. This period may be referred to as the vertical blanking period, during which substantially all rows of the image sensor are simultaneously integrating charge (i.e., integrating). This simultaneous integration period can be utilized to create a global shutter effect, as described further below.

[0121] In step 406, the target tissue of the subject is illuminated with illumination light from the illuminator for an illumination period. The illumination period is longer than the vertical blanking period (the time from the reset of the last row to the readout of the first row), and may span most or all of the vertical blanking period. The illumination period can begin before, simultaneously with, or after the resetting of at least the last row of pixels in step 402. In any of these embodiments, the illumination period can end before, simultaneously with, or after the readout of at least the first row of pixels in step 402.

[0122] Optionally, the illumination light can be white light (or any other visible light spectrum) to generate white light (or any other visible light spectrum) image frames. Alternatively, or in addition, the illumination light can be fluorescent excitation light, such as infrared light, to generate fluorescent image frames.

[0123] The length of the illumination period may be selected based on the desired imaging sensitivity for a given application. In applications with relatively high illumination light levels, the illumination period may be shorter than in applications with relatively low illumination light levels. Generally, the longer the illumination period, the greater the number of rows that may be distorted and, therefore, the greater the number of rows that need to be removed or replaced with a predetermined value. Therefore, selecting the length of the illumination period may require a balance between sensitivity and field of view.

[0124] In step 408, an image frame is generated from the readouts of the charges accumulated in the pixel rows. Because the illumination period in step 406 is extended beyond the vertical blanking period, not all rows are exposed to light from the target tissue due to the illumination light for the same amount of time during each frame capture. For example, in an embodiment in which the illumination period begins when the penultimate row is reset (i.e., before the last row is reset), the last row integrates light received from the target tissue due to a shorter illumination period than the previous rows. This can result in visual artifacts in one or more rows that integrate light received due to illumination light for a shorter period than the illumination period within each frame capture period. As a result, one or more rows affected by the extended illumination period, i.e., rows whose illumination period begins before the respective row reset and / or ends after the respective row readout, can be removed or replaced to generate an image frame. One or more affected rows can be replaced with one or more predetermined values. An image frame can include one or more rows in which each pixel value within a given row is a predetermined value. When displayed, the image frame may appear with, for example, one or more black lines at the top and / or bottom. In some embodiments, the affected row or rows are removed, such as by cropping the affected rows. Thus, the generated image frame may include fewer rows than were included in the pixel data from the imaging sensor.

[0125] Optionally, the method may further include alternately generating white-light (or any other visible light spectrum) image frames and fluorescence image frames. In step 406 above, the illumination light may be white light, and the image frames generated in step 408 may be white-light image frames. The capture period of a subsequent frame may be used to generate a fluorescence image frame. Thus, during the capture period of a subsequent frame, the white-light illumination remains off for an extended vertical blanking period. With the white light off, the illuminator generates fluorescence excitation light, such as infrared light, to cause fluorescence emission from the target tissue (e.g., due to fluorescent agents in the tissue and / or tissue autofluorescence). The fluorescence emission from the target tissue is received by pixel rows, and readout of the pixel rows is used to generate the fluorescence image frame.

[0126] The subsequent frame can be used to capture a white light frame, and as such, white light illumination can be provided for a white light illumination period spanning the next vertical blanking period, which can span the integration period of at least some of the rows, e.g., the bottom row, for the current fluorescence image capture, so that the affected rows can be replaced or removed, as described above.

[0127] Similarly, the illumination period from the previous white-light frame (e.g., step 406 above) may extend into the integration period of at least one row of pixels during capture of a fluorescence image frame, causing distortion in the affected row or rows, and therefore, as described above, the affected row or rows may be removed or the associated pixel data may be replaced with one or more predetermined values.

[0128] Fluorescence excitation illumination during capture of a fluorescence image may be provided for an illumination period spanning the vertical blanking period of a frame, as described above with respect to the white light illumination period. The fluorescence excitation illumination period may be the same as the white light illumination period, i.e., have the same duration, the same relative start time, and / or the same relative end time, or may be different.

[0129] The fluorescence excitation light may be provided continuously for the duration of the imaging. This may increase sensitivity to fluorescence emissions from tissue because the tissue will emit fluorescence for the duration of the pixel integration period, rather than over a relatively short, extended vertical blanking period. Although the continuous fluorescence emissions will overlap with the capture of previous and subsequent white-light frames for at least some of the pixel rows (due to the rolling shutter), the intensity of the fluorescence emissions from the tissue may be sufficiently lower than the intensity of the reflected white light, and as a result, the contribution of the fluorescence emissions to the pixel readout during the capture of the white-light frames may be negligible, so that the white-light image frames may not be significantly affected.

[0130] According to some embodiments, the imager may include a mechanical shutter that may be left open or otherwise unused for regulation of the light at the sensor, at least during the illumination period. Control of light intensity at the imager may instead be controlled via pulsing of the illumination light.

[0131] 5 illustrates an exemplary relative timing of resetting and readout of pixel rows of an image sensor and illumination of an illumination device to produce alternating white-light and fluorescent image frames, according to one embodiment. The image sensor includes N rows, and the resetting and readout of the first four rows and the last three rows are shown. A period 502 during which a white-light image frame is captured, followed by a period 503 during which a fluorescent image frame is captured, is shown.

[0132] At time 504, the first row 506 is read out and reset, beginning the white light image frame integration period 510. Each row is read out and reset successively until the last row 508 is read out and reset at time 505. The period from the readout / reset of the first row to the readout / reset of the last row is the readout time 512 (also referred to herein as the reset time).

[0133] At time 514, the first row 506 is read out (and reset) to provide intensity values ​​for the pixels of the first row 506 related to the amount of light received by the pixels of the first row during the integration period 510 of the first row 506. The readout of the first row 506 occurs after the reset of the last row 508. Each row is read out (and reset) successively until the last row 508 is read out (and reset) at time 534. Each row has the same amount of integration time 516, as shown below the row in FIG. 5.

[0134] The period of time between the reset of the last row 508 and the readout of the first row 506 is a vertical blanking period 517. During the vertical blanking period 517, white light illumination is provided by the illuminator. An exemplary illumination control signal 518 is shown at the bottom of FIG. 5. An illumination period 520 begins at time 522 and ends at time 524; i.e., the illuminator does not provide illumination outside of this illumination period 520 in the illustrated time frame. As can be seen, in the illustrated embodiment, the start time 522 of white light illumination is before the reset of the last row 508, and the end time 524 of the illumination period 520 is after the readout time 514 of the first row 506. This extended illumination period (also referred to herein as an extended vertical blanking period, even though the actual vertical blanking period is not extended) provides more time for the middle pixel rows to accumulate charge than if the illumination period 520 were simply limited to the vertical blanking period 517.

[0135] Fluorescent light frames are captured in much the same manner as white light frames, as described above. In the illustration of Figure 5, the first row reset for the fluorescent frame capture period 503 is depicted, for simplicity, as the same as the first row readout time 514 for the white light image frame capture period 502. For a fluorescent frame, white light control signal 518 remains low, and fluorescent excitation light signal 526 goes high, causing the illuminator to emit fluorescent excitation light. Fluorescent excitation light emission period 528 spans vertical blanking period 517 and begins at time 530 and ends at time 532.

[0136] Because the illumination period 520 of the white light frame acquisition period 502 overlaps with the integration period of at least the first row 506 of the fluorescence image frame acquisition period 503, the readout of the first row of the fluorescence frame will be inaccurate with respect to the fluorescence response of the portion of tissue relative to the first row 506. Therefore, the readout of at least the first row 506 is replaced with a predetermined value, such as zero, or an image frame is generated by omitting at least the first row. Furthermore, because the subsequent image frame will be another white light image frame having a white light illumination period 520 that begins before the readout 536 of the last row 508 of the fluorescence image frame acquisition period 503, the readout 536 of the last row 508 will also be inaccurate and therefore will be replaced or removed when generating the fluorescence image frame.

[0137] The illumination period 528 for the fluorescence image frames can be of the same length, relative start time, and / or relative end time as the illumination period 520 for the white-light image frames. Alternatively, the illumination period 528 can differ from the illumination period 520 according to the relative sensitivity required for imaging the tissue's fluorescence response to reflected white light. In some embodiments, fluorescence excitation is always provided, which, as discussed above, may not significantly affect the capture of white-light images due to the relatively low intensity of fluorescence emission from tissue (and filtering of fluorescence excitation light as discussed herein).

[0138] By repeating the above process, white-light image frames and fluorescent image frames are generated alternately. The white-light image frames and fluorescent image frames may be displayed separately on the display, or, for example, the fluorescent image frame may be superimposed on the white-light image frame. The white-light and fluorescent frames may each be generated at half the frame rate of the imager.

[0139] According to various embodiments, the extended vertical blanking period can be selected depending on the balance between desired sensitivity and desired field of view. According to various embodiments, the extended vertical blanking period can be less than twice the length of the vertical blanking period, less than three times the vertical blanking period, less than four times the vertical blanking period, less than five times the vertical blanking period, or less than ten times the vertical blanking period. According to various embodiments, the extended vertical blanking period can be at least twice the length of the vertical blanking period, at least three times the vertical blanking period, at least four times the vertical blanking period, at least five times the vertical blanking period, or at least ten times the vertical blanking period. In various embodiments, the length of the extended vertical blanking illumination period can be dynamically adjusted based on, for example, intensity measurements at the imaging sensor. For example, if the amount of illumination light results in a relatively high light intensity at the imaging sensor in the previous frame, the illumination period can be shortened in the subsequent imaging frame. This can be done instead of or in addition to adjusting the illumination light intensity.

[0140] Optionally, the relative number of rows of pixel data removed or replaced due to artifacts created by the extended vertical blanking illumination period may be less than 50%, less than 40%, less than 30%, less than 20%, less than 10%, less than 5%, less than 2%, less than 1%, less than 0.1%, less than 0.01%, less than 0.001%, or less than 0.0001%. In some embodiments, rows of pixel data from one type of imaging frame (e.g., fluorescence imaging frames) are removed or replaced, but rows of pixel data from another type of imaging frame (e.g., white-light imaging frames) are not removed or replaced. This may be useful, for example, when the fluorescence frames and white-light frames are displayed on different displays or side-by-side on the same display.

[0141] Method for generating image frames with increased sensitivity 6 provides an exemplary method 600 for generating image frames with increased sensitivity by utilizing two frame-worth integration times. Method 600 may be performed by an imaging system, such as imaging system 300 of FIG. 3, having a rolling shutter imager, such as rolling shutter imager 302 of system 300, and a light source, such as light source 330 of system 300.

[0142] In step 602, pixel rows of the rolling shutter imager are reset sequentially from the first row to the last row according to well-known rolling shutter imager operation. This reset (readout) period is an amount of time referred to herein as the reset or readout period. In step 604, illumination light from an illuminator is illuminated onto the subject's target tissue for an illumination period that begins when the last row of the rolling shutter imager is reset. The amount of time that the illumination period lasts is at least the amount of time that the reset / readout period lasts.

[0143] In step 606, charge is accumulated in pixel rows of the imager for at least the illumination period based on light received from the subject's tissue while the tissue is illuminated with illumination light. In step 608, the accumulated charge is read out sequentially from the first row to the last row. Substantially once the first row is read out, the illumination period ends and the tissue is no longer illuminated with illumination light. Thus, the tissue is illuminated for the period between the reset of the last row and the subsequent readout of the first row. In this manner, each row of the imager receives light from the subject's tissue resulting from illuminating the same tissue for the same period of time, effectively creating a global shutter effect in the rolling shutter imager. The duration of this global shutter period is similar to the vertical blanking period described above with respect to method 400, but is significantly longer because it lasts at least as long as the readout period.

[0144] Optionally, this global shutter period is created by integrating each row over two frame periods, skipping all other readout / reset steps the imager can perform. This results in a reduction in the maximum imaging frame rate, but increases the imager's sensitivity without introducing a rolling shutter effect. In other words, the imager is driven to produce a certain nominal frame rate (i.e., the time required to reset / readout all lines provides the ability to achieve the nominal frame rate), but every other frame readout / reset operation is skipped so that the imaging frame rate is half the nominal frame rate. The amount of time each row integrates while exposed to light from the target tissue (exposure period) can be equal to twice the vertical blanking period plus the amount of time required to reset / readout all lines. In some embodiments, the vertical blanking period may be eliminated, so that the length of the exposure period is simply the amount of time required to reset / readout all lines.

[0145] In step 610, an image frame is generated from the readouts of the charge accumulated in the pixel rows. Unlike the extended illumination period of method 400, the illumination period in method 600 does not overlap with previous or subsequent frames. Therefore, no rows need to be removed or replaced when generating the image frame.

[0146] The imager may include a mechanical shutter that may be left open or otherwise unused during at least the illumination period to regulate the light at the sensor. Control of light intensity at the imager may alternatively be controlled via pulsing of the illumination light.

[0147] 7 illustrates an exemplary relative timing of resetting and readout of pixel rows of an image sensor and illumination of an illumination device to produce an image frame with increased sensitivity, according to one embodiment. The image sensor includes N rows, and the resetting and readout of the first four rows and the last three rows are shown. An image frame capture period 702 and the very beginning of a subsequent image frame capture period 703 are shown.

[0148] At time 704, the first row 706 is read out and reset, beginning the integration period 710 of the image frame. Each row is read out and reset successively until the last row 708 is read out and reset at time 705. The period from the readout / reset of the first row to the readout / reset of the last row is the readout time 712 (also referred to herein as the reset time).

[0149] At time 711, the reset period and vertical blanking period 717 are complete, so the imager is ready for another reset / readout operation. However, this step is skipped so that the integration period for each row is longer. This readout / reset capability is indicated by dashed line 713. At time 714, the first row 706 is readout (and reset), providing intensity values ​​for the pixels of the first row 706 related to the amount of light received by the pixels of the first row during the integration period 710 of the first row 706. The readout of the first row 706 occurs (relatively) long after the reset of the last row 508. Each row is readout (and reset) successively until the last row 708 is readout (and reset) at time 734. Each row has the same amount of integration time 716. Due to skipping the readout / reset at time 714, the integration period for each row can effectively double.

[0150] In the illustrated example, the period of time between the reset of the last row 708 and the readout of the first row 706 is equal to the readout time plus twice the vertical blanking period 717. During this time, substantially all rows of the imager are exposed simultaneously, effectively creating a global shutter window through which illumination light can be provided so that rolling shutter effects are prevented. In an exemplary embodiment, illumination light is provided during the entire global shutter window. An exemplary illumination control signal 718 is shown at the bottom of FIG. 7. The start time 722 of the illumination period 720 may be the same as the readout / reset time 705 of the last row 708. The end time 724 of the illumination period 720 may be the same as the readout / reset time 714 of the first row 706. The illuminator is off outside of this illumination window. The illumination period may be shorter than the global shutter period, i.e., it may start sometime after the readout of the last row and / or end before the readout of the first row.

[0151] According to some embodiments, the imager may be configured with any suitable frame rate. Exemplary frame rates include at least 30 fps, at least 60 fps, at least 120 fps, and at least 240 fps. The global shutter window time and / or extended vertical blanking time are generally related to the frame rate capability of the imager, whereby a faster frame rate will be associated with a shorter global shutter window time and / or a shorter extended vertical blanking time.

[0152] Illumination device for pulse-width modulated illumination An imaging system, such as the imaging system 300 of FIG. 3, may be configured to adjust the amount of illumination light provided to a target object (e.g., target tissue) by pulsing an illumination light source of an illuminator, as described above. The amount of illumination light can be adjusted so that the amount of reflected light (or fluorescent light) received at the imaging sensor can be controlled. Pulse-width modulation (PWM) allows the amount of light received at the target object to be rapidly varied, such that the amount of light received at the imager can be rapidly adjusted, such as from one imaging frame to the next. In some embodiments, PWM of the light source can replace other means of adjusting light intensity at the imaging sensor, such as a mechanical shutter. In some embodiments, PWM of the light source can provide a primary mechanism for adjusting light at the sensor, with the mechanical shutter providing a secondary mechanism if light source adjustment is insufficient. The imager may also rely on gain adjustment in addition to PWM adjustment of the illuminator during the generation of an image frame.

[0153] FIG. 8 is a block diagram illustrating components of an exemplary lighting device 800 that can be used to generate pulse-width-modulated illumination light, according to one embodiment. The lighting device 800 can be configured to generate not only white light but also fluorescent excitation light, such as infrared light. The lighting device 800 includes a laser diode 830, a first LED 832, a second LED 834, and a third LED 836. The laser diode 830 can be for providing the fluorescent excitation light, and the three LEDs can be for providing visible light, such as red light, green light, blue light, and any combination thereof (e.g., white light). The laser diode 830 is driven by a laser diode driver 838. The first LED 832 is driven by a first LED driver 840, the second LED 834 is driven by a second LED driver 842, and the third LED 836 is driven by a third LED driver 844.

[0154] In some embodiments, the laser diode 830 is an infrared diode that emits light having a wavelength in the range of about 805 nm to about 810 nm. In some embodiments, the laser diode emits light having a wavelength of about 808 nm. Preferably, the first LED 832 emits light in the blue wavelength spectrum, the second LED 834 emits light in the green wavelength spectrum, and the third LED 836 emits light in the red wavelength spectrum.

[0155] In front of the laser diode 830 is a first dichroic filter 850, in front of the first LED 832 is a second dichroic filter 852, and in front of both the second LED 834 and the third LED 836 is a third dichroic filter 854. The dichroic filters 850, 852, and 854 are each designed to reflect certain light and allow other light to pass. The first dichroic filter 850 reflects the laser light from the laser diode 830 while allowing light from all three LEDs 832, 834, and 836 (e.g., light in the blue, green, and red wavelength spectrums) to pass. The second dichroic filter 852 reflects light from the first LED 832 while allowing light from the second and third LEDs 834 and 836 to pass. The third dichroic filter 854 reflects light from the second LED 834 while allowing light from the third LED 836 to pass.

[0156] A first optical lens 866 is positioned between the first dichroic filter 850 and the second dichroic filter 852 to collect light received from the second dichroic filter 852 and pass it through the first dichroic filter 850. A second optical lens 868 is positioned between the second dichroic filter 852 and the third dichroic filter 854 to collect light received from the third dichroic filter 854 and pass it through the second dichroic filter 852. A third optical lens 870 may be provided to collect light received from the first dichroic filter 850. Light may be supplied by the illumination device 800 via an optical transmission line 880, such as a fiber optic light pipe.

[0157] A controller 864 is provided for driving and adjusting the illumination source through various modes and control signals that may be received, for example, from a camera control unit, via control line 810. An exemplary illumination mode, according to various embodiments, includes a visible white light mode in which laser diode 830 is off, first LED 832 is on, second LED 834 is on, and third LED 836 is on. This mode may be used, for example, in method 600 of FIG. 6 described above.

[0158] Another exemplary illumination mode is an alternating infrared excitation light and white light mode that may be used in generating alternating white light and fluorescent light image frames, such as in method 400 of Figure 4. In this mode, the first LED 832, the second LED 834, and the third LED 836 are periodically turned on simultaneously, and the laser diode 830 is turned on all the time or periodically.

[0159] Switching from mode to mode, such as from white light mode to white light plus fluorescent light illumination mode, in both the illumination device 800 and the imager can be accomplished by use of a camera head button, a CCU touchscreen, a light source touchscreen, a wireless controller touchscreen, voice control, a foot pedal, or any other suitable mechanism.

[0160] The controller 864 may be configured to control the pulsing of individual light sources within the lighting device 800. The controller 864 may provide pulse control signals to the illumination source drivers, causing the drivers to activate and deactivate the light sources with the desired pulse-width modulation. In some embodiments, the controller 864 receives a PWM signal from an external control system, such as a camera control system, via control line 810 and pulses the light sources using the PWM signal. In other words, a waveform corresponding to the desired PWM of the light source may be provided to the lighting device 800 via control line 810, and this PWM waveform may be provided to the driver via the controller without software-based processing of the PWM signal. In other embodiments, the controller 864 receives a control signal from an external control system and generates its own PWM signal according to the received control signal. For example, an external controller, such as a CCU, may provide relative intensity control parameters, and the controller 864 may generate a PWM signal according to the intensity control parameters.

[0161] Exemplary Imager FIG. 9 shows an exemplary rolling shutter imager 916 including three imaging sensors. The imager 916 may be used as the camera head 16 of the system 10 of FIG. 1 or as the imager 302 of the system 300. Light reaches the imager via a path 988. The imager 916 includes, among other components not shown, a trichroic prism 990. The trichroic prism 990 includes a first glass prism 992, a second glass prism 994, and a third glass prism 996. Between the first glass prism 992 and the second glass prism 994 is a first prism filter 998, which may be coated on the outside of the first glass prism 992. The first prism filter 998 reflects blue light but transmits other light, such as red, green, and infrared light. Between the second glass prism 994 and the third glass prism 996 is a second prism filter 900, which may be in the form of a coating on the second glass prism 994. The second prism filter 900 reflects red and infrared light but allows the transmission of other wavelengths of light, such as green light. Thus, blue light is transmitted along path 902, red and infrared light is transmitted along path 904, and green light is transmitted along path 906.

[0162] A first color sensor 908 is adjacent to and preferably fixedly mounted on the first glass prism 992. A second color sensor 910 is adjacent to and preferably fixedly mounted on the second glass prism 994. A third color sensor 912 is adjacent to and preferably fixedly mounted on the third glass prism 996. The first color sensor 908 is capable of detecting light in the blue wavelength spectrum, the second color sensor 910 is capable of detecting light in the red and infrared wavelength spectrum, and the third color sensor 912 is capable of detecting light in the green wavelength spectrum. The color sensors 908, 910, 912 are rolling shutter imagers, such as CMOS imagers, that receive light and convert the light into electronic signals that are, in turn, transmitted to a processor (e.g., the CCU 18 of the system 10) and processed into analog or digital signals that are provided to a display, such as the monitor 20 of the system 10.

[0163] Optionally, the imaging system is a fluorescence imaging system. FIG. 10 is a schematic example of a fluorescence imaging system 1010. The fluorescence imaging system 1010 includes a light source 1012 that illuminates a subject's tissue to induce fluorescence emission from a fluorescent contrast agent 1014 in the subject's tissue (e.g., blood, urine, lymph, cerebrospinal fluid, or other bodily fluid or tissue), an image acquisition assembly 1016 arranged to generate time-series and / or subject time-series fluorescence images from the fluorescence emission, and a processor assembly 1018 arranged to process the generated time-series / subject time-series fluorescence images according to any of the variations of the methods described herein. The processor assembly 1018 may include a memory 1068 having instructions thereon, a processor module 1062 arranged to execute the instructions on the memory 1068 to process the time-series and / or subject time-series fluorescence images, and a data storage module 1064 for storing raw and / or processed time-series and / or subject time-series fluorescence images. In some variations, the memory 1068 and the data storage module 1064 may be embodied in the same storage medium, while in other variations, the memory 1068 and the data storage module 1064 may be embodied in different storage media. The system 1010 may further include a communications module 1066 for transmitting image and other data, such as some or all of the time-series / subject time-series fluoroscopic images or other input data, spatial maps, subject spatial maps, and / or tissue quantifiers, to the imaging data processing hub.

[0164] In some variations, the light source 1012 includes, for example, an illumination module 1020. The illumination module 1020 may include a fluorescence excitation source arranged to generate excitation light having an appropriate intensity and an appropriate wavelength for exciting the fluorescent imaging agent 1014. As shown in FIG. 11 , the illumination module 1020 may include a laser diode 1022 (which may include, for example, one or more fiber-coupled diode lasers) arranged to provide excitation light for exciting the fluorescent imaging agent (not shown) in the subject's tissue. Examples of other sources of excitation light that may be used in various embodiments include one or more LEDs, arc lamps, or other illumination technologies having sufficient intensity and an appropriate wavelength to excite the fluorescent imaging agent in the tissue. For example, excitation of a fluorescent imaging agent in blood, where the fluorescent imaging agent is a fluorescent dye with near-infrared excitation and emission characteristics, may be performed using one or more 793 nm, conductively cooled, single-bar, fiber-coupled laser diode modules from DILAS Diode Lasers GmbH, Germany.

[0165] In some variations, light output from the light source 1012 may be projected through one or more optical elements to shape and direct the output used to illuminate the tissue area of ​​interest. The optical elements may include one or more lenses, light guides, and / or diffractive elements to ensure a flat field across substantially the entire field of view of the image acquisition assembly 1016. The fluorescent excitation source may be selected to emit at a wavelength near the absorption maximum of the fluorescent contrast agent 1014 (e.g., indocyanine green (ICG)). For example, as shown in FIG. 11 , output 1024 from a laser diode 1022 may pass through one or more focusing lenses 1026 and then through a homogenizing light pipe 1028, such as a light pipe commonly available from Newport Corporation, USA. Finally, the light may pass through an optical diffractive element 1032 (i.e., one or more optical diffusers), such as a ground-glass diffractive element also available from Newport Corporation, USA. Power to the laser diode 1022 may be provided by a high-current laser driver, such as those available from Lumina Power, Inc., USA. The laser may optionally be operated in a pulsed mode during the image acquisition process. An optical sensor, such as a solid-state photodiode 1030, may be incorporated into the illumination module 1020 and may sample the illumination intensity produced by the illumination module 1020 via scattered or diffuse reflection from various optical elements. In some variations, an additional illumination source may be used to provide guidance in aligning and positioning the module over the region of interest.

[0166] Referring again to FIG. 10 , in some variations, the image acquisition assembly 1016 may be a component of a fluorescence imaging system 1010 configured to acquire time-series fluorescence images from the fluorescence emission from the fluorescent contrast agent 1014 and / or a subject time-series fluorescence image. The image acquisition assembly 1016 may include a camera module 1040. As shown in FIG. 12 , the camera module 1040 may acquire images of the fluorescence emission 1042 from the fluorescent contrast agent in the tissue by collecting and focusing the fluorescence emission onto the image sensor assembly 1044 using a system of imaging optics (e.g., 1046 a, 1046 b, 1048, and 1050). The imaging sensor assembly 1044 may include at least one 2D solid-state image sensor. The solid-state image sensor may be a charge-coupled device (CCD), a CMOS sensor, a CID, or similar 2D sensor technology. The electrical charges resulting from the light signals converted by the imaging sensor assembly 1044 are converted into electrical video signals, including both digital and analog video signals, by appropriate readout and amplification electronics within the camera module 1040.

[0167] According to an exemplary variation of the fluorescence imaging system, the light source may provide an excitation wavelength of about 800 nm + / - 10 nm, and the image acquisition assembly uses an emission wavelength of >820 nm with NIR-compatible optics, e.g., for ICG fluorescence imaging. In an exemplary embodiment, the NIR-compatible optics may include a CCD monochrome imaging sensor with a GigE standard interface and a lens that is compatible with the sensor in terms of optical format and mount format (e.g., C / CS mount).

[0168] In some variations, the processor module 1062 comprises any computer or computing means, such as, for example, a tablet, laptop, desktop, network computer, or dedicated standalone microprocessor. For example, the processor module 1062 may include one or more central processing units (CPUs). In an exemplary embodiment, the processor module 1062 is a quad-core 2.5 GHz processor having four CPUs, each of which is a microprocessor such as a 64-bit microprocessor (e.g., commercially available as an INTEL Core i3, i5, or i7, or as an AMD Core FX series). However, in other embodiments, the processor module 1062 may be any suitable number of CPUs and / or any other suitable processor having a suitable clock speed.

[0169] Input for the processor module 1062 may be obtained, for example, from the imaging sensor 1044 of the camera module 1040 shown in FIG. 12, from the solid-state photodiode 1030 of the illumination module 1020 in FIG. 11, and / or from any external control hardware such as a foot switch or remote control. Output is provided to a laser diode driver and optical alignment aid. As shown in FIG. 10, in some variations, the processor assembly 1018 may include a data storage module 1064 capable of storing time-series / subject time-series images, or data representative thereof, or other input data, in a tangible, non-transitory, computer-readable medium, such as internal memory (e.g., a hard disk or flash memory), to enable recording and processing of acquired data. In some variations, the processor module 1062 may include an internal clock to enable control of various elements and ensure precise timing of illumination and sensor shutters. In some variations, the processor module 1062 may also provide user input and graphical displays. The fluorescence imaging system may optionally be configured with a communications unit 1066, such as a wired or wireless network connection or a video output connection, for transmitting the time series of fluorescence images as they are acquired or for playback after recording. The communications unit 1066 may additionally or alternatively transmit processed data such as spatial maps, object space maps, and / or tissue values.

[0170] 10-12 , the subject is positioned relative to the fluorescence imaging system 1010 so that the region of interest (e.g., a target tissue region) is located below the light source 1012 and acquisition assembly 1016, and so that the illumination module 1020 of the light source 1012 generates a substantially uniform field of illumination across substantially the entire region of interest. In some variations, an image of the region of interest may be acquired for background subtraction purposes before the fluorescent imaging agent 1014 is administered to the subject or before the fluorescent imaging agent reaches the region of interest. To acquire the fluorescence images / subject fluorescence images, the operator of the fluorescence imaging system 1010 may initiate acquisition of the time-series / subject time-series fluorescence images by pressing a remote switch or foot control or via a keyboard (not shown) connected to the processor assembly 1018. As a result, the light source 1012 is turned on, and the processor assembly 1018 begins recording the fluorescence image data / subject fluorescence image data provided by the image acquisition assembly 1016. When operating in pulsed mode in this embodiment, the imaging sensor 1044 of the camera module 1040 is synchronized to collect the fluorescent emission following the laser pulse generated by the diode laser 822 of the illumination module 1020. In this way, maximum fluorescent emission intensity is recorded and the signal-to-noise ratio is optimized. In this embodiment, a fluorescent contrast agent 1014 is administered to the subject and delivered to the region of interest via arterial flow. Acquisition of time-series / non-analyte time-series fluorescent images begins, for example, immediately after administration of the fluorescent contrast agent 1014, and time-series fluorescent images are acquired from substantially the entire region of interest during the infusion of the fluorescent contrast agent. The fluorescent emission from the region of interest is collected by the collection optics of the camera module 1040. Residual ambient light and reflected excitation light are attenuated by subsequent optical elements of the camera module 1040 (e.g., optical element 1050 of FIG. 12, which may be a filter), so that the fluorescent emission can be acquired by the imaging sensor assembly 1044 with minimal interference from light from other light sources.

[0171] In some variations, following acquisition or generation of the time-series / subject-time-series fluorescence images, the processor assembly 1018 (e.g., processor module 1062 or another processor) may then be initiated to execute instructions stored in memory 1068 to process the imaging data before transmission to an imaging data processing system (e.g., hub 102 of system 100). System 1010 may transmit via connection 1066 the spatial maps / subject-space maps and / or any clinical correlations or diagnoses derived therefrom for display to a user in a composite display feed, e.g., as grayscale or colorized images, and / or for storage for subsequent use.

[0172] A computer program product, or tangible, non-transitory computer-readable medium having computer-executable (readable) program code embedded therein, can provide one or more processors with instructions that, when executed, cause the processor to perform one or more of the methods described herein. The program code can be written in any suitable programming language and can be delivered to the processor in many forms, including, but not limited to, persistently stored on a non-writable storage medium (e.g., a read-only memory device such as a ROM, CD-ROM disk, etc.), mutably stored on a writable storage medium (e.g., a hard drive, etc.), transmitted to the processor via a communications medium such as a local area network, a public network such as the Internet, or any type of medium suitable for storing electronic instructions. When carrying computer-readable instructions that implement various embodiments of the methods described herein, such computer-readable medium represents an example of various embodiments. In various embodiments, tangible, non-transitory computer-readable medium includes all computer-readable media, and the scope of the present invention is limited to computer-readable media, whether the media is tangible or non-transitory.

[0173] A kit may include any portion of a system described herein and a fluorescent imaging agent, e.g., a fluorescent dye such as ICG or any suitable fluorescent imaging agent. In further embodiments, a kit may include a tangible, non-transitory computer-readable medium having computer-executable (readable) program code embedded therein, the computer-readable medium being capable of providing instructions to one or more processors, when executed, for performing one or more methods for tissue characterization and / or clinical data prediction described herein. The kit may include instructions for use of at least some of the components (e.g., for using a fluorescent imaging agent, for installing computer-executable (readable) program code having embedded instructions, etc.). In yet another embodiment, a fluorescent imaging agent, e.g., a fluorescent dye, for use in the methods and systems described herein is provided. In further variations, a kit may include any portion or all of a system described herein and a fluorescent agent, e.g., a fluorescent dye such as ICG or any other suitable fluorescent agent or combination of fluorescent agents.

[0174] Examples of contrast agents used to generate imaging data According to some embodiments, in fluorescent medical imaging applications, the contrast agent is a fluorescent contrast agent, such as, for example, an ICG dye. The fluorescent contrast agent, such as ICG, may be pre-administered to the subject prior to performing the methods described herein. When administered to a subject, ICG binds to blood proteins and circulates with the blood within tissues. The fluorescent contrast agent (e.g., ICG) may be administered to the subject as a bolus injection (e.g., into a vein or artery) at a concentration suitable for imaging, such that the bolus circulates through the vasculature and passes through the microvasculature. In other embodiments in which multiple fluorescent contrast agents are used, such agents may be administered simultaneously, for example, in a single bolus, or sequentially in separate boluses. The fluorescent contrast agent may be pre-administered to the subject prior to performing the methods described herein. In some embodiments, the fluorescent contrast agent may be administered via a catheter. In certain embodiments, the fluorescent contrast agent may be administered less than one hour before making measurements of signal intensity resulting from the fluorescent contrast agent. For example, the fluorescent contrast agent may be administered to the subject less than 30 minutes before making measurements. In yet other embodiments, the fluorescent imaging agent may be administered at least 30 seconds before the measurement is taken, hi yet other embodiments, the fluorescent imaging agent may be administered simultaneously with the measurement being taken.

[0175] According to some embodiments, the fluorescent imaging agent may be administered at various concentrations to achieve a desired circulating concentration in the blood. For example, in an embodiment where the fluorescent imaging agent is ICG, it may be administered at a concentration of about 2.5 mg / mL to achieve a circulating concentration of about 5 μM to about 10 μM in the blood. In various embodiments, the upper concentration limit for administration of the fluorescent imaging agent is the concentration at which the fluorescent imaging agent becomes clinically toxic in the circulation, and the lower concentration limit is the instrument's limit for acquiring signal intensity data resulting from the fluorescent imaging agent circulating with the blood to detect the fluorescent imaging agent. In various other embodiments, the upper concentration limit for administration of the fluorescent imaging agent is the concentration at which the fluorescent imaging agent self-quenches. For example, the circulating concentration of ICG may range from about 2 μM to about 10 mM. Thus, in one aspect, a method includes administering a contrast agent (e.g., a fluorescent imaging agent) to a subject and acquiring signal intensity data (e.g., images) prior to processing the signal intensity data according to various embodiments. In another aspect, the method excludes any step of administering an imaging agent to the subject.

[0176] According to some embodiments, fluorescent contrast agents suitable for use in fluorescence imaging applications to generate fluorescence image data are contrast agents that circulate with blood (e.g., fluorescent dyes capable of circulating with blood components such as lipoproteins or serum plasma in blood), can pass through the vasculature (i.e., macrovasculature and microvasculature) of tissues, and generate signal intensity when the contrast agent is exposed to appropriate light energy (e.g., excitation light energy or absorption light energy). In various embodiments, the fluorescent contrast agent includes a fluorescent dye, an analog thereof, a derivative thereof, or a combination thereof. The fluorescent dye includes any non-toxic fluorescent dye. In certain embodiments, the fluorescent dye optimally emits fluorescence in the near-infrared spectrum. In certain embodiments, the fluorescent dye is or includes a tricarbocyanine dye. In certain embodiments, the fluorescent dye is or includes ICG, methylene blue, or a combination thereof. In other embodiments, the fluorescent dye is or includes fluorescein isothiocyanate, rhodamine, phycoerythrin, phycocyanin, allophycocyanin, orthophthalaldehyde, fluorescamine, rose bengal, trypan blue, fluorogold, or a combination thereof, excitable using excitation light wavelengths appropriate for each dye. In some embodiments, analogs or derivatives of fluorescent dyes may be used. For example, fluorescent dye analogs or derivatives include fluorescent dyes that have been chemically modified but retain the ability to fluoresce when exposed to light energy of the appropriate wavelength.

[0177] In various embodiments, the fluorescent imaging agent may be provided as a lyophilized powder, lyophilized solid, or liquid. In certain embodiments, the fluorescent imaging agent may be provided in a vial (e.g., a sterile vial) that can be reconstituted to the appropriate concentration by administering a sterile fluid with a sterile syringe. Reconstitution may be performed using any suitable carrier or diluent. For example, the fluorescent imaging agent may be reconstituted with an aqueous diluent immediately prior to administration. In various embodiments, any diluent or carrier that maintains the fluorescent imaging agent in solution may be used. As an example, ICG may be reconstituted with water. In some embodiments, once the fluorescent imaging agent is reconstituted, it may be mixed with additional diluents or carriers. In some embodiments, the fluorescent imaging agent may be conjugated to another molecule, such as a protein, peptide, amino acid, synthetic polymer, or sugar, to enhance, for example, solubility, stability, imaging properties, or a combination thereof. Additional buffers, including Tris, HCl, NaOH, phosphate buffer, and / or HEPES, may optionally be added.

[0178] Although fluorescent contrast agents are described in detail above, those skilled in the art will appreciate that other contrast agents may be used in connection with the systems, methods, and techniques described herein, depending on the optical imaging modality. Such fluorescent agents may be administered to bodily fluids (e.g., lymphatic fluid, cerebrospinal fluid) or bodily tissues.

[0179] In some variations, the fluorescent imaging agents used in combination with the methods, systems, and kits described herein may be used for blood flow imaging, tissue perfusion imaging, lymphatic imaging, or a combination thereof, which may be performed before, during, or after an invasive surgical procedure, a minimally invasive surgical procedure, a non-invasive surgical procedure, or a combination thereof. The method of imaging a subject's tissue or the method of operating an electronic rolling shutter imager may itself exclude any invasive surgical steps. The method of blood flow imaging, tissue perfusion imaging, lymphatic imaging, or a combination thereof may itself exclude any invasive surgical steps. Examples of invasive surgical procedures involving blood flow and tissue perfusion include cardiac-related surgical procedures (e.g., on-pump or off-pump CABG) and reconstructive surgical procedures. Examples of non-invasive or minimally invasive procedures include the treatment and / or management of wounds (e.g., chronic wounds such as pressure ulcers). In this regard, changes in a wound over time, such as changes in wound dimensions (e.g., diameter, area) or changes in tissue perfusion in and / or around the wound, can be tracked over time by application of the methods and systems. Examples of lymphatic imaging include identification of one or more lymph nodes, lymph node drainage, lymphatic system mapping, or combinations thereof. In some variations, such lymphatic system imaging can relate to the female reproductive system (e.g., uterus, cervix, vulva).

[0180] In a variation for cardiac applications, contrast agents (e.g., ICG alone or in combination with other contrast agents) may be injected intravenously, for example, through a central venous circuit, bypass pump, and / or cardioplegic circuit, to flow and / or perfuse the coronary vasculature, microvasculature, and / or graft. ICG may be administered below the grafted vessel as a diluted ICG / blood / saline solution so that the final concentration of ICG in the coronary artery is approximately the same as or less than that obtained by injecting approximately 2.5 mg (i.e., 1 ml of 2.5 mg / ml) into the central circuit or bypass pump. ICG may be prepared, for example, by dissolving 25 mg of the solid in 10 ml of sterile aqueous solvent, which may be provided with the ICG by the manufacturer. One milliliter of ICG solution may be mixed with 500 ml of sterile saline (e.g., injecting 1 ml of ICG into a 500 ml saline bag). Thirty milliliters of diluted ICG / saline solution can be added to 10 ml of subject blood obtained aseptically from a central arterial circuit or bypass pump. ICG in the blood binds to plasma proteins, preventing extravasation. Mixing of the ICG and blood can be performed in a sterile surgical field using standard sterile techniques. For each graft, 10 ml of the ICG / saline / blood mixture can be administered. Rather than administering ICG by injecting it through the graft wall using a needle, ICG can be administered using a syringe attached to the (open) proximal end of the graft. When a graft is harvested, surgeons routinely attach an adapter to the proximal end of the graft so that a saline-filled syringe can be attached, the distal end of the graft can be sealed, and saline can be injected underneath the graft to pressurize the graft and assess the integrity of the conduit (for leaks, side branches, etc.) before performing the first anastomosis. In other variations, the methods, dosages, or combinations thereof described herein in connection with cardiac imaging may be used in any vascular and / or tissue perfusion imaging application.

[0181] Lymphatic mapping is an important part of effective surgical staging for cancers that spread via the lymphatic system (e.g., breast, gastric, and gynecological cancers). Removal of multiple nodes from a specific nodal basin can lead to serious complications, including acute or chronic lymphedema, paresthesia, and / or seroma formation. However, in practice, if sentinel lymph nodes are negative for metastasis, surrounding lymph nodes are most likely negative as well. For example, during breast cancer surgery, identification of tumor-draining lymph nodes (LNs) has become a critical step in staging cancers that spread via the lymphatic system. LN mapping involves the use of dyes and / or radioactive tracers to identify LNs for either biopsy or resection, followed by pathological evaluation of metastases. The goal of lymph node resection during surgical staging is to identify and remove LNs that are at high risk for local cancer spread. Sentinel lymph node (SLN) mapping has emerged as an effective surgical strategy in the treatment of breast cancer. It is generally based on the concept that if metastasis (spread of cancer to the axillary LNs) is observed, it should be located in the SLN, which is defined in the art as the node group to which cancer cells are most likely to spread from the first LN or primary tumor. If the SLN is negative for metastasis, the surrounding secondary and tertiary LNs should also be negative. The main advantage of SLN mapping is that it reduces the number of subjects who undergo traditional partial or complete lymphadenectomy and, consequently, the number of subjects who suffer from concomitant morbidity such as lymphedema and lymphocysts.

[0182] The current standard of care for SLN mapping involves the injection of a tracer to identify lymphatic drainage pathways from the primary tumor. The tracer used can be a radioisotope (e.g., technetium-99 or Tc-99m) for intraoperative localization using a gamma probe. Radiotracer techniques (known as scintigraphy) are limited to hospitals with access to radioisotopes, require the involvement of a nuclear medicine physician, and do not provide real-time visual guidance. The colored dye isosulfan blue has also been used, but this dye cannot be seen through skin and fatty tissue. Furthermore, blue staining can result in breast tattooing that lasts for months, subcutaneous injection can cause skin necrosis, and rare allergic reactions, including anaphylaxis, have been reported. Severe anaphylactic reactions have occurred after isosulfan blue injection (approximately 2% of patients). Symptoms include dyspnea, shock, angioedema, urticaria, and pruritus. Reactions are more likely in subjects with a history of bronchial asthma or an allergic or drug reaction to triphenylmethane dyes. Isosulfan blue is known to interfere with measurements of oxygen saturation by pulse oximetry and methemoglobin by gas analyzers. Use of isosulfan blue may result in transient or long-lasting (tattoo-like) blue staining.

[0183] In contrast, fluorescence imaging according to various embodiments for use in visualizing and mapping SLNs facilitates direct, real-time visual identification of LNs and / or afferent lymphatic vessels during surgery, facilitating real-time, high-resolution optical navigation through skin and adipose tissue, organization of blood flow, tissue perfusion, or a combination thereof.

[0184] In some variations, lymph node visualization, classification, or both during fluorescence imaging may be based on imaging with one or more contrast agents, which may further be based on visualization and / or classification using a gamma probe (e.g., technetium Tc-99m, a clear, colorless aqueous solution typically injected into the periareolar region as a standard of care), another conventionally used colored contrast agent (isosulfan blue), and / or other evaluations, such as histology. For example, a subject's breast may be injected twice with approximately 1% isosulfan blue (for comparison) and twice with an ICG solution having a concentration of approximately 2.5 mg / ml. The injection of isosulfan blue may precede the injection of ICG, or vice versa. For example, an anesthetized subject may be injected with 0.4 ml (0.2 ml at each site) of isosulfan blue into the periareolar region of the breast using a TB syringe and a 30G needle. The subject may be injected at the 12 and 9 o'clock positions for the right breast and at the 12 and 3 o'clock positions for the left breast. The total dose of intradermal isosulfan blue injection into each breast may be approximately 4.0 mg (0.4 ml of a 1% solution: 10 mg / ml). In another exemplary variation, the subject may receive an ICG injection first, followed by isosulfan blue (for comparison). A 25 mg vial of ICG may be reconstituted with 10 ml of sterile water for injection to obtain a 2.5 mg / ml solution immediately prior to ICG administration. Using a TB syringe and a 30G needle, for example, the subject may be injected with approximately 0.1 ml of ICG (0.05 ml per site) into the periareolar region of the breast (injections may be performed at the 12 and 9 o'clock positions for the right breast and at the 12 and 3 o'clock positions for the left breast). The total dose of intradermal isosulfan blue injection into each breast may be approximately 0.25 mg (0.1 ml of a 2.5 mg / ml solution) per breast. ICG can be injected at a rate of, for example, 5–10 seconds per injection. When ICG is injected intradermally, its protein-binding properties allow it to be rapidly taken up by lymphatics and transported through ducts to the LN. In some variations, ICG can be provided in the form of a sterile, lyophilized powder containing 5% or less sodium iodide and 25 mg of ICG.ICG can be packaged with an aqueous solvent consisting of sterile water for injection, which is used to reconstitute the ICG. In some variations, the ICG dosage (mg) for breast cancer sentinel lymphatic mapping can range from about 0.5 mg to about 10 mg, depending on the route of administration. In some variations, the ICG dosage can be about 0.6 mg to about 0.75 mg, about 0.75 mg to about 5 mg, or about 5 mg to about 10 mg. The route of administration can be, for example, subcutaneous, intradermal (e.g., periareolar), subareolar, skin overlying the tumor, intradermal into the areola closest to the tumor, subcutaneous into the areola, intradermal over the tumor, periareolar throughout the breast, or a combination thereof. Injection can occur prior to visualization and / or classification. NIR fluorescence-positive LNs (e.g., using ICG) can be represented, for example, as black-and-white NIR fluorescence images and / or full or partial color (white light) images, full or partially desaturated white light images, enhanced color images, overlays (e.g., fluorescence with any other image), composite images (e.g., fluorescence incorporated into other images) that may have various colors, various desaturation levels, or various ranges of colors to highlight / visualize particular features of interest. Further processing of the images can be performed for further visualization and / or other analysis (e.g., quantification). In accordance with the American Society of Breast Surgeons (ASBrS) clinical practice guidelines for SLN biopsies in breast cancer patients, lymph nodes and lymphatic vessels can be visualized (e.g., intraoperatively, in real time) using the fluorescence imaging systems and methods according to various embodiments, using ICG and SLNs alone or in combination with a gamma probe (Tc-99m). Fluorescence imaging of the LNs can begin at the injection site by following the lymphatics to the axillary LNs. Once a visual image of the LNs is identified, the skin is incised and LN mapping and identification can be performed until the ICG-visualized node is identified. This method of LN mapping itself can eliminate any surgical steps. For comparison, mapping using isosulfan blue can be performed until a "blue" node is identified.Identified LNs can be labeled for resection using ICG alone or in combination with another imaging technique (e.g., isosulfan blue and / or Tc-99m). Subjects can have various stages of breast cancer (e.g., IA, IB, IIA).

[0185] In some variations, for example, in gynecological cancers (e.g., uterine, endometrial, vulvar, and cervical malignancies), ICG can be administered intracellularly to visualize lymph nodes, lymphatic vessels, or a combination thereof. Upon intracellular injection, ICG's protein-binding properties allow it to be rapidly taken up by lymphatic fluid and transported through the ducts to the SLN. ICG can be provided for injection in the form of a sterile, lyophilized powder containing 25 mg of ICG (e.g., 25 mg / vial) and 5.0% or less sodium iodide. The ICG can then be reconstituted with commercially available water for injection (sterile) prior to use. According to one embodiment, a vial containing 25 mg of ICG can be reconstituted with 20 ml of water for injection, resulting in a 1.25 mg / ml solution. A total of 4 ml of this 1.25 mg / ml solution is injected into the subject (4 x 1 ml injections), for a total dose of 5 mg of ICG per subject. Alternatively, 1 ml of a 1% 10 mg / ml solution of isosulfan blue (for comparison purposes) can be injected into the cervix four times for a total dose of 40 mg. The injections can be performed while the subject is under anesthesia in an operating room. In some variations, the ICG dosage (mg) for sentinel lymph node detection and / or mapping in gynecological cancers can range from about 0.1 mg to about 5 mg, depending on the route of administration. In some variations, the ICG dose can be from about 0.1 mg to about 0.75 mg, from about 0.75 mg to about 1.5 mg, from about 1.5 mg to about 2.5 mg, or from about 2.5 mg to about 5 mg. The route of administration can be, for example, cervical injection, perivulval injection, hysteroscopic endometrial injection, or a combination thereof. To minimize spillage of isosulfan blue or ICG, which can interfere with the mapping procedure when resecting LNs, mapping can be performed on the hemipelvis, and mapping with both isosulfan blue and ICG can be performed before resection of any LNs. LN mapping for clinical stage I endometrial cancer can be performed according to the NCCN Guidelines for Uterine Neoplasia: SLN Algorithm for Surgical Staging of Endometrial Cancer, and SLN mapping for clinical stage I cervical cancer can be performed according to the NCCN Guidelines for Cervical Neoplasia: Surgical / SLN Mapping Algorithm for Early Cervical Cancer.Therefore, identification of LNs may be based on ICG fluorescence imaging alone or in combination or co-administration with colorimetric dyes (isosulfan blue) and / or radiotracers.

[0186] Lymph node visualization can be qualitative and / or quantitative. Such visualization can include, for example, lymph node detection, detection rate, and anatomical distribution of lymph nodes. Lymph node visualization according to various embodiments can be used alone or in combination with other variables (e.g., vital signs, height, weight, demographics, surgical predictors, relevant medical history and underlying conditions, histological visualization and / or evaluation, Tc-99m visualization and / or evaluation, concomitant medications). Follow-up visits can occur on the day of discharge and thereafter (e.g., one month).

[0187] Because lymph contains high levels of proteins, ICG can bind to endogenous proteins upon entering the lymphatic system. When used in accordance with the methods and systems described herein, fluorescence imaging (e.g., ICG imaging) for lymphatic mapping offers the following exemplary advantages: high signal-to-background ratio (or tumor-to-background ratio) because NIR does not produce significant autofluorescence; real-time visualization features for lymphatic mapping; tissue definition (i.e., structural visualization); rapid excretion and clearance after entering the vasculature; and avoidance of non-ionizing radiation. Furthermore, NIR imaging has superior tissue penetration (approximately 5-10 mm of tissue) compared to visible light (1-3 mm of tissue). For example, the use of ICG facilitates visualization through the peritoneum overlying the para-aortic nodes. Tissue fluorescence can be observed for extended periods with NIR light but not with visible light, resulting in no impact on pathological evaluation or LN processing. Fluorescence is also easier to detect intraoperatively than blue lymph node staining (isosulfan blue). In other variations, the methods, dosages, or combinations thereof described herein in connection with lymphatic imaging may be used in any vascular and / or tissue perfusion imaging application.

[0188] Tissue perfusion relates to the microcirculatory flow of blood per unit tissue volume, which supplies oxygen and nutrients and removes waste products from the capillary bed of the perfused tissue. Tissue perfusion is related to, but distinct from, blood flow within blood vessels. Quantified blood flow through blood vessels can be expressed in terms that define flow (i.e., volume / time) or in terms that define velocity (i.e., distance / time). Tissue blood perfusion defines the movement of blood through the microvasculature, such as arterioles, capillaries, or venules, within a tissue volume. Quantified tissue blood perfusion can be expressed as blood flow through a tissue volume, i.e., blood volume / time / tissue volume (or tissue mass). Perfusion relates to nutrient vessels (e.g., microvessels known as capillaries), which include vessels involved in the exchange of metabolites between blood and tissue, rather than larger diameter non-nutrient vessels. In some embodiments, quantification of the target tissue may involve calculating or determining a parameter or quantity associated with the target tissue, such as velocity, size, volume, time, distance / time, and / or volume / time, and / or a change associated with any one or more of the foregoing parameters or quantities. However, compared to the movement of blood through larger diameter blood vessels, the movement of blood through individual capillaries is highly irregular, primarily due to vasomotion, and spontaneous oscillations in vascular tone manifest as pulsations in the movement of red blood cells. In some embodiments, the blood flow and tissue perfusion imaging described herein in connection with the systems and methods may be used to image tumor tissue and distinguish such tissue from other tissues.

[0189] The foregoing disclosure has been disclosed with reference to specific embodiments for purposes of illustration. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and disclosed in order to best explain the principles of the technology and their practical application. This will enable those skilled in the art to best utilize the technology and various embodiments with various modifications suited to the particular use contemplated. Although features may be described herein for clarity and conciseness as part of the same or separate embodiments, it will be understood that the scope of the present invention includes embodiments having all or any combination of the described features.

[0190] Although the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosure and examples as defined by the claims. Finally, the entire disclosures of the patents and publications referenced in this application are incorporated herein by reference.

Claims

1. 1. A method of operating a system including an illumination source and a control unit for imaging tissue of a subject using an electronic rolling shutter imager, the method comprising: the control unit sequentially resetting pixel rows of the rolling shutter imager from a first row to a last row, the sequential resetting from the first row to the last row lasting a first amount of time; the control unit causing the illumination source to illuminate the tissue of the subject with illumination light for an illumination period beginning when the last row of the rolling shutter imager is reset and lasting at least the first amount of time; accumulating charge in the rows of pixels of the rolling shutter imager for at least the illumination period based on light received from the tissue of the subject while the tissue of the subject is illuminated with the illumination light; When the illumination period ends, the control unit sequentially reads out the charges accumulated in the pixel rows from the first row to the last row; the control unit generating an image frame from readouts of charge accumulated in the pixel rows; A method comprising:

2. 10. The method of claim 1, wherein causing the illumination source to illuminate the tissue of the subject with illumination light comprises causing the illumination source to pulse the illumination light.

3. 3. The method of claim 1, further comprising the control unit controlling a pulse width of the pulsed illumination light based on a readout of charge accumulated in the pixel row during a previous frame.

4. 4. A method according to any one of claims 1 to 3, wherein each row of pixels accumulates charge over the illumination period.

5. 5. A method according to any one of claims 1 to 4, wherein each row of pixels accumulates charge for at least twice the first amount of time.

6. 6. The method according to any one of claims 1 to 5, wherein the illumination light is generated by at least one LED.

7. 7. The method of any one of claims 1 to 6, wherein the rolling shutter imager is part of an endoscopic imager.

8. 8. The method of claim 1, wherein the rolling shutter imager includes a mechanical shutter, the mechanical shutter remaining at least partially open from before the illumination period begins until after the illumination period ends.

9. 9. The method of claim 8, further comprising the control unit operating the mechanical shutter to reduce the amount of light received at the rolling shutter imager.

10. 10. The method of claim 1, further comprising the control unit adjusting a gain of the rolling shutter imager based on a readout of charge accumulated in the pixel row during a previous frame.

11. 1. A system for imaging tissue of a subject, the system comprising: an illumination source; an imaging device including an electronic rolling shutter imager, wherein the imaging device sequentially resetting pixel rows of the rolling shutter imager from a first row to a last row, wherein the sequential resetting from the first row to the last row lasts for a first amount of time; controlling the illumination source to illuminate the tissue of the subject with illumination light for an illumination period beginning when the last row of the rolling shutter imager is reset and lasting for at least the first amount of time; accumulating charge in the pixel row based on light received from the tissue of the subject while the tissue of the subject is illuminated with the illumination light, for at least the illumination period; When the illumination period ends, the charges accumulated in the pixel rows are read out sequentially from the first row to the last row; generating an image frame from readouts of the charges accumulated in said rows of pixels; A system characterized by being configured as follows.

12. 12. The system of claim 11, wherein the imaging device includes a camera control unit connected to an imaging head that includes the rolling shutter imager.

13. 13. The system of claim 12, wherein the illumination source is configured for pulse-width modulated illumination, and the camera control unit generates a pulse-width modulated waveform to control the illumination source.

14. 14. The system of claim 11, wherein the imaging device is configured to control the illumination source to illuminate the tissue of the subject with pulsed illumination light.

15. 15. The system of claim 14, wherein the imaging device is configured to control the illumination source to control a pulse width of the pulsed illumination light based on a readout of charge accumulated in the pixel row during a previous frame.

16. 16. A system according to any one of claims 11 to 15, wherein the imager is configured such that each row of pixels accumulates charge over the illumination period.

17. 17. A system according to any one of claims 11 to 16, wherein the imager is configured such that each row of pixels accumulates charge for at least twice the first amount of time.

18. 18. The system of any one of claims 11 to 17, wherein the illumination source comprises at least one LED.

19. 19. The system of any one of claims 11 to 18, wherein the imaging device comprises an endoscopic imager.

20. 20. The system of claim 11, wherein the rolling shutter imager includes a mechanical shutter, and the imaging device is configured to control the mechanical shutter such that the mechanical shutter remains at least partially open from before the illumination period begins until after the illumination period ends.

21. 21. The system of claim 20, wherein the imager is configured to reduce the amount of light received at the rolling shutter imager by operating the mechanical shutter.

22. 22. The system of claim 11, wherein the imaging device is configured to adjust the gain of the rolling shutter imager based on a readout of charge accumulated in the pixel row during a previous frame.

Images