System and method for providing medical fluorescence imaging using a controlled fluorescence excitation illumination source

Abstract

This disclosure relates to medical imaging in general, and more specifically to a technique for providing medical fluorescence imaging using a controlled illumination source. An exemplary method includes illuminating a tissue of interest using a white light illumination source during a first illumination period, wherein the fluorescence excitation illumination source is off during the first illumination period, and generating a first set of imaging data; illuminating the tissue of interest using a fluorescence excitation illumination source during a second illumination period following the first illumination period, wherein the white light illumination source is off during the second illumination period, and generating a second set of imaging data; and generating one or more image frames based on the first set of imaging data and the second set of imaging data.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority of U.S. Provisional Application No. 63 / 494,989, filed Apr. 7, 2023, which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present disclosure generally relates to medical imaging, and more specifically, to techniques for providing medical fluorescence imaging using a regulated fluorescence excitation light source.

Background Art

[0003] Medical systems, devices or tools are utilized before, during, or after surgery for various purposes. In particular, medical imaging systems can be used to enable a surgeon to visualize a surgical site in open - field surgery and endoscopic surgery. For example, endoscopy in the medical field enables viewing of internal features of a patient's body without using traditional fully invasive surgery. An endoscopic imaging system incorporates an endoscope so that a surgeon can view the surgical site, and endoscopic tools enable non - invasive surgery of that site. An endoscope can be used with a camera system for processing the images received by the endoscope. An endoscopic camera system typically includes a camera head connected to a camera control unit (CCU) that processes input image data received from the image sensor of the camera and outputs the image data for display. The CCU can control an illumination device that generates illumination light provided to the imaged scene.

[0004] Various image sensors, including charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors, can be used in imaging systems (e.g., endoscopic imaging systems, open-field imaging systems). CCD configurations are generally more complex than CMOS sensor configurations, and CMOS sensors can be integrated into high-capacity wafer manufacturing facilities used for related technologies such as microprocessors and chipsets. As a result, for similar performance, CMOS sensors are often less expensive than CCDs. In addition to lower cost, common fabrication processes used to manufacture CMOS imagers allow for the integration of CMOS pixel arrays with other electronic devices such as clock drivers, digital logic, analog-to-digital converters, and other appropriate electronics onto a single circuit. The compact structure possible for CMOS imagers can also reduce space requirements and power consumption. CMOS imagers can also have higher sensitivity and offer higher video frame rates.

[0005] CMOS-based imagers can utilize an electronic rolling shutter for exposing pixels in a sensor array. In an electronic rolling shutter, pixel rows are cleared (or reset), exposed, and read out sequentially. During integration, the pixel rows are exposed to light energy, and each pixel generates a charge corresponding to the amount and wavelength of light impacting it. Because the rows are driven and read out sequentially, there is an elapsed time between the integration of the first row and the integration of the last row. Due to this elapsed time between the start of integration of the first row and the start of integration of subsequent rows, a CMOS imager with an electronic rolling shutter can capture video images with blur or other rolling shutter effects. CMOS-based imagers can also utilize a global shutter for exposing pixels in a sensor array. In a global shutter, all rows of pixels are exposed simultaneously (i.e., the start and end of exposure are the same), but readout can (usually) be sequential.

[0006] Medical imaging systems (e.g., endoscopic imaging systems, open-field imaging systems) may include both white light illumination sources and fluorescence excitation illumination sources. White light and fluorescence excitation illumination sources can be used to illuminate the tissue of interest to obtain different types of image data (e.g., fluorescence image frames, white light image frames, mixed image frames). In particular, fluorescence excitation illumination sources, such as infrared or blue light illumination sources, can provide illumination to the tissue being imaged and excite fluorophores (or fluorescent dyes) to produce fluorescence emission. In many medical imaging systems, the fluorescence excitation illumination source is configured to be always on during the imaging session to provide the maximum exposure time for the fluorescence image frame and the maximum fluorescence excitation intensity. However, this always-on configuration has several drawbacks. For example, when the tissue being imaged is close to the surgical imaging device illuminating it (e.g., at the minimum working distance), the fluorescence excitation intensity can become excessively high, causing strong fluorescence emission and consequently generating a strong fluorescence signal in the imaging device's camera. Strong fluorescence signals generated by luminescence can degrade the background / white light average brightness of image frames containing fluorescent components, potentially contaminating the image frame. For example, if the fluorescence signal is green, the entire scene within the image frame may appear green, thus creating a false fluorescence signal illusion within the imaging device's camera. This can be problematic for open-field imaging systems and endoscopic imaging systems. For instance, endoscopic cameras can approach tissues very closely and move rapidly around them during surgery, thus generating false fluorescence signals that could compromise surgical safety. [Overview of the project]

[0007] Examples of this disclosure include techniques for tuning the fluorescence excitation illuminator of an imaging system to improve the stability and consistency of the fluorescence signal. According to some examples of tuning schemes, the fluorescence excitation illuminator and the white light illuminator are configured to alternately provide illumination periods to the tissue of interest. According to some examples of tuning schemes, the fluorescence excitation illuminator is configured to provide illumination periods to the tissue of interest at a certain frequency, and the white light illuminator is configured to provide illumination during at least a portion of every other illumination period of the fluorescence excitation illuminator, and furthermore, the length of each illumination period of the fluorescence excitation illuminator may be dynamically adjusted based on the distance between the imaging device transmitting the illumination and the tissue. Both tuning schemes offer many technical advantages over the continuously-on fluorescence illumination scheme, as described below.

[0008] While some of the techniques are described in relation to specific types of imagers (e.g., rolling shutter imagers, global shutter imagers), it should be understood that the techniques can be applied to any type of imager. Furthermore, these techniques can be applied to non-surgical or non-medical uses.

[0009] An exemplary method for imaging a target tissue using an imaging system comprising a rolling shutter imager, a fluorescence excitation illumination source, and a white light illumination source includes illuminating the target tissue with the white light illumination source during a first illumination period to accumulate charge in the pixels of multiple rows of the rolling shutter imager, wherein the fluorescence excitation illumination source is off during the first illumination period and sequentially reading the first set of charge accumulated in the pixels of multiple rows from the first row to the last row in order to generate a first set of imaging data; illuminating the target tissue with the fluorescence excitation illumination source during a second illumination period following the first illumination period to accumulate charge in the pixels of multiple rows of the rolling shutter imager, wherein the white light illumination source is off during the second illumination period and sequentially reading the second set of charge accumulated in the pixels of multiple rows from the first row to the last row in order to generate a second set of imaging data; and generating one or more image frames based on the first set of imaging data and the second set of imaging data.

[0010] In some examples, a fluorescence-induced illumination source and a white light illumination source are configured to alternately provide illumination periods to the target tissue.

[0011] In some examples, a fluorescence-induced illumination source is configured to provide illumination periods at a first frequency, and a white light illumination source is configured to provide illumination periods at a second frequency.

[0012] In some examples, the first or second frequency corresponds to the frame rate of the rolling shutter imager.

[0013] In some cases, the first or second frequency is half the frame rate of the rolling shutter imager.

[0014] In some examples, each illumination period of a white light source is a variable configured so as not to exceed a first maximum value.

[0015] In some examples, the first maximum value is approximately 1 millisecond.

[0016] In some examples, the duration of each illumination period for the fluorescence excitation illumination source is fixed.

[0017] In some examples, each illumination period of a fluorescence-excited illumination source is a variable configured so as not to exceed a second maximum value.

[0018] In some examples, one or more image frames include white light image frames based on a first set of imaging data.

[0019] In some examples, one or more image frames include fluorescence image frames based on a second set of imaging data.

[0020] In some examples, one or more image frames include a mixed image frame based on a fluorescence image frame and a white light image frame.

[0021] In some examples, the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

[0022] In some examples, the mixed image frame is derived from coloring the white light image frame based on the fluorescence image frame.

[0023] In some examples, the method further includes adding one or more image frames to the video stream.

[0024] In some examples, the white light illumination source comprises an LED.

[0025] In some examples, the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

[0026] In some examples, the rolling shutter imager is part of an endoscope imager. The endoscope imager can be pre-inserted prior to the start of the imaging method.

[0027] In some examples, the rolling shutter imager comprises a CMOS sensor.

[0028] An exemplary method of imaging a target tissue using an imaging system comprising a rolling shutter imager, a fluorescence excitation light source, and a white light illumination source includes illuminating the target tissue using the white light illumination source during a first illumination period and using the fluorescence excitation light source during a second illumination period to accumulate charge in pixels of a plurality of rows of the rolling shutter imager; sequentially reading out a first set of charge accumulated in pixels of the plurality of rows from a first row to a last row of the plurality of rows to generate a first set of imaging data; illuminating the target tissue using the fluorescence excitation light source during a third illumination period after the second illumination period to accumulate charge in pixels of the plurality of rows of the rolling shutter imager, wherein the white light illumination source is off during the second illumination period; sequentially reading out a second set of charge accumulated in pixels of the plurality of rows from the first row to the last row of the plurality of rows to generate a second set of imaging data; and generating one or more image frames based on the first set of imaging data and the second set of imaging data.

[0029] In some examples, the fluorescence excitation light source is configured to provide illumination periods to the target tissue at a first frequency, and the white light illumination source is configured to provide illumination during at least a portion of every other illumination period of the fluorescence excitation light source.

[0030] In some examples, the first frequency corresponds to the frame rate of the rolling shutter imager.

[0031] In some examples, the first frequency is equal to the frame rate of the rolling shutter imager.

[0032] In some examples, each illumination period of the white light illumination source is a variable configured not to exceed a first maximum value.

[0033] In some examples, the first maximum value is about 1 millisecond.

[0034] In some examples, the duration of each illumination period of a fluorescence-induced illumination source is a variable based on the distance between the imaging device transmitting the illumination and the tissue.

[0035] In some cases, the distance is determined based on luminance values ​​calculated from the first set of imaging data.

[0036] In some examples, one or more image frames include white light image frames acquired based on a first set of imaging data.

[0037] In some examples, one or more image frames include fluorescence frames acquired based on a second set of imaging data.

[0038] In some examples, one or more image frames include a mixed image frame of a fluorescence image frame and a white light image frame.

[0039] In some examples, the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

[0040] In some examples, the mixed image frame is derived from coloring the white light image frame based on the fluorescence image frame.

[0041] In some examples, the method further involves adding one or more image frames to the video stream.

[0042] In some examples, the white light source includes an LED.

[0043] In some examples, the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

[0044] In some cases, a rolling shutter imager is part of an endoscopic imager. The endoscopic imager may be pre-inserted before the start of the imaging method.

[0045] In some examples, rolling shutter imagers are equipped with a CMOS sensor.

[0046] In some cases, the first illumination period and the second illumination period begin simultaneously.

[0047] In some cases, the second illumination period is longer than the first illumination period.

[0048] An exemplary method for improving a fluorescence medical image includes receiving a white light medical image corresponding to the fluorescence medical image, and improving the fluorescence medical image by determining the maximum value among the multiple color components of each pixel in the white light medical image, and dividing the corresponding pixel in the fluorescence medical image by the maximum value. The method may also include acquiring a white light medical image.

[0049] In some examples, the method further includes displaying improved fluorescent medical images.

[0050] In some cases, improved fluorescence medical images are displayed according to a color scale in which different colors indicate different fluorescence intensities.

[0051] In some cases, according to the color scale, red shows higher fluorescence intensity than green, and green shows higher fluorescence intensity than blue.

[0052] In some cases, the improved fluorescence medical image is superimposed on the corresponding white light image.

[0053] In some examples, displaying an improved fluorescent medical image includes coloring a corresponding white light image based on the improved medical image and displaying the colored white light image.

[0054] In some examples, multiple color components include red, green, and blue components, or any combination thereof.

[0055] In some examples, fluorescence medical images and white light medical images illustrate the same tissue in question.

[0056] In some cases, the tissue includes the lymph nodes in question.

[0057] In some cases, fluorescent medical images and white light medical images are acquired using a rolling shutter imager.

[0058] In some cases, fluorescence medical images and white light medical images are acquired using a global shutter imager.

[0059] In some cases, fluorescence and white light medical images are acquired using an endoscopic imager. The endoscopic imager may be pre-inserted before initiating the improvement procedure.

[0060] In some cases, fluorescence medical images and white light medical images are acquired using open-field imagers.

[0061] An exemplary system for imaging a target tissue comprises a fluorescence excitation illumination source, a white light illumination source, and an imaging device equipped with an electron rolling shutter. The imaging device may be configured to illuminate the target tissue using the white light illumination source during a first illumination period to accumulate charge in multiple rows of pixels of a rolling shutter imager, where the fluorescence excitation illumination source is off during the first illumination period and the first set of accumulated charge in the multiple rows is sequentially read out from the first row to the last row to generate a first set of imaging data; to illuminate the target tissue using the fluorescence excitation illumination source during a second illumination period after the first illumination period to accumulate charge in multiple rows of pixels of a rolling shutter imager, where the white light illumination source is off during the second illumination period and the second set of accumulated charge in the multiple rows is sequentially read out from the first row to the last row to generate a second set of imaging data; and to generate one or more image frames based on the first set of imaging data and the second set of imaging data.

[0062] In some examples, a fluorescence-induced illumination source and a white light illumination source are configured to alternately provide illumination periods to the target tissue.

[0063] In some examples, a fluorescence-induced illumination source is configured to provide illumination periods at a first frequency, and a white light illumination source is configured to provide illumination periods at a second frequency.

[0064] In some examples, the first or second frequency corresponds to the frame rate of the rolling shutter imager.

[0065] In some cases, the first or second frequency is half the frame rate of the rolling shutter imager.

[0066] In some examples, each illumination period of a white light source is a variable configured so as not to exceed a first maximum value.

[0067] In some examples, the first maximum value is approximately 1 millisecond.

[0068] In some examples, the duration of each illumination period for the fluorescence excitation illumination source is fixed.

[0069] In some examples, each illumination period of a fluorescence-excited illumination source is a variable configured so as not to exceed a second maximum value.

[0070] In some examples, one or more image frames include white light image frames based on a first set of imaging data.

[0071] In some examples, one or more image frames include fluorescence image frames based on a second set of imaging data.

[0072] In some examples, one or more image frames include a mixed image frame based on a fluorescence image frame and a white light image frame.

[0073] In some examples, the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

[0074] In some examples, the mixed image frame is derived from coloring the white light image frame based on the fluorescence image frame.

[0075] In some examples, the imaging device is further configured to add one or more image frames to the video stream.

[0076] In some examples, the white light source includes an LED.

[0077] In some examples, the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

[0078] In some cases, rolling shutter imagers are part of endoscopic imagers.

[0079] In some examples, rolling shutter imagers are equipped with a CMOS sensor.

[0080] An exemplary system for imaging a target tissue comprises a fluorescence-excitation illumination source, a white light illumination source, and an imaging device equipped with an electron rolling shutter. The imaging device may be configured to illuminate the target tissue using the white light illumination source during a first illumination period and using the fluorescence-excitation illumination source during a second illumination period in order to accumulate charge in the pixels of multiple rows of the rolling shutter imager, and to sequentially read out the first set of charges accumulated in the pixels of multiple rows from the first row to the last row in order to generate a first set of imaging data, and to illuminate the target tissue using the fluorescence-excitation illumination source during a third illumination period after the second illumination period in order to accumulate charge in the pixels of multiple rows of the rolling shutter imager, where the white light illumination source is off during the second illumination period, and to sequentially read out the second set of charges accumulated in the pixels of multiple rows from the first row to the last row in order to generate a second set of imaging data, and to generate one or more image frames based on the first set of imaging data and the second set of imaging data.

[0081] In some examples, the fluorescence-induced illumination source is configured to provide illumination to the target tissue at a first frequency, and the white light illumination source is configured to provide illumination during at least a portion of every other illumination period of the fluorescence-induced illumination source.

[0082] In some examples, the first frequency corresponds to the frame rate of the rolling shutter imager.

[0083] In some cases, the first frequency is equal to the frame rate of the rolling shutter imager.

[0084] In some examples, each illumination period of a white light source is a variable configured so as not to exceed a first maximum value.

[0085] In some examples, the first maximum value is approximately 1 millisecond.

[0086] In some examples, the duration of each illumination period of a fluorescence-excited illumination source is a variable based on the distance between the imaging device transmitting the illumination and the tissue.

[0087] In some cases, the distance is determined based on luminance values ​​calculated from the first set of imaging data.

[0088] In some examples, one or more image frames include white light image frames acquired based on a first set of imaging data.

[0089] In some examples, one or more image frames include fluorescence frames acquired based on a second set of imaging data.

[0090] In some examples, one or more image frames include a mixed image frame of a fluorescence image frame and a white light image frame.

[0091] In some examples, the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

[0092] In some examples, the mixed image frame is derived from coloring the white light image frame based on the fluorescence image frame.

[0093] In some examples, the imaging device is further configured to add one or more image frames to the video stream.

[0094] In some examples, the white light source includes an LED.

[0095] In some examples, the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

[0096] In some cases, rolling shutter imagers are part of endoscopic imagers.

[0097] In some examples, rolling shutter imagers are equipped with a CMOS sensor.

[0098] In some cases, the first illumination period and the second illumination period begin simultaneously.

[0099] In some cases, the second illumination period is longer than the first illumination period.

[0100] An exemplary system for improving fluorescence medical images comprises one or more processors, one or more memories, and one or more programs. The one or more programs are stored in one or more memories and are configured to be executed by one or more processors. The one or more programs include instructions for acquiring a white light medical image corresponding to a fluorescence medical image, determining the maximum value among multiple color components of each pixel in the white light medical image, and improving the fluorescence medical image by dividing the corresponding pixel in the fluorescence medical image by the maximum value.

[0101] In some examples, one or more programs further include instructions to display improved fluorescent medical images.

[0102] In some cases, improved fluorescence medical images are displayed according to a color scale in which different colors indicate different fluorescence intensities.

[0103] In some cases, according to the color scale, red shows higher fluorescence intensity than green, and green shows higher fluorescence intensity than blue.

[0104] In some cases, the improved fluorescence medical image is superimposed on the corresponding white light image.

[0105] In some examples, displaying an improved fluorescent medical image includes coloring a corresponding white light image based on the improved medical image and displaying the colored white light image.

[0106] In some examples, multiple color components include red, green, and blue components, or any combination thereof.

[0107] In some examples, fluorescence medical images and white light medical images illustrate the same tissue in question.

[0108] In some cases, the tissue includes the lymph nodes in question.

[0109] In some cases, fluorescent medical images and white light medical images are acquired using a rolling shutter imager.

[0110] In some cases, fluorescence medical images and white light medical images are acquired using a global shutter imager.

[0111] In some cases, fluorescent medical images and white light medical images are acquired using an endoscopic imager.

[0112] In some cases, fluorescence medical images and white light medical images are acquired using open-field imagers. [Brief explanation of the drawing]

[0113] The present invention is described merely as an example with reference to the accompanying drawings.

[0114] [Figure 1A] Figure 1A shows diagrams of endoscopic camera systems, including several examples.

[0115] [Figure 1B] Figure 1B shows some examples of the endoscopic camera system from Figure 1A, along with the target object for imaging.

[0116] [Figure 2]Figure 2 shows schematic diagrams of systems for illumination and imaging in open-field surgery, using several examples.

[0117] [Figure 3] Figure 3 shows block diagrams of imaging systems, including several examples.

[0118] [Figure 4A] , [Figure 4B] Figures 4A and 4B illustrate imaging systems in which the fluorescence excitation illumination source is set to always be on, without the option to adjust the power supply, in several examples.

[0119] [Figure 5] Figure 5 provides illustrative methods for imaging target tissue, using several examples.

[0120] [Figure 6A] Figure 6A shows exemplary operation of an exemplary imaging system, using several examples.

[0121] [Figure 6B] Figure 6B shows a timing diagram of an exemplary imaging system, using several examples.

[0122] [Figure 7] Figure 7 provides illustrative methods for imaging target tissue, using several examples.

[0123] [Figure 8A] Figure 8A shows exemplary operation of an exemplary imaging system, using several examples.

[0124] [Figure 8B] Figure 8B shows a timing diagram of an exemplary imaging system, using several examples.

[0125] [Figure 9]Figure 9 shows exemplary methods for improving fluorescence medical images, using several examples.

[0126] [Figure 10] Figure 10 shows a comparison between three segmented images using several examples.

[0127] [Figure 11] Figure 11 shows an exemplary blue photofluorescence imaging system according to several embodiments. [Modes for carrying out the invention]

[0128] Herein, we refer in detail to implementations and examples of various embodiments and variations of the systems and methods described herein. While several exemplary variations of the systems and methods are described herein, other variations of the systems and methods may include embodiments of the systems and methods described herein that are combined in any preferred manner, having all or some combinations of the embodiments described herein. Examples are described below in full with reference to the accompanying drawings; however, they may be embodied in different forms and should not be construed as being limited to the examples described herein. Rather, these examples are provided to make this disclosure complete and comprehensive and to fully convey exemplary implementations to those skilled in the art.

[0129] Examples of this disclosure include techniques for tuning the fluorescence excitation illuminator of an imaging system to improve the stability and consistency of the fluorescence signal. According to some examples of tuning schemes, the fluorescence excitation illuminator and the white light illuminator are configured to alternately provide illumination periods to the tissue of interest. According to some examples of tuning schemes, the fluorescence excitation illuminator is configured to provide illumination periods to the tissue of interest at a certain frequency, and the white light illuminator is configured to provide illumination during at least a portion of every other illumination period of the fluorescence excitation illuminator, and furthermore, the length of each illumination period of the fluorescence excitation illuminator may be dynamically adjusted based on the distance between the imaging device transmitting the illumination and the tissue. Both tuning schemes offer many technical advantages over the continuously-on fluorescence illumination scheme, as described below.

[0130] According to several examples of the adjustment method, the exemplary system illuminates the target tissue using a white light source during a first illumination period to accumulate charge in the pixels of multiple rows of a rolling shutter imager. The fluorescence excitation source is off during the first illumination period. The system sequentially reads out the first set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the first set of imaging data. The system then illuminates the target tissue using a fluorescence excitation source during a second illumination period following the first illumination period to accumulate charge in the pixels of multiple rows of a rolling shutter imager. The white light source is off during the second illumination period. The system sequentially reads out the second set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the second set of imaging data. Based on the first and second sets of imaging data, the system generates one or more image frames based on the first and second sets of imaging data. In one particular implementation, the fluorescence excitation illuminator is tuned to turn on for approximately 1 millisecond at approximately 60 Hz and with a 180° phase offset relative to the white light illuminator.

[0131] Imaging systems utilizing the above-described adjustment methods can offer several technical advantages over imaging systems with a continuously-on fluorescence excitation source. By separating white light illumination and fluorescence excitation illumination, the imaging system can reduce cross-contamination, achieve better separation between frames, and produce sharper mixed images. The imaging system can provide better control and consistency of fluorescence images. For example, the imaging system can provide improved excitation light intensity at the minimum working distance. However, the imaging system may perform worse as the surgical imaging device emitting the illumination moves further away from the tissue. Since the fluorescence excitation light duration is set not to exceed a fixed value (e.g., only about 1 millisecond), if the imaging device transmitting the illumination is far enough away from the tissue, the imaging system will lose fluorescence intensity due to the distance and fixed exposure time.

[0132] According to several examples of adjustment methods, an exemplary system illuminates the target tissue using a white light source during the first illumination period and fluorescence excitation light during the second illumination period to accumulate charge in the pixels of multiple rows of a rolling shutter imager. The first and second illumination periods may start simultaneously, and the second illumination period may be longer than the first. The system sequentially reads out the first set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the first set of imaging data. The system then illuminates the target tissue using a fluorescence excitation light source during the third illumination period, following the second illumination period, to accumulate charge in the pixels of multiple rows of the rolling shutter imager. The white light source is off during the third illumination period. The system sequentially reads out the second set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the second set of imaging data. Based on the first and second sets of imaging data, the system generates one or more image frames based on the first and second sets of imaging data. In one particular implementation, the fluorescence excitation illumination source is adjusted to dynamically change its pulse width between 10 microseconds and 8.342 milliseconds at 120 Hz, based on the distance to the tissue being imaged.

[0133] The imaging system utilizing the above-described adjustment method offers several technical advantages. First, the length of each illumination period of the fluorescence excitation illuminator is dynamically adjusted based on the distance between the imaging device transmitting the illumination and the tissue. For example, the fluorescence excitation illuminator can be adjusted to dynamically vary its pulse width between 10 microseconds and 8.342 milliseconds at 120 Hz based on the distance. As the illuminator moves away from the tissue, the illumination period is dynamically set to be longer to increase exposure. In use cases where the working distance of the imaging device can vary significantly, this imaging system can offer several technical advantages compared to imaging systems using the adjustment method examples in Figures 6A-B, where a fixed illumination period of the fluorescence excitation illuminator is provided and therefore may have lower performance as the distance increases. In such use cases, when the illuminator is close to the tissue, the illumination period is automatically set to be shorter to avoid generating false perceptions of the fluorescence signal, thus the imaging system can offer several technical advantages compared to imaging systems using an always-on illumination method. Therefore, the imaging system utilizing the adjustment method in Figures 8A-B can maintain the fluorescence signal in accordance with any working distance.

[0134] Furthermore, when the white light illumination source is on, the fluorescence excitation illumination source is also on, and the illumination period of the fluorescence excitation illumination source (e.g., the second illumination period) extends beyond the illumination period of the white light illumination source (e.g., the first illumination period). As described herein, the extended period overlaps with the exposure of the next fluorescence frame, so as to increase the exposure time of the fluorescence frame. This system may offer several technical advantages, for example, for imaging relatively weak fluorescence signals, compared to imaging systems that utilize the example adjustment schemes shown in Figures 6A-B, where the fluorescence excitation illumination source is not on when the white light illumination source is on.

[0135] Therefore, the examples of this disclosure can improve the stability and integrity of fluorescence images. In some examples, the user can select from a number of illumination methods, including continuous illumination methods and various examples of the adjustment methods described herein. In some examples, one particular exemplary adjustment method can be set as the default setting for the imaging system.

[0136] Accordingly, this specification describes exemplary devices, apparatus, systems, methods, and non-temporary storage media for medical imaging. More generally, exemplary devices, systems, and methods for regulating fluorescence excitation illumination sources are described. Systems, devices, and methods may be used to image target tissue in endoscopic imaging surgery and open-field surgery, etc. Endoscopic imagers may be pre-inserted into the target before the start of imaging surgery. Imaging may be performed preoperatively, intraoperatively, postoperatively, and during diagnostic imaging sessions and surgery. Some techniques are described in relation to specific types of imagers (e.g., rolling shutter imagers, global shutter imagers), but it should be understood that the techniques may be applicable to any type of imager. Furthermore, these techniques may be applicable to non-surgical or non-medical uses.

[0137] Please understand that the singular forms "a," "an," and "the" used below are intended to include the plural form unless the context clearly indicates otherwise. Furthermore, please understand that, as used herein, the terms "and / or" refer to and encompass one or any combination of the related enumerated items. Additionally, please understand that, as used herein, the terms "includes," "including," "comprises," and / or "comprising" identify the presence of the described features, integers, steps, operations, elements, components, and / or units, and do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, units, and / or groups thereof.

[0138] Certain aspects of this disclosure include process steps and instructions described herein in relation to the formation of algorithms. Note that the process steps and instructions of this disclosure may be embodied in software, firmware, or hardware, and if embodied in software, they may reside on and be downloadable for operation on different platforms used by various operating systems. As will be evident from the following discussion, it should be understood that discussions throughout this description using terms such as “process,” “calculate,” “determine,” “display,” and “generate” refer to the operations and processes of computer systems and similar electronic computing devices that manipulate and transform data represented as physical (electronic) quantities in information storage devices such as memory and registers of computer systems, transmitters, and display devices.

[0139] In some examples, this disclosure also relates to devices for performing the operations described herein. These devices may include general-purpose computers that are configured specifically for a required purpose or that are selectively driven or reconfigured by computer programs stored in the computer. Such computer programs may be stored in non-temporary, computer-readable storage media such as floppy disks, USB flash drives, external hard drives, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROMs, EEPROMs, magnetic or optical cards, application-specific integrated circuits (ASICs), or any type of medium suitable for storing electronic instructions, each of which may be coupled to a computer system bus. Furthermore, the computers referred to herein may include a single processor or may have an architecture employing multiple processor designs to enhance computing power.

[0140] The methods, devices, and systems described herein are not inherently related to any particular computer or other device. Various general-purpose systems may be used with programs following the teachings herein, or it may be shown to be convenient to configure more specific devices to perform the steps of the requested method. The required structures for various such systems will become apparent below. In addition, the present invention is not written with reference to any particular programming language. It should be understood that various programming languages ​​may be used to carry out the teachings of the present invention as described herein.

[0141] Figure 1A shows an example of an endoscopic imaging system 10, including a scope assembly 11 that may be used in endoscopic surgery. The scope assembly 11 contains an endoscope or scope 12 coupled to a camera head 16 by a coupler 13 located at the distal end of the camera head 16. Light is supplied to the scope by a light source 14 via an optical guide 26, such as an optical fiber cable. The camera head 16 is coupled to a camera control unit (CCU) 18 by an electrical cable 15. The CCU 18 is connected to and communicates with the light source 14. The operation of the camera 16 is partially controlled by the CCU 18. The cable 15 transmits video and / or still image data from the camera head 16 to the CCU 18 and can transmit various control signals bidirectionally between the camera head 16 and the CCU 18.

[0142] As described later, the camera head 16 may be provided with a control or switch arrangement 17 so that the user can manually control various functions of the system 10, which may include switching from one imaging mode to another. Voice commands can be input to a microphone 25 attached to a headset 27 worn by the operator and coupled to a voice control unit 23. A handheld control device 29, such as a tablet with a touchscreen user interface or PDA, may be coupled to the voice control unit 23 as a further control interface. In the illustrated example, a recorder 31 and a printer 33 are also coupled to the CCU 18. Additional devices, such as an image capture device and an archiving device, may be included in the system 10 and coupled to the CCU 18. Video image data acquired by the camera head 16 and processed by the CCU 18 is converted into an image, which may be displayed on the monitor 20, recorded by the recorder 31, and / or used to generate a still image, and a hard copy of the still image may be generated by the printer 33.

[0143] Figure 1B shows an example of a part of an endoscopic system 10 used to illuminate and receive light from an object 1, such as a patient's surgical site. Object 1 may include a fluorescent marker 2, for example, as a result of administering a fluorescent contrast agent to the patient. The fluorescent contrast agent may be administered in advance before the start of the imaging surgery. Fluorescent marker 2 may include, for example, indocyanine green (ICG).

[0144] The light source 14 can generate visible illumination light (such as any combination of red, green, and blue light) to generate a visible (e.g., white light) image of the target object 1, and can also generate fluorescence excitation illumination light to excite a fluorescent marker 2 in the target object to generate a fluorescence image. The illumination light is transmitted through an optical lens system 22 that focuses the light onto an optical pipe 24. The optical pipe 24 can generate homogeneous light, which is then transmitted to an optical fiber guide 26. The optical guide 26 may contain multiple optical fibers and is connected to a light post 28, which is part of the endoscope 12. The endoscope 12 includes an illumination path 12' and an optical channel path 12''.

[0145] The endoscope 12 may include a notch filter 131 that allows some or all (preferably at least 80%) of the fluorescent light emitted by the fluorescent marker 2 in the target object 1 (e.g., in the wavelength range of 830 nm to 870 nm) to pass through, and some or all (preferably at least 80%) of the visible light (e.g., in the wavelength range of 400 nm to 700 nm), such as the visible illumination light reflected by the target object 1, to pass through, but substantially blocks the fluorescent excitation light (e.g., infrared light with a wavelength of 808 nm) used to excite the fluorescence emission from the fluorescent marker 2 in the target object 1. The notch filter 131 may have an optical density of OD5 or greater. In some examples, the notch filter 131 may be located in the coupler 13.

[0146] Figure 2 illustrates an exemplary open-field imaging system in several examples. Figure 2 shows a schematic diagram of an illumination and imaging system 210 that may be used in an open-field surgical procedure. As seen therein, the system 210 may include an illumination module 211, an imaging module 213, and a video processor / illuminator (VPI) 214. The VPI 214 may include an illumination source 215 for providing illumination to the illumination module 211, and a processor assembly 216 for transmitting control signals and receiving data on the light detected by the imaging module 213 from a target 212 illuminated by the light output by the illumination module 211. In one variation, the video processor / illuminator 214 may comprise a separately housed illumination source 215 and processor assembly 216. In one variation, the video processor / illuminator 214 may comprise a processor assembly 216, and one or more illumination sources 215 are separately housed within the housing of the illumination module 211. The illumination source 215 can output light in different wavelength bands, such as white (RGB) light, excitation light to induce fluorescence in the target 212, or combinations thereof, depending on the feature being examined and the material of the target 212. Light in different wavelength bands can be output by the illumination source 215 simultaneously, sequentially, or both. The illumination and imaging system 210 can be used, for example, to facilitate medical (e.g., surgical) decision-making during a surgical procedure. The target 212 may be a biological material, including a topographically complex target, such as tissue, anatomical structure, or other objects with contours and shapes that produce shadowing when illuminated. The VPI 214 can record, process, and display the resulting images and related information.

[0147] Figure 3 schematically illustrates an exemplary imaging system 300 using an electronic imager 302 to generate images (e.g., still images and / or videos) of a target object, such as a patient's target tissue, in several examples. The imager 302 may be a rolling shutter imager (e.g., a CMOS sensor) or a global shutter imager (e.g., a CCD sensor). System 300 may be used, for example, in the endoscopic imaging system 10 of Figure 1A. The imager 302 includes a CMOS sensor 304 having an array of pixels 305 arranged in pixel rows 308 and pixel columns 310. The imager 302 may include a control component 306 that controls the signals generated by the CMOS sensor 304. Examples of the control component include a gain circuit for generating a multibit signal indicating the light incident on each pixel of the sensor 304, one or more analog-to-digital converters, one or more line drivers acting as buffers and supplying driving power to the sensor 304, a row circuit, and a timing circuit. The timing circuit may include components such as a bias circuit, a clock / timing generation circuit, and / or an oscillator. The row circuit may enable one or more processing and / or operational tasks, such as addressing pixel row 308, addressing pixel column 310, resetting the charge on pixel row 308, enabling exposure of pixel 305, decoding signals, amplifying signals, analog-to-digital signal conversion, applying timing, applying read and reset signals, and other appropriate processes or tasks. The imager 302 may also include a mechanical shutter 312, which can be used, for example, to control the exposure of image sensor 304 and / or to control the amount of light received by image sensor 304.

[0148] One or more control components may be integrated into the same integrated circuit as the sensor 304, or they may be separate components. The imager 302 may be incorporated into an imaging head, such as the camera head 16 of the system 10.

[0149] One or more control components 306, such as row circuits and timing circuits, may be electrically connected to an imaging controller 320, such as a camera control unit 18 of the system 10. The imaging controller 320 may include one or more processors 322 and memory 324. The imaging controller 320 may receive row readouts from the imager and control other imager operations, including readout timing and mechanical shutter operation. The imaging controller 320 may generate image frames, such as video frames, from row and / or column readouts from the imager 302. The generated frames may be provided to a display unit 350 for display to a user, such as a surgeon.

[0150] In this example, system 300 includes a light source 330 for illuminating a target scene. The light source 330 is controlled by an imaging controller 320. The imaging controller 320 may determine the type of illumination provided by the light source 330 (e.g., white light, fluorescence-excited light, or both), the intensity of the illumination provided by the light source 330, and / or the on / off time of the illumination synchronized with the rolling shutter operation. The light source 330 may include a first photogenerator 332 for generating light of a first wavelength and a second photogenerator 334 for generating light of a second wavelength. For example, in some examples, the first photogenerator 332 may be a white light generator that includes multiple individual photogenerating components (e.g., multiple LEDs of different colors), and the second photogenerator 334 may be a fluorescence-excited light generator such as a laser diode.

[0151] The light source 330 includes a controller 336 that controls the light output of the photogenerator. The controller 336 may be configured to provide pulse-width modulation of the photogenerator to adjust the intensity of the light provided by the light source 330 and can be used to manage overexposure and underexposure. In some examples, the nominal current and / or voltage of each photogenerator remains constant, and the light intensity is adjusted by switching the photogenerator (e.g., LED) on and off according to the pulse-width control signal. In some examples, the PWM control signal is provided by the imaging controller 336. This control signal may be a waveform corresponding to the desired pulse-width modulation operation of the photogenerator.

[0152] The imaging controller 320 may be configured to determine the required illumination intensity for the light source 330 and may generate a PWM signal to be communicated to the light source 330. In some examples, depending on the amount of light received by the sensor 304 and the integration time, the light source may be pulsed at different rates to change the illumination intensity in the target scene. The imaging controller 320 may determine the required illumination intensity for subsequent frames based on the amount of light received by the sensor 304 in the current frame and / or one or more previous frames. In some examples, the imaging controller 320 may control the intensity of pixels via the PWM of the light source 330 (to increase / decrease the amount of light in the pixels), via the operation of the mechanical shutter 312 (to increase / decrease the amount of light in the pixels), and / or via a change in gain (to increase / decrease the sensitivity of the pixels to light reception). In some examples, the imaging controller 320 primarily uses PWM of the illumination source to control pixel intensity while maintaining the gain level, mainly keeping the shutter open (or at least not operating the shutter). The controller 320 may operate the shutter 312 and / or modify the gain when the light intensity is at its maximum or minimum and further adjustment is needed.

[0153] As described herein, the illumination source may or may not be separated from the imaging device in the imaging system. In some endoscopic imaging systems, the illumination source may be separated from the imaging device, while in some endoscopic imaging systems, the illumination source may be housed within the imaging device (e.g., one or more LEDs may be housed within the camera head). In some open-field imaging systems, the camera head may route the illumination itself. In some examples, the imaging device may be an endoscope combined with a connected camera head (e.g., connected by a coupler or directly to the endoscope). In some examples, the imaging device may be a single camera head, such as an open-field imaging device.

[0154] Disadvantages of the always-on fluorescence excitation illumination method Figures 4A and 4B illustrate the operation of an imaging system in which the fluorescence excitation illumination source is set to always on without the option to adjust the power supply. Figure 4A shows control signal 402 for controlling the white light illumination source of the imaging system and control signal 406 for controlling the fluorescence excitation illumination source of the imaging system. In Figure 4A, pulses in the control signals indicate that the illumination source is on to illuminate the target tissue, and the pulse width indicates the length of the illumination period.

[0155] Referring to Figure 4A, the white light illumination source is on during illumination periods 404a, 404b, 404c, etc., and off otherwise. Furthermore, the fluorescence excitation illumination source is always on during the imaging session. In the illustrated example, the white light illumination source is configured to provide illumination periods of 59.94 Hz (i.e., approximately 60 Hz), and the length of each illumination period is configured to be between 10 microseconds and 1060 microseconds (i.e., approximately 1 millisecond).

[0156] Figure 4B shows timing diagrams of imaging systems in several examples. The timing of the operation of the white light illumination source is shown in row 410, and the timing of the operation of the fluorescence excitation illumination source is shown in row 420. In the illustrated examples, the white light illumination source is configured to turn on at a frequency equal to half the frame rate of the rolling shutter imager (i.e., 120 Hz) (i.e., 60 Hz) during illumination periods 412a, 412b, 412c, 412d, 412e, etc. Furthermore, the fluorescence excitation illumination source is configured to be always on, as shown by row 420.

[0157] Referring further to Figure 4B, row 440 shows imaging data acquired in the green channel, row 450 shows imaging data acquired in the blue channel, and row 460 shows imaging data acquired in the red channel. As shown, the system illuminates the target tissue with a white light illumination source during illumination period 412a to accumulate charge in the pixels of multiple rows of the rolling shutter imager. After the first illumination period 412a, the system sequentially reads the accumulated charge in the pixels of multiple rows from the first row to the last row, generating a set of imaging data 480a including imaging data G2 from the green channel, imaging data B2 from the blue channel, and imaging data R2 and IR2 from the red channel (i.e., fluorescence imaging data from an always-on fluorescence excitation illumination source). Data 480a can be used to generate a white light frame with the fluorescence component IR2.

[0158] As described below, Figure 4B illustrates the operation of an imaging system that transmits infrared illumination to acquire IR fluorescence imaging data. However, this imaging system is merely illustrative, and it should be understood that other imaging systems with other fluorophore configurations (e.g., blue light imaging, other fluorophores emitting in other wavelength ranges) may operate according to the illumination scheme of Figure 4B. In these other imaging systems, fluorescence image data may be acquired in a channel different from the red channel, or even in multiple channels.

[0159] Furthermore, in the example illustrated in Figure 4B (similar to Figures 6B and 8B), the timing of each illumination period is shown to the left (i.e., before) the start time of the corresponding white light RGB channel readout period, rather than starting simultaneously with the start of the readout period. For example, the timing of illumination period 412a is shown to the left of the start time of the corresponding white light RGB channel readout period (indicated by the diagonal line 491), rather than starting simultaneously with the start of the period. This is due to the fact that, in this particular example, each illumination period is configured to extend into the previous frame over the last few lines to increase the exposure time of the image frame of the corresponding readout period (e.g., to avoid reaching full illumination intensity later than the start of the readout period), masking the top and bottom of each image to eliminate the resulting artifacts. It should be understood that this timing configuration is merely illustrative, and in some examples, the timing of each illumination period may be aligned with the start of the corresponding readout period.

[0160] Furthermore, referring to Figure 4B, between illumination periods 412a and 412b, the white light illumination source is off, but the fluorescence excitation illumination source is always on. Therefore, the system can sequentially read out the accumulated charge to generate imaging data 480b, which contains only the fluorescence image data IR3 acquired in the red channel. Imaging data 480b (also referred to as IR3 in Figure 4B) can be used to generate a fluorescence frame.

[0161] Further sets of imaging data, such as W4 (including G4, B4, R4), IR4, IR5, W6 (including G6, B6, R6), IR6, IR7, W8 (including G8, B8, R8), IR8, IR9, W10 (including G10, B10, R10), IR10, IR11, etc., are generated in the same manner as described above. The system can generate image frames based on these sets of imaging data. In the illustrated example, the imaging system generates approximately 60 fluorescence image frames and 60 white light image frames per second, each frame exposed to either a fluorescence excitation illuminator alone or both illuminators simultaneously. In other words, a rolling shutter imager (e.g., in the camera head) outputs image frames at a sensor frame rate of 120 Hz, which are then processed by the camera control unit to generate a display output. In the illustrated example, the mixed image is generated every 1 / 60 s (i.e., at 60 Hz), as shown in row 470 (video output). Therefore, if the camera control unit is configured to output the mixed image, the display output frame rate will be 60 fps. Each mixed frame may be based on a combination of one or more white light images and one or more fluorescence images. For example, mixed frame 490 is a combination of previously acquired W4 and IR4 and the average of two previously acquired fluorescence image frames IR3 and IR5.

[0162] The imaging system shown in Figures 4A-B has several drawbacks. In the imaging system, the fluorescence excitation illumination source is always on, and therefore provides the maximum exposure time for the fluorescence image frame and the maximum fluorescence excitation intensity over the entire working distance (e.g., the distance from the front-end lens element of the imaging objective lens (e.g., the front-end lens element of an endoscope) to the specimen / tissue being observed). However, when the tissue to be imaged is close to the imaging device illuminating it (e.g., the minimum working distance), the fluorescence excitation intensity can become excessively high, causing strong fluorescence emission and consequently generating a strong fluorescence signal in the camera of the imaging device. Since each white light image frame and / or mixed image produced by the imaging system has a fluorescence component, the strong fluorescence signal generated by the emission can reduce the background / white light average brightness and contaminate the image frame. Specifically, because there is some cross-contamination of the image signal from one color channel to another, an abnormally high fluorescence signal can contaminate other color channels. For example, if the fluorescence signal is represented in green, the entire scene in the image frame may appear green, which can generate a false perception of the fluorescence signal in the camera of the imaging device. This is a problem for both open-field cameras and endoscopic cameras. For example, because endoscopes can get very close to tissue and move quickly in and out of the area during surgery, the distance between the endoscope and the imaged tissue can change rapidly, thus resulting in false fluorescence signals.

[0163] Examples of controlled fluorescence excitation illumination sources Figure 5 provides an exemplary method 500 for imaging a target tissue, with several examples. Method 500 may be performed by an imaging system comprising a rolling shutter imager, a fluorescence excitation illumination source, and a white light illumination source. The fluorescence excitation illumination source may comprise an infrared illumination source, a blue light illumination source, or any combination thereof. The white light illumination source may comprise one or more LEDs. The rolling shutter imager may be part of an endoscope imager or an open-field imager and may comprise a CMOS sensor. The imaging system may be the imaging system 300 of Figure 3, comprising a rolling shutter imager (e.g., the rolling shutter imager 302 of system 300) and an illumination source (e.g., illumination source 330 of system 300) which may include a fluorescence excitation illumination source and / or a white light illumination source.

[0164] In process 500, some blocks are arbitrarily combined, the order of some blocks is arbitrarily changed, and some blocks are arbitrarily omitted. In some examples, additional steps may be performed in combination with process 500. Therefore, the behavior shown (and described in more detail below) is essentially illustrative and should not be considered limiting.

[0165] In block 502, the exemplary system illuminates the target tissue using a white light source during a first illumination period to accumulate charge in the pixels of multiple rows of the rolling shutter imager. The fluorescence excitation source is off during the first illumination period. In block 504, the system sequentially reads out the first set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the first set of imaging data. In block 506, the system illuminates the target tissue using a fluorescence excitation source during a second illumination period following the first illumination period to accumulate charge in the pixels of multiple rows of the rolling shutter imager. The white light source is off during the second illumination period. In block 508, the system sequentially reads out the second set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the second set of imaging data. In block 510, the system generates one or more image frames based on the first set of imaging data and the second set of imaging data.

[0166] In some examples, the fluorescence-excitation illumination source and the white light illumination source are configured to alternately provide illumination periods to the tissue of interest. Exemplary operation of the imaging system is shown in Figures 6A and 6B, which are described in detail below.

[0167] In some examples, the length of each illumination period of a white light source is a variable configured not to exceed a first maximum value (e.g., about 1 millisecond). The variation can be dynamically determined based on the distance between the imaging device transmitting the illumination and the subject / scene being visualized. The distance can be measured based on the average brightness / average intensity of previously acquired frames. Thus, when the illumination source is closer, less light is needed, and the illumination period will be shorter. Conversely, when the illumination source is farther away, the illumination period will be longer to increase exposure. Those skilled in the art will understand that the first maximum value may vary depending on the speed of the sensor used in the system.

[0168] In some examples, the length of each illumination period of the fluorescence excitation illuminator is a fixed value, or, instead (as shown in Figures 6A-B), the length of each illumination period of the fluorescence excitation illuminator is a variable configured so as not to exceed a second maximum value.

[0169] In some examples, the fluorescence-excited illumination source is configured to provide illumination periods at a first frequency. The first frequency may correspond to the frame rate of a rolling shutter imager. For example, the first frequency may be half the frame rate of the rolling shutter imager. Furthermore, the white light illumination source is configured to provide illumination periods at a second frequency. The second frequency may correspond to the frame rate of a rolling shutter imager. For example, the second frequency may be half the frame rate of the rolling shutter imager.

[0170] In some examples, one or more image frames generated by process 500 include a white light image frame based on a first set of imaging data, a fluorescence image frame based on a second set of imaging data, or both. One or more image frames may further include a mixed image frame based on the fluorescence image frame and the white light image frame. The fluorescence image frame may be superimposed on the white light image frame within the mixed image frame. The mixed image frame may be derived from coloring the white light image frame based on the fluorescence image frame. In some examples, the system adds one or more image frames to the video stream.

[0171] Figure 6A illustrates the exemplary operation of an exemplary imaging system performing process 500, using several examples. Figure 6A shows control signal 602 for controlling the white light illumination source of the imaging system and control signal 606 for controlling the fluorescence excitation illumination source of the imaging system. In Figure 6A, pulses in the control signals indicate that the illumination source is turned on to illuminate the target tissue, and the pulse width indicates the length of the illumination period. Figure 6A shows the frequency of the white light illumination period and the fluorescence excitation illumination period, and the general correspondence between the two, but it should be understood that it does not show the exact start and end times of the illumination periods.

[0172] Referring to Figure 6A, the exemplary system first illuminates the target tissue using a white light source during a first illumination period 604a in order to accumulate charge in the pixels of multiple rows of a rolling shutter imager. During the first illumination period 604a, the fluorescence excitation light source is off, as indicated by the control signal 606. After the first illumination period 604a, the system sequentially reads out the first set of charges accumulated in the pixels of multiple rows, from the first row to the last row, in order to generate the first set of imaging data.

[0173] Referring to Figure 6A, the system illuminates the target tissue using a fluorescence excitation illuminator during a second illumination period 608a following a first illumination period 604a in order to accumulate charge in the pixels of multiple rows of the rolling shutter imager. During the second illumination period 608a, the white light illuminator is turned off, as indicated by the control signal 602. After the second illumination period 608a, the system sequentially reads out the second set of charges accumulated in the pixels of multiple rows, from the first row to the last row, in order to generate a second set of imaging data. Based on the first set of imaging data and the second set of imaging data, the system generates one or more image frames. The generation of one or more image frames is described in detail herein with reference to Figure 6B.

[0174] In the example shown in Figure 6A, the fluorescence excitation illuminator and the white light illuminator are configured to alternately provide illumination periods to the tissue of interest. As shown, the white light illuminator is on during illumination periods 604a, 604b, 604c, etc., and off otherwise. Similarly, the fluorescence excitation illuminator is on during illumination periods 608a, 608b, etc., and off otherwise. The illumination periods of the white light illuminator and the fluorescence excitation illuminator are offset such that the tissue is first illuminated by the white light illuminator alone in 604a, then by the fluorescence excitation illuminator alone in 608a, then by the white light illuminator alone in 604b, then by the fluorescence excitation illuminator alone in 608b, then by the white light illuminator alone in 604c, and so on.

[0175] As indicated by the control signal 602, each illumination period of a white light illuminator may be a variable configured not to exceed a first maximum value. The variation may be dynamically determined based on the distance between the imaging device transmitting the illumination and the subject / scene being visualized. The distance can be measured based on the average brightness / average intensity of previously acquired frames. Thus, when the illumination source is closer, less light is needed, and therefore the illumination period will be shorter. In contrast, when the illumination source is farther away, the illumination period will be longer to increase the exposure. In the illustrated example, the length of the illumination period of the white light illuminator may be between 10 microseconds and 1060 microseconds (i.e., about 1 millisecond). In other words, the first maximum value may be about 1 millisecond.

[0176] As indicated by control signal 606, each illumination period of the fluorescence excitation illumination source may be a fixed value. In the illustrated example, the length may be set to 1060 microseconds (i.e., approximately 1 millisecond). Alternatively, the length may be configured to be a variable configured not to exceed a second maximum value.

[0177] The fluorescence-induced illumination source may be configured to provide illumination at a first frequency. The first frequency may correspond to the frame rate of the rolling shutter imager. For example, the first frequency may be half the frame rate of the rolling shutter imager (e.g., the image sensor frame rate output from the camera head). In the illustrated example, the first frequency is 59.94 Hz (i.e., about 60 Hz), which is half the frame rate of the rolling shutter imager (i.e., about 120 Hz). Furthermore, the white light illumination source may be configured to provide illumination at a second frequency. The second frequency may correspond to the frame rate of the rolling shutter imager. For example, the second frequency may be half the frame rate of the rolling shutter imager. In the illustrated example, the second frequency is the same as the fluorescence-induced illumination source, 59.94 Hz (i.e., about 60 Hz).

[0178] Figure 6B shows a timing diagram or adjustment scheme for an exemplary imaging system, with several examples. The timing of the operation of the white light illuminator is shown in row 610, and the timing of the operation of the fluorescence excitation illuminator is shown in row 620. In the illustrated example, the white light illuminator is configured to turn on at a frequency equal to half the frame rate of the rolling shutter imager (i.e., 120 Hz) (i.e., 60 Hz) during illumination periods 612a, 612b, 612c, 612d, 612e, etc. Furthermore, the fluorescence excitation illuminator is configured to turn on at a frequency equal to half the frame rate of the rolling shutter imager (i.e., 120 Hz) (i.e., 60 Hz) during illumination periods 622a, 622b, 622c, 622d, 622e, etc.

[0179] The illumination periods for the white light source and the fluorescence excitation source are offset by 1 / 120 s so that they alternately illuminate the target tissue at a frequency of 120 Hz. Thus, as shown by line 630, an exposure window occurs at a frequency of 120 Hz.

[0180] Referring further to Figure 6B, row 640 shows imaging data acquired in the green channel, row 650 shows imaging data acquired in the blue channel, and row 660 shows imaging data acquired in the red channel. As shown, the system illuminates the target tissue with a white light source during illumination period 612a to accumulate charge in the pixels of multiple rows of the rolling shutter imager. After the first illumination period 612a, the system sequentially reads the accumulated charge in the pixels of the multiple rows from the first row to the last row of the multiple rows, generating a set of imaging data 680a including imaging data G2 for the green channel, imaging data B2 for the blue channel, and imaging data R2 for the red channel. The imaging data 680 (also called W2 in Figure 6B) can be used to generate a white light frame.

[0181] Referring further to Figure 6B, the system illuminates the target tissue using a fluorescence excitation illumination source during illumination period 622a to accumulate charge in the pixels of multiple rows of the rolling shutter imager. After the second illumination period 622a, the system sequentially reads the charges accumulated in the pixels of multiple rows from the first row to the last row to generate imaging data 680b, which contains only the fluorescence image data IR3 acquired in the red channel. The imaging data 680b (also called IR3 in Figure 6B) can be used to generate a fluorescence frame. Figure 6B shows the operation of an imaging system that transmits infrared illumination to acquire IR fluorescence imaging data, but the imaging system is merely illustrative, and it should be understood that other imaging systems with other fluorophore configurations (e.g., blue light imaging, other fluorophores emitting in other wavelength ranges) may operate according to the illumination scheme in Figure 6B. In these other imaging systems, fluorescence image data may be acquired in a channel different from the red channel, or even in multiple channels.

[0182] Further sets of imaging data, such as W4 (including G4, B4, R4), IR5, W6 (including G6, B6, R6), IR7, W8 (including G8, B8, R8), IR9, W10 (including G10, B10, R10), and IR11, are generated in the same manner as described above. The system can generate image frames based on these sets of imaging data. In the illustrated example, a rolling shutter imager (e.g., in the camera head) outputs a set of imaging data at a sensor frame rate of 120 Hz, which is then processed by the camera control unit to generate a display output. In the illustrated example, a mixed image is generated every 1 / 60 s (i.e., 60 Hz), as shown in line 670 (video output). Therefore, if the camera control unit is configured to output a mixed image, the display output frame rate will be 60 fps.

[0183] Each mixed frame may be based on a combination of one or more white light images and one or more fluorescence images. A mixed image frame may be derived by coloring a white light image frame based on a fluorescence image frame. In the illustrated example, a mixed frame is a combination of a previously acquired white light image and the average value of two previously acquired fluorescence image frames. For example, mixed frame 690 is a combination of a previously acquired white light image W4 and the average value of two previously acquired fluorescence image frames IR3 and IR5. The average value of the two fluorescence image frames may be superimposed on the white light image. In some examples, the system adds the mixed image frames to a video stream, which may be provided during the surgical procedure.

[0184] In some examples, the system may be configured to receive user input indicating which type of image data to display (e.g., white light image data, fluorescence image data, mixed image data) and update the display based on the user input. Thus, the system can allow the user to switch between multiple viewing modes, for example, during surgery.

[0185] The imaging system in Figures 6A-B may offer several technical advantages over the imaging system in Figures 4A-B. The fluorescence excitation illumination source is tuned to be on for approximately 1 millisecond at approximately 60 Hz and a 180° phase offset relative to the white light illumination source. By separating the white light illumination and the fluorescence excitation illumination, the imaging system can reduce cross-contamination and achieve better separation between frames, resulting in a sharper mixed image. Compared to the imaging system in Figures 4-B (i.e., always-on), the imaging system in Figures 6A-B may offer better control and consistency of the fluorescence image. The imaging system may provide improved excitation light intensity at the minimum working distance.

[0186] However, the imaging systems shown in Figures 6A-B may perform worse as the illuminating surgical imaging device moves further away from the tissue. Since the fluorescence excitation period is set to a fixed value (e.g., only about 1 millisecond), if the imaging device is far enough away from the tissue, the imaging system will lose fluorescence intensity due to the distance and fixed exposure time.

[0187] Figure 7 provides an exemplary method 700 for imaging a target tissue, with several examples. Method 700 may be performed by an imaging system comprising a rolling shutter imager, a fluorescence excitation illumination source, and a white light illumination source. The fluorescence excitation illumination source may comprise an infrared illumination source, a blue light illumination source, or any combination thereof. The white light illumination source may comprise one or more LEDs. The rolling shutter imager may be part of an endoscope imager or an open-field imager and may comprise a CMOS sensor. The imaging system may be the imaging system 300 of Figure 3, comprising a rolling shutter imager (e.g., the rolling shutter imager 302 of system 300) and an illumination source (e.g., illumination source 330 of system 300) which may comprise a fluorescence excitation illumination source and / or a white light illumination source.

[0188] In process 700, some blocks are arbitrarily combined, the order of some blocks is arbitrarily changed, and some blocks are arbitrarily omitted. In some examples, additional steps may be performed in combination with process 700. Therefore, the behavior shown (and described in more detail below) is essentially illustrative and should not be considered limiting.

[0189] In block 702, the exemplary system illuminates the target tissue using a white light source during the first illumination period and fluorescence excitation light during the second illumination period to accumulate charge in the pixels of multiple rows of the rolling shutter imager. The first and second illumination periods may start simultaneously, and the second illumination period may be longer than the first illumination period. In block 704, the system sequentially reads out the first set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the first set of imaging data. In block 706, the system illuminates the target tissue using a fluorescence excitation light source during the third illumination period following the second illumination period, to accumulate charge in the pixels of multiple rows of the rolling shutter imager. The white light source is off during the third illumination period. In block 708, the system sequentially reads out the second set of charges accumulated in the pixels of multiple rows, from the first row to the last row, to generate the second set of imaging data. In block 710, the system generates one or more image frames based on the first set of imaging data and the second set of imaging data.

[0190] In some examples, a fluorescence-induced illumination source is configured to provide illumination to the target tissue at a first frequency, and a white light illumination source is configured to provide illumination during at least a portion of every other illumination period of the fluorescence-induced illumination source. In some examples, the first frequency corresponds to the frame rate of a rolling shutter imager. For example, the first frequency may be equal to the frame rate of the rolling shutter imager.

[0191] In some examples, each illumination period of a white light illumination source is a variable configured not to exceed a first maximum value (e.g., approximately 1 millisecond). In some examples, each illumination period of a fluorescence excitation illumination source is a variable based on the distance between the imaging device and the tissue. The distance may be determined based on luminance values ​​calculated based on a first set of imaging data.

[0192] In some examples, one or more image frames generated by process 700 include a white light image frame based on a first set of imaging data, a fluorescence image frame based on a second set of imaging data, or both. One or more image frames may further include a mixed image frame based on the fluorescence image frame and the white light image frame. The fluorescence image frame may be superimposed on the white light image frame within the mixed image frame. The mixed image frame may be derived from coloring the white light image frame based on the fluorescence image frame. In some examples, the system adds one or more image frames to the video stream.

[0193] Figure 8A illustrates the exemplary operation of an exemplary imaging system performing process 700, using several examples. Figure 8A shows control signal 802 for controlling the white light illumination source of the imaging system and control signal 806 for controlling the fluorescence excitation illumination source of the imaging system. In Figure 8A, pulses indicate that the illumination source is turned on to illuminate the tissue of interest, and pulse width indicates the length of the illumination period. As described above, Figure 8A shows the frequency of white light illumination period and fluorescence excitation illumination period, and the general correspondence between the two, but does not illustrate the exact start and end times of the illumination periods. For example, as will be described in more detail with reference to Figure 8B, the first illumination period 804a and the second illumination period 808a may start simultaneously, and the second illumination period 808a may be longer than the first illumination period 804a, which is not shown in Figure 8A.

[0194] Referring to Figure 8A, the exemplary system first illuminates the target tissue using a white light source during the first illumination period 804a and a fluorescence excitation light source during the second illumination period 808a in order to accumulate charge in the pixels of multiple rows of the rolling shutter imager. The system then sequentially reads out the first set of charges accumulated in the pixels of multiple rows, from the first row to the last row, in order to generate the first set of imaging data. The system illuminates the target tissue using the fluorescence excitation light source only during the third illumination period 808b, after the second illumination period 808a, in order to accumulate charge in the pixels of multiple rows of the rolling shutter imager. The white light source is off during the third illumination period 808b, as indicated by the control signal 802. The system then sequentially reads out the second set of charges accumulated in the pixels of multiple rows, from the first row to the last row, in order to generate the second set of imaging data. The system can then generate one or more image frames based on the first set of imaging data and the second set of imaging data.

[0195] Referring further to Figure 8A, the fluorescence excitation illuminator is configured to provide illumination to the target tissue at a first frequency, and the white light illuminator is configured to provide illumination during at least a portion of every other illumination period of the fluorescence excitation illuminator. The first frequency corresponds to the frame rate of the rolling shutter imager. For example, the first frequency may be equal to the frame rate of the rolling shutter imager. In the illustrated example, the frame rate of the rolling shutter imager is approximately 120 Hz. As indicated by control signal 806, the fluorescence excitation illuminator is configured to provide illumination to the target tissue at approximately 120 Hz. In contrast, as indicated by control signal 802, the white light illuminator is configured to provide illumination at approximately 60 Hz during at least a portion of every other illumination period of the fluorescence excitation illuminator.

[0196] As indicated by the control signal 802, the length of each illumination period of a white light illuminator may be a variable configured not to exceed a first maximum value. The change may be dynamically determined based on the distance between the imaging device transmitting the illumination and the subject / scene being visualized. The distance can be measured based on the average brightness / average intensity of previously acquired frames. Thus, when the illumination source is closer, less light is needed, and therefore the illumination period will be shorter. In contrast, when the illumination source is farther away, the illumination period will be longer to increase the exposure. In the illustrated example, the length of the illumination period of the white light illuminator may be between 10 microseconds and 1060 microseconds (i.e., about 1 millisecond). In other words, the first maximum value may be about 1 millisecond.

[0197] As indicated by control signal 806, the length of each illumination period (i.e., pulse width) of a fluorescence excitation illuminator can be a dynamic variable based on the distance between the imaging device transmitting the illumination and the tissue. If the distance is longer, the illumination period length is automatically set to be longer. Conversely, if the distance is shorter, the illumination period length is automatically set to be shorter. In the illustrated example, the illumination period length of the fluorescence excitation illuminator may be configured to be in the range of 10 microseconds to 8342 microseconds. The distance may be determined, for example, based on a luminance value calculated based on a first set of imaging data. For example, the distance may be determined based on the average luminance of previous white light frames, because the average luminance changes based on distance, increasing for shorter distances and decreasing for longer distances. It should be understood that the distance may be determined using other means, such as a proximity sensor.

[0198] Figure 8B shows a timing diagram or configuration for an exemplary imaging system, with several examples. The timing of the operation of the white light illuminator is shown in row 810, and the timing of the operation of the fluorescence excitation illuminator is shown in row 820. In the illustrated example, the white light illuminator is configured to turn on at a frequency equal to half the frame rate of the rolling shutter imager (i.e., 120 Hz) (i.e., 60 Hz) during illumination periods 812a, 812b, 812c, 812d, 812e, 812e, etc. Furthermore, the fluorescence excitation illuminator is configured to turn on at a frequency equal to the frame rate of the rolling shutter imager (i.e., 120 Hz) (i.e., 60 Hz) during illumination periods 822a, 822b, 822c, 822d, 822e, 822f, etc. In the illustrated example, illumination periods 812a and 822a start simultaneously, and illumination period 822a extends beyond illumination period 812a.

[0199] Referring to Figure 8B, both the white light illuminator and the fluorescence excitation illuminator are on during illumination periods 812a (overlapping with the first part of 822a), 812b (overlapping with the first part of 822c), 812c (overlapping with the first part of 822e), etc. The fluorescence excitation illuminator is also on for the remainder of illumination periods 822a, 822c, 833e, etc., as well as during illumination periods 822b, 822d, 822f, etc. Therefore, the tissue is first illuminated by both a white light source and a fluorescence-induced illuminator in the overlapping portion of 812a and 822a, then illuminated by the fluorescence-induced illuminator alone for the remainder of period 822a, then illuminated by the fluorescence-induced illuminator alone in 822b, then illuminated by both illuminators in the overlapping portion of 812b and 822c, then illuminated by the fluorescence-induced illuminator alone for the remainder of period 822c, then illuminated by the fluorescence-induced illuminator alone in 822d, and so on.

[0200] Referring further to Figure 8B, row 840 shows imaging data acquired in the green channel, row 850 shows imaging data acquired in the blue channel, and row 860 shows imaging data acquired in the red channel. As shown, the system illuminates the target tissue using both illumination sources during illumination period 812a to accumulate charge in the pixels of multiple rows of the rolling shutter imager. After the first illumination period 812a, the system sequentially reads the accumulated charge in the pixels of the multiple rows from the first row to the last row of the multiple rows, generating a set of imaging data 880a including imaging data G2 for the green channel, imaging data B2 for the blue channel, and imaging data R2 and IR2 for the red channel. In addition, since both illumination sources are on during illumination window 812a, the system acquires fluorescence imaging data IR2 in the red channel. In addition to data G2, B2 and R2 (collectively referred to as W2 in Figure 8B), IR2 may be used to generate a white light frame with a fluorescence component. Figure 8B illustrates the operation of an imaging system that transmits infrared illumination to acquire IR fluorescence imaging data. However, this imaging system is merely illustrative, and it should be understood that other imaging systems with other fluorophore configurations (e.g., blue light imaging, other fluorophores emitting in other wavelength ranges) may operate according to the illumination scheme in Figure 8B. In these other imaging systems, fluorescence image data may be acquired in a channel different from the red channel, or even in multiple channels.

[0201] Referring further to Figure 8B, the system illuminates the target tissue using a fluorescence excitation illumination source during illumination period 822b to accumulate charge in the pixels of multiple rows of the rolling shutter imager. After the second illumination period 822b, the system sequentially reads the charges accumulated in the pixels of multiple rows from the first row to the last row to generate imaging data 880b. Imaging data 880b is the result of illumination periods 822a and 822b and includes only the fluorescence image data IR3 acquired in the red channel. Imaging data 880b (also referred to as IR3 in Figure 8B) can be used to generate a fluorescence frame.

[0202] Further sets of imaging data, such as W4 (including G4, B4, R4), IR4, IR5, W6 (including G6, B6, R6), IR6, IR7, W8 (including G8, B8, R8), IR8, IR9, W10 (including G10, B10, R10), IR10, IR11, etc., are generated in the same manner as described above. The system can generate image frames based on these sets of imaging data. In the illustrated example, a rolling shutter imager (e.g., in the camera head) outputs a set of imaging data at a sensor frame rate of 120 Hz, which is then processed by the camera control unit to generate a display output. In the illustrated example, a mixed image is generated every 1 / 60 s (i.e., 60 Hz), as shown in line 870 (video output). Therefore, if the camera control unit is configured to output a mixed image, the display output frame rate will be 60 fps.

[0203] Each mixed frame may be based on a combination of one or more white light images and one or more fluorescence images. A mixed image frame may be derived by coloring the white light image frame based on the fluorescence image frame. For example, mixed frame 890 is a combination of previously acquired W4 and IR4 and the average of previously acquired IR3 and IR5. Fluorescence signals may be superimposed on the white light image. In some examples, the system may add the mixed image frames to a video stream, which may be displayed during the surgical procedure.

[0204] An imaging system operating according to the adjustment method shown in Figures 8A-B can offer several technical advantages. First, the length of each illumination period of the fluorescence excitation illuminator is dynamically adjusted based on the distance between the imaging device transmitting the illumination and the tissue. In the example illustrated in Figures 8A-B, the fluorescence excitation illuminator is adjusted to dynamically vary its pulse width between 10 microseconds and 8.342 milliseconds at 120 Hz based on the distance. As the imaging device transmitting the illumination moves away from the tissue, the illumination period is dynamically set to be longer to increase exposure. The imaging system can offer several technical advantages compared to the imaging system in Figures 6A-B, which provides a fixed illumination period for the fluorescence excitation illuminator and therefore has lower performance as the distance increases. The imaging system can also offer technical advantages compared to the imaging system in Figures 4A-B (i.e., always-on method) because the illumination period is automatically set to be shorter when the imaging device transmitting the illumination is close to the tissue to avoid producing a false perception of the fluorescence signal. Therefore, the imaging system in Figures 8A-B can maintain the fluorescence signal in accordance with any operating distance.

[0205] Furthermore, in an imaging system operating according to the exemplary adjustment schemes shown in Figures 8A-B, when the white light illumination source is on, the fluorescence excitation illumination source is also on, and the illumination period of the fluorescence excitation illumination source (e.g., 822a) extends beyond the illumination period of the white light illumination source (e.g., 812a). As shown in Figure 8B, the extended period overlaps with the exposure of the next fluorescence frame, increasing the exposure time of the fluorescence frame. In other words, IR3 is the result of at least a portion of illumination period 822a and at least a portion of illumination period 822b, as indicated by the circle 823. This system may offer several technical advantages compared to the imaging system in Figures 6A-B where the fluorescence excitation illumination source is not on when the white light illumination source is on.

[0206] Therefore, an imaging system operating according to the exemplary adjustment schemes shown in Figures 8A-B can improve the stability and consistency of fluorescence images. In some examples, the user can select from several illumination schemes, including the always-on illumination scheme shown in Figures 4A-B, the example adjustment schemes shown in Figures 6A-B, and / or the example adjustment schemes shown in Figures 8A-B. In some examples, the adjustment schemes illustrated in the examples of Figures 8A-B may be set as the default settings.

[0207] Examples of adjustment methods described herein may be used in imaging systems with other fluorophore configurations (e.g., blue light imaging, any other fluorophore emitting in other wavelength ranges). Figure 11 shows exemplary blue light fluorescence imaging systems in several embodiments. In blue light fluorescence imaging systems, excitation wavelength blocking filters may be ineffective because they overlap with the blue spectrum. Therefore, excitation light may leak to the imaging sensor (due to being in the visible range). Pulse width adjustment of the illumination source control, such as in the examples described herein, may be particularly important to reduce image contamination. In some examples, white balance adjustment may also be performed to further reduce image contamination.

[0208] Segmentation of specificity of fluorescence data Figure 9 illustrates exemplary methods 900 for improving fluorescence medical images, using several examples. The process 900 is performed, for example, using one or more electronic devices implementing a software platform. In some examples, the process 400 is performed using a client-server system, and blocks of the process 900 are divided in any way between the server and one or more client devices. In some examples, the process 900 is performed using only one client device, or only multiple client devices. In the process 900, some blocks are arbitrarily combined, some blocks are arbitrarily rearranged, and some blocks are arbitrarily omitted. In some examples, additional steps may be performed in combination with the process 900. Therefore, the behavior shown (and described in more detail below) is illustrative in nature and should not be considered limiting.

[0209] In block 902, the exemplary system receives a white light medical image corresponding to a fluorescence medical image. The system may, for example, acquire a white light medical image corresponding to a fluorescence medical image. In some examples, the fluorescence medical image and the white light medical image illustrate the same tissue of interest (e.g., lymph nodes of interest). In some examples, the two images are acquired using the same imager (e.g., a rolling shutter imager, a global shutter imager) which is part of an endoscopic imaging system or an open-field imaging system. In block 904, the system enhances the fluorescence medical image. Enhancements may include blocks 906 and 908. In block 906, for each pixel in the white light medical image, the system determines the maximum value among the multiple color components of the pixel. In block 908, the system divides the corresponding pixel in the fluorescence medical image by the maximum value.

[0210] In some cases, the system displays improved fluorescent medical images. In some cases, the improved fluorescent medical images are displayed according to a color scale in which different colors indicate different fluorescence intensities. In some cases, according to the color scale, red shows a higher fluorescence intensity than green, and green shows a higher fluorescence intensity than blue.

[0211] In some examples, displaying an improved fluorescent medical image includes coloring a corresponding white light image based on the improved medical image and displaying the colored white light image. In some examples, the multiple color components include a red component, a green component, a blue component, or any combination thereof. In some examples, the improved fluorescent medical image is displayed superimposed on the corresponding white light image.

[0212] Figure 10 shows a comparison between three segmented images 1004–1008 using several examples. Color segmentation can refer to the display of a fluorescence image using a color scale in which different colors exhibit different fluorescence intensities. In the illustrated example, according to color scale 1000, red exhibits a higher fluorescence intensity than green, and green exhibits a higher fluorescence intensity than blue.

[0213] Image 1002 is a white light image of three cups containing red, green, and blue liquids, respectively. Each of the three liquids in the cups is mixed with an equal concentration of the fluorescent dye, indocyanine green (ICG).

[0214] Image 1004 is a fluorescence image of the same three cups displayed according to a color scale of 1000 without normalization. In Image 1004, the amplitude of each pixel is the ratio of the original fluorescence signal to a constant (i.e., IR / constant). No normalization has been performed on the pixels.

[0215] Image 1006 is a fluorescence image of the same three cups displayed according to a color scale 1000 normalized using red reflectivity. In Image 1006, the amplitude of each pixel is the ratio (i.e., IR / R) of the original fluorescence signal of the corresponding pixel in the corresponding white light image to its red reflectivity. As shown in Figure 1006, the red reflectivity (R component) of blue is very low. Therefore, the IR / R ratio of the blue cup is very high, as shown by the bright red of the blue cup in Image 1006. The bright red inaccurately indicates a very high fluorescence signal, even if the blue cup is known to contain the same concentration of ICG as the other cups, and thus generates a false fluorescence signal that could compromise surgical safety.

[0216] Image 1008 is a fluorescence image of the same three cups displayed according to a color scale 1000 normalized according to process 900. Specifically, a white light medical image corresponding to the fluorescence medical image is obtained. For example, the white light medical image may be an image frame captured immediately before the fluorescence image by the same imager. The system can then improve the fluorescence medical image. First, for each pixel of the white light medical image, the system can determine the maximum value among the multiple color components of the pixel, such as red reflected light, blue reflected light, and green reflected light. Next, the system can divide each corresponding pixel of the fluorescence medical image by the maximum value (i.e., IR / max(RGB)). The improved image 1008 can then be displayed according to the color scale 1000. For the blue cup in image 1008, since the blue component is dominant, the pixel corresponding to the blue cup is actually divided by the blue component. Thus, the segmented image 1008 provides a more accurate representation of the fluorescence signal.

[0217] The above description is illustrated by reference to specific examples for illustrative purposes. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the exact form disclosed. While features are described here as part of the same or separate examples for clarity and brevity, it will be understood that the scope of this disclosure includes examples having all or some combinations of the described features. Many improvements and modifications are possible in light of the above teachings. The examples have been selected and described to best illustrate the principles of the art and their practical applications. Therefore, those skilled in the art will be able to best utilize the art and various examples with various improvements to suit their specific intended use.

[0218] While this disclosure and examples are adequately described with reference to the accompanying drawings, it should be noted that various changes and improvements will be apparent to those skilled in the art. Such changes and improvements should be understood to fall within the scope of this disclosure and examples as defined by the claims. Finally, all disclosures of patents and publications referenced in this application are incorporated herein by reference.

Claims

1. A method for imaging a target tissue using an imaging system comprising a rolling shutter imager, a fluorescence excitation illumination source, and a white light illumination source, wherein the method is: In order to accumulate charge in the pixels of multiple rows of the rolling shutter imager, the tissue of the target is illuminated using the white light illumination source during a first illumination period, wherein the fluorescence excitation illumination source is off during the first illumination period. To generate the first set of imaging data, the first set of charges accumulated in the pixels of the plurality of rows is sequentially read from the first row to the last row of the plurality of rows, In order to accumulate charge in the pixels of the plurality of rows of the rolling shutter imager, the tissue of the target is illuminated using the fluorescence excitation illumination source during a second illumination period after the first illumination period, wherein the white light illumination source is off during the second illumination period. In order to generate a second set of imaging data, the second set of charges accumulated in the pixels of the plurality of rows is sequentially read from the first row to the last row of the plurality of rows, Based on the first set of imaging data and the second set of imaging data, one or more image frames are generated, Methods that include...

2. A method according to claim 1, wherein the fluorescence excitation illumination source and the white light illumination source are configured to alternately provide illumination periods to the tissue of the target.

3. A method according to claim 2, wherein the fluorescence-excitation illumination source is configured to provide illumination periods at a first frequency, and the white light illumination source is configured to provide illumination periods at a second frequency.

4. A method according to claim 3, wherein the first frequency or the second frequency corresponds to the frame rate of the rolling shutter imager.

5. A method according to claim 4, wherein the first frequency or the second frequency is half the frame rate of the rolling shutter imager.

6. A method according to any one of claims 1 to 5, wherein each illumination period of the white light illumination source is a variable configured not to exceed a first maximum value.

7. A method according to claim 6, wherein the first maximum value is approximately 1 millisecond.

8. A method according to any one of claims 1 to 7, wherein each illumination period of the fluorescence excitation illumination source is a fixed value.

9. A method according to any one of claims 1 to 7, wherein each illumination period of the fluorescence excitation illumination source is a variable configured so as not to exceed a second maximum value.

10. A method according to any one of claims 1 to 9, wherein the one or more image frames include white light image frames based on the first set of imaging data.

11. A method according to any one of claims 1 to 10, wherein the one or more image frames include fluorescence image frames based on the second set of imaging data.

12. A method according to claim 11, wherein the one or more image frames include a mixed image frame based on the fluorescent image frame and the white light image frame.

13. A method according to claim 12, wherein the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

14. A method according to claim 12, wherein the mixed image frame is derived from coloring the white light image frame based on the fluorescent image frame.

15. A method according to any one of claims 1 to 14, further comprising adding the one or more image frames to a video stream.

16. A method according to any one of claims 1 to 15, wherein the white light source comprises an LED.

17. A method according to any one of claims 1 to 16, wherein the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

18. A method according to any one of claims 1 to 17, wherein the rolling shutter imager is part of an endoscope imager.

19. A method according to any one of claims 1 to 18, wherein the rolling shutter imager comprises a CMOS sensor.

20. A method for imaging a target tissue using an imaging system comprising a rolling shutter imager, a fluorescence excitation illumination source, and a white light illumination source, wherein the method is: In order to accumulate charge in the pixels of multiple rows of the rolling shutter imager, the target tissue is illuminated using the white light illumination source during the first illumination period and using the fluorescence excitation illumination source during the second illumination period. To generate the first set of imaging data, the first set of charges accumulated in the pixels of the plurality of rows is sequentially read from the first row to the last row of the plurality of rows, In order to accumulate charge in the pixels of the plurality of rows of the rolling shutter imager, the tissue of the target is illuminated using the fluorescence excitation illumination source during a third illumination period after the second illumination period, wherein the white light illumination source is off during the second illumination period. In order to generate a second set of imaging data, the second set of charges accumulated in the pixels of the plurality of rows is sequentially read from the first row to the last row of the plurality of rows, Based on the first set of imaging data and the second set of imaging data, one or more image frames are generated, Methods that include...

21. A method according to claim 20, wherein the fluorescence excitation illumination source is configured to provide illumination periods to the tissue of the subject at a first frequency, and the white light illumination source is configured to provide illumination during at least a portion of every other illumination periods of the fluorescence excitation illumination source.

22. A method according to claim 21, wherein the first frequency corresponds to the frame rate of the rolling shutter imager.

23. A method according to claim 22, wherein the first frequency is equal to the frame rate of the rolling shutter imager.

24. A method according to any one of claims 20 to 23, wherein each illumination period of the white light illumination source is a variable configured not to exceed a first maximum value.

25. A method according to claim 24, wherein the first maximum value is about 1 millisecond.

26. A method according to any one of claims 20 to 25, wherein each illumination period of the fluorescence excitation illumination source is a variable based on the distance between the imaging device transmitting the illumination and the tissue.

27. A method according to claim 26, wherein the distance is determined based on a luminance value calculated based on the first set of imaging data.

28. A method according to any one of claims 20 to 27, wherein the one or more image frames include white light image frames acquired based on the first set of imaging data.

29. A method according to any one of claims 20 to 28, wherein the one or more image frames include fluorescence frames obtained based on the second set of imaging data.

30. A method according to claim 29, wherein the one or more image frames include a mixed image frame of the fluorescent image frame and the white light image frame.

31. A method according to claim 30, wherein the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

32. A method according to claim 30, wherein the mixed image frame is derived from coloring the white light image frame based on the fluorescent image frame.

33. A method according to any one of claims 20 to 32, further comprising adding the one or more image frames to a video stream.

34. A method according to any one of claims 20 to 33, wherein the white light source comprises an LED.

35. A method according to any one of claims 20 to 34, wherein the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

36. A method according to any one of claims 20 to 35, wherein the rolling shutter imager is part of an endoscope imager.

37. A method according to any one of claims 20 to 36, wherein the rolling shutter imager comprises a CMOS sensor.

38. A method according to any one of claims 20 to 37, wherein the first illumination period and the second illumination period begin simultaneously.

39. A method according to any one of claims 20 to 38, wherein the second illumination period is longer than the first illumination period.

40. A method for improving fluorescent medical images, To acquire a white light medical image corresponding to the aforementioned fluorescent medical image, The aforementioned fluorescent medical image, For each pixel of the aforementioned white light medical image, the maximum value among the multiple color components of the pixel is determined. Dividing the corresponding pixels of the aforementioned fluorescent medical image by the maximum value, To improve by, Methods that include...

41. A method according to claim 40, further comprising displaying the improved fluorescent medical image.

42. A method according to claim 41, wherein the improved fluorescent medical image is displayed according to a color scale in which different colors indicate different fluorescence intensities.

43. A method according to claim 42, wherein, according to the color scale, red exhibits a higher fluorescence intensity than green, and green exhibits a higher fluorescence intensity than blue.

44. A method according to any one of claims 41 to 43, wherein the improved fluorescent medical image is displayed superimposed on the corresponding white light image.

45. A method according to any one of claims 41 to 43, wherein displaying the improved fluorescent medical image comprises coloring the corresponding white light image based on the improved medical image, and displaying the colored white light image.

46. A method according to any one of claims 40 to 45, wherein the plurality of color components include a red component, a green component, a blue component, or any combination thereof.

47. A method according to any one of claims 40 to 46, wherein the fluorescent medical image and the white light medical image illustrate the same tissue of the subject.

48. A method according to claim 47, wherein the tissue includes the lymph node of the subject.

49. A method according to any one of claims 40 to 48, wherein the fluorescent medical image and the white light medical image are acquired using a rolling shutter imager.

50. A method according to any one of claims 40 to 49, wherein the fluorescence medical image and the white light medical image are acquired using a global shutter imager.

51. A method according to any one of claims 40 to 50, wherein the fluorescent medical image and the white light medical image are acquired using an endoscope imager.

52. A method according to any one of claims 40 to 50, wherein the fluorescence medical image and the white light medical image are acquired using an open-field imager.

53. A system for imaging a target tissue, wherein the system is A fluorescence excitation illumination source, White light source and An imaging device equipped with an electronic rolling shutter, In order to accumulate charge in the pixels of multiple rows of the rolling shutter imager, the tissue of the target is illuminated using the white light illumination source during a first illumination period, wherein the fluorescence excitation illumination source is off during the first illumination period. In order to generate the first set of imaging data, the first set of charges accumulated in the pixels of the plurality of rows is sequentially read out from the first row to the last row of the plurality of rows, In order to accumulate charge in the pixels of the plurality of rows of the rolling shutter imager, the tissue of the target is illuminated using the fluorescence excitation illumination source during a second illumination period following the first illumination period, wherein the white light illumination source is off during the second illumination period. In order to generate a second set of imaging data, the second set of charges accumulated in the pixels of the plurality of rows is sequentially read out from the first row to the last row of the plurality of rows, Based on the first set of imaging data and the second set of imaging data, one or more image frames are generated. An imaging device configured as follows, A system equipped with these features.

54. The system according to claim 53, wherein the fluorescence excitation illumination source and the white light illumination source are configured to alternately provide illumination periods to the tissue of the target.

55. The system according to claim 54, wherein the fluorescence-excitation illumination source is configured to provide illumination periods at a first frequency, and the white light illumination source is configured to provide illumination periods at a second frequency.

56. The system according to claim 55, wherein the first frequency or the second frequency corresponds to the frame rate of the rolling shutter imager.

57. The system according to claim 56, wherein the first frequency or the second frequency is half the frame rate of the rolling shutter imager.

58. A system according to any one of claims 53 to 57, wherein each illumination period of the white light illumination source is a variable configured so as not to exceed a first maximum value.

59. The system according to claim 58, wherein the first maximum value is approximately 1 millisecond.

60. A system according to any one of claims 53 to 59, wherein each illumination period of the fluorescence excitation illumination source is a fixed value.

61. A system according to any one of claims 53 to 59, wherein each illumination period of the fluorescence excitation illumination source is a variable configured so as not to exceed a second maximum value.

62. A system according to any one of claims 53 to 61, wherein one or more image frames include white light image frames based on the first set of imaging data.

63. A system according to any one of claims 53 to 62, wherein one or more image frames include fluorescence image frames based on the second set of imaging data.

64. The system according to claim 63, wherein the one or more image frames include a mixed image frame based on the fluorescent image frame and the white light image frame.

65. The system according to claim 64, wherein the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

66. A system according to claim 64, wherein the mixed image frame is derived from coloring the white light image frame based on the fluorescent image frame.

67. A system according to any one of claims 53 to 66, wherein the imaging device is further configured to add the one or more image frames to the video stream.

68. A system according to any one of claims 53 to 67, wherein the white light source comprises an LED.

69. A system according to any one of claims 53 to 68, wherein the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

70. A system according to any one of claims 53 to 69, wherein the rolling shutter imager is part of an endoscope imager.

71. A system according to any one of claims 53 to 70, wherein the rolling shutter imager comprises a CMOS sensor.

72. A system for imaging a target tissue, wherein the system is A fluorescence excitation illumination source, White light source and An imaging device equipped with an electronic rolling shutter, In order to accumulate charge in the pixels of multiple rows of the rolling shutter imager, the target tissue is illuminated using the white light illumination source during the first illumination period and the fluorescence excitation illumination source during the second illumination period. In order to generate the first set of imaging data, the first set of charges accumulated in the pixels of the plurality of rows is sequentially read out from the first row to the last row of the plurality of rows, In order to accumulate charge in the pixels of the plurality of rows of the rolling shutter imager, the tissue of the target is illuminated using the fluorescence excitation illumination source during a third illumination period following the second illumination period, wherein the white light illumination source is off during the second illumination period. In order to generate a second set of imaging data, the second set of charges accumulated in the pixels of the plurality of rows is sequentially read out from the first row to the last row of the plurality of rows, Based on the first set of imaging data and the second set of imaging data, one or more image frames are generated. An imaging device configured as follows, A system equipped with these features.

73. A system according to claim 72, wherein the fluorescence excitation illumination source is configured to provide illumination periods to the tissue of the subject at a first frequency, and the white light illumination source is configured to provide illumination during at least a portion of every other illumination periods of the fluorescence excitation illumination source.

74. A system according to claim 73, wherein the first frequency corresponds to the frame rate of the rolling shutter imager.

75. The system according to claim 74, wherein the first frequency is equal to the frame rate of the rolling shutter imager.

76. A system according to any one of claims 72 to 75, wherein each illumination period of the white light illumination source is a variable configured so as not to exceed a first maximum value.

77. The system according to claim 76, wherein the first maximum value is approximately 1 millisecond.

78. A system according to any one of claims 72 to 77, wherein each illumination period of the fluorescence excitation illumination source is a variable based on the distance between the imaging device transmitting the illumination and the tissue.

79. The system according to claim 78, wherein the distance is determined based on a luminance value calculated based on the first set of imaging data.

80. A system according to any one of claims 72 to 79, wherein one or more image frames include white light image frames acquired based on the first set of imaging data.

81. A system according to any one of claims 72 to 80, wherein one or more image frames include fluorescence frames acquired based on the second set of imaging data.

82. The system according to claim 81, wherein the one or more image frames include a mixed image frame of the fluorescent image frame and the white light image frame.

83. The system according to claim 82, wherein the fluorescence image frame is superimposed on the white light image frame within the mixed image frame.

84. A system according to claim 82, wherein the mixed image frame is derived from coloring the white light image frame based on the fluorescent image frame.

85. A system according to any one of claims 72 to 82, wherein the imaging device is further configured to add the one or more image frames to the video stream.

86. A system according to any one of claims 72 to 85, wherein the white light illumination source comprises an LED.

87. A system according to any one of claims 72 to 86, wherein the fluorescence excitation illumination source comprises an infrared light illumination source, a blue light illumination source, or any combination thereof.

88. A system according to any one of claims 72 to 87, wherein the rolling shutter imager is part of an endoscope imager.

89. A system according to any one of claims 72 to 88, wherein the rolling shutter imager comprises a CMOS sensor.

90. A system according to any one of claims 72 to 89, wherein the first illumination period and the second illumination period start simultaneously.

91. A system according to any one of claims 72 to 90, wherein the second illumination period is longer than the first illumination period.

92. A system for improving fluorescent medical images, One or more processors, One or more memory devices, A system comprising one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs are A white light medical image corresponding to the aforementioned fluorescent medical image is acquired. For each pixel of the white light medical image, the maximum value among the multiple color components of the pixel is determined, and the fluorescent medical image is improved by dividing the corresponding pixel of the fluorescent medical image by the maximum value. Includes instructions for, system.

93. The system according to claim 92, wherein one or more programs further include instructions for displaying the improved fluorescent medical image.

94. The system according to claim 93, wherein the improved fluorescent medical image is displayed according to a color scale in which different colors indicate different fluorescence intensities.

95. A system according to claim 94, wherein, according to the color scale, red exhibits a higher fluorescence intensity than green, and green exhibits a higher fluorescence intensity than blue.

96. A system according to any one of claims 92 to 95, wherein the improved fluorescent medical image is displayed superimposed on the corresponding white light image.

97. A system according to any one of claims 92 to 95, wherein displaying the improved fluorescent medical image includes coloring the corresponding white light image based on the improved medical image, and displaying the colored white light image.

98. A system according to any one of claims 92 to 97, wherein the plurality of color components include a red component, a green component, a blue component, or any combination thereof.

99. A system according to any one of claims 92 to 98, wherein the fluorescent medical image and the white light medical image illustrate the same tissue of the subject.

100. The system according to claim 99, wherein the tissue includes the target lymph nodes.

101. A system according to any one of claims 92 to 100, wherein the fluorescent medical image and the white light medical image are acquired using a rolling shutter imager.

102. A system according to any one of claims 92 to 101, wherein the fluorescent medical image and the white light medical image are acquired using a global shutter imager.

103. A system according to any one of claims 92 to 102, wherein the fluorescent medical image and the white light medical image are acquired using an endoscope imager.

104. A system according to any one of claims 92 to 102, wherein the fluorescence medical image and the white light medical image are acquired using an open-field imager.

Images