Fluorescence imaging system and method for weighting image values

Abstract

The present invention relates to a fluorescence imaging system and a method for weighting image values. The enhanced fluorescence imaging system includes a light source for emitting invisible light and visible light, and an image sensor for capturing invisible light image data and visible light image data. Data processing hardware operates, and the operation includes determining an invisible value associated with the amount of invisible light captured by the image sensor, and applying a color map to each invisible value to generate an invisible light selected color value. The operation further includes weighting the visible light chromaticity value and the invisible chromaticity value to generate a weighted chromaticity value, and combining the luminance value of each pixel of the visible light image data with the weighted chromaticity value. The operation further includes generating RGB values based on the luminance value of the visible light image data and the weighted chromaticity value, and sending the RGB values to a display.

Description

Technical Field

[0001] The present invention relates to a fluorescence imaging system for medical procedures. Background Art

[0002] Endoscopes are commonly used to provide access to body cavities while reducing the invasiveness of surgical procedures. A fluorescence imaging system can include an endoscope, one or more light sources that emit both visible (e.g., white) light and invisible (e.g., infrared) light, a camera control unit, and a display control unit. Visible light is typically used as reference light or illumination light, while invisible light is typically used as excitation light. That is, the invisible light is used to irradiate a fluorescent substance (e.g., dye) administered to a patient, which in turn causes the fluorescent substance to emit fluorescence. The endoscope includes one or more image sensors configured to capture the reflected visible light and / or the emitted fluorescence. The fluorescence imaging system can overlay a visual representation of the invisible light onto the visible light image. However, combining or mixing the image data can result in a situation where the invisible light component is too prominent, too bright, too scattered, or too faded. Summary of the Invention

[0003] One aspect of the present invention provides an enhanced fluorescence imaging system that includes a light source configured to emit invisible light and visible light, and an image sensor that includes a plurality of pixels configured to capture invisible light image data and visible light image data. The enhanced fluorescence imaging system is configured to generate a video image on a display, and includes data processing hardware in communication with the image sensor and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including determining an invisible light value for each of the plurality of pixels. The invisible light value is associated with the amount of invisible light captured by the image sensor. The operations further include: applying a color map to each invisible light value to associate the invisible light value with a selected color, thereby generating an invisible light selected color value; and weighting a visible light chromaticity value of the visible light image data with an invisible light chromaticity value of the invisible light selected color value to generate a weighted chromaticity value. The operations further include combining a brightness value of each pixel of the visible light image data with the weighted chromaticity value to enhance the video image.

[0004] The implementation of the present invention may include one or more of the following optional features. In some implementations, the color mapping is configured to transform each non-visible light value into a set of values corresponding to a selected color. The magnitude of the non-visible light value may be associated with the intensity of the selected color. In some examples, the selected color is green. Optionally, weighting the visible light chromaticity values of the visible light image data with the non-visible light chromaticity values of the non-visible light selected color values includes: converting the relevant visible light image data into visible light luminance values, visible light blue-difference chromaticity values, and visible light red-difference chromaticity values for each pixel; and converting the non-visible light selected color values into non-visible light luminance values, non-visible light blue-difference chromaticity values, and non-visible light red-difference chromaticity values. Weighting the visible light chromaticity values of the visible light image data with the non-visible light chromaticity values of the non-visible light selected color values may further include: weighting the visible light blue-difference chromaticity value and the non-visible light blue-difference chromaticity value based on a weighting factor to generate a weighted blue-difference chromaticity value; and weighting the visible light red-difference chromaticity value and the non-visible light red-difference chromaticity value based on the weighting factor to generate a weighted red-difference chromaticity value.

[0005] In some implementations, the weighting factor is based on the non-visible light value of the relevant pixel. In the case where the non-visible light value is lower than a first threshold, the weighted blue-difference chromaticity value may be equal to the visible light blue-difference chromaticity value, and the weighted red-difference chromaticity value may be equal to the visible light red-difference chromaticity value. In the case where the non-visible light value is higher than a second threshold, the weighted blue-difference chromaticity value may be equal to the non-visible light blue-difference chromaticity value, and the weighted red-difference chromaticity value may be equal to the non-visible light red-difference chromaticity value. In the case where the non-visible light value is between the first threshold and the second threshold, the weighted blue-difference chromaticity value may be between the visible light blue-difference chromaticity value and the non-visible light blue-difference chromaticity value, and the weighted red-difference chromaticity value may be between the visible light red-difference chromaticity value and the non-visible light red-difference chromaticity value. Optionally, each non-visible light value is between 0 and 4095.

[0006] Another aspect of the present invention provides a fluorescence imaging system, which includes a light source configured to emit invisible light and visible light, and an image sensor including a plurality of pixels configured to capture invisible light image data and visible light image data. The fluorescence imaging system is configured to generate a video image on a display, and includes data processing hardware communicating with the image sensor and memory hardware communicating with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including determining an invisible light value for each of the plurality of pixels. The invisible light value is associated with the amount of invisible light captured by the image sensor. The operations further include adding the invisible light value of each pixel to a selected color of the visible light image data to generate an additive light value. The operations further include: applying a color map to each invisible light value to associate the invisible light value with a selected color, thereby generating an invisible light selected color value; and weighting a visible light chromaticity value of the visible light image data with an invisible light chromaticity value of the invisible light selected color value to generate a weighted chromaticity value. The operations further include combining a brightness value of the visible light image data with the weighted chromaticity value to generate a combined brightness-chromaticity light value. The operations further include: converting the combined brightness-chromaticity light value to a replacement color light value; and weighting the additive light value with the replacement color light value to generate a weighted color light value, thereby enhancing the video image.

[0007] This aspect may include one or more of the following optional features. In some implementations, adding the invisible light value to the selected color of a pixel of the visible light image data includes: for each pixel, determining a set of visible RGB values; for each pixel, determining a set of invisible light RGB values based on the invisible light image data; and for each pixel, adding the set of invisible light RGB values to the set of visible RGB values. In some examples, weighting the additive light value with the replacement color light value includes weighting based on a weighting factor. The weighting factor may be based on the invisible light value of each relevant pixel. In some implementations, the weighting factor is based on the invisible light value. Optionally, the weighting factor is based on the additive light value. The weighting factor may be based on the bit depth of the image sensor. In some examples, the weight of each additive light value is inversely correlated with the associated invisible light value.

[0008] Another aspect of the present invention provides a method for weighting image values of a fluorescence imaging system, the fluorescence imaging system including a light source configured to emit invisible light and visible light and an image sensor including a plurality of pixels configured to capture invisible light image data and visible light image data. The fluorescence imaging system is configured to generate a video image on a display. The method includes determining an invisible light value for each of the plurality of pixels. The invisible light value is associated with the amount of invisible light captured by the image sensor. The method further includes adding the invisible light value of each pixel to a selected color of the visible light image data to generate an additive light value. The method further includes: applying a color map to each invisible light value to associate the invisible light value with a selected color, thereby generating an invisible light selected color value; and weighting a visible light chrominance value of the visible light image data with an invisible light chrominance value of the invisible light selected color value to generate a weighted chrominance value. The method further includes: combining a luminance value of the visible light image data with the weighted chrominance value to generate a combined luminance-chrominance light value; and converting the combined luminance-chrominance light value to a replacement color light value. The method further includes weighting the additive light value with the replacement color light value to generate a weighted color light value.

[0009] One or more implementations of the invention are elaborated in detail in the following figures and description. Other aspects, features, and advantages will be apparent from the description, figures, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The embodiments illustrated in the figures are illustrative and exemplary in nature and are not intended to limit the subject matter defined by the claims. The following description of the illustrative embodiments can be understood when read in conjunction with the following figures, where like structures are denoted with like reference numerals and in which:

[0011] Figure 1 is a schematic diagram of an example system for enhanced fluorescence imaging.

[0012] Figure 2 is a perspective view of a known Bayer filter array image sensor.

[0013] Figure 3 is a table of invisible pixel values color-mapped to a set of RGB values.

[0014] Figure 4 is a graph of a weighting function for selected YCbCr values.

[0015] Figure 5 are three tables of replacement YCbCr values.

[0016] Figure 6Schematic diagram of another example system for enhanced fluorescence imaging.

[0017] Figure 7 Graph of the look-up table.

[0018] Figure 8 Three tables for adding RGB values.

[0019] Figure 9 Plot of the combined weighting function.

[0020] Figure 10 Flowchart of an example method for enhanced fluorescence imaging.

[0021] Figure 11 Schematic diagram of an example computing device that can be used to implement the systems and methods described herein. Detailed Description

[0022] The implementation herein is directed to an enhanced fluorescence imaging system that includes an imaging sensor for capturing visible light data and non-visible light data. The system combines, mixes, or weights the visible light data and the non-visible light data together to provide enhanced visible light data image coverage, wherein the visible indication of the intensity of the non-visible light is not too obvious, too bright, too scattered, or too faded.

[0023] Many devices, such as medical tools, include imaging devices for capturing visible white light images. For example, an endoscopic system in its most basic form includes a rigid or flexible tube having a light source and an imaging system. The flexible tube passes through a patient's orifice (e.g., the mouth), and the imaging system records the image illuminated by the light.

[0024] In addition to visible white light, many medical devices (such as endoscopic systems) are also capable of emitting light of other spectra (i.e., non-visible light). For example, an endoscopic system typically also emits infrared light to support fluorescence imaging. The infrared light is absorbed by a fluorescent dye, which in turn emits fluorescence. As used herein, the term "fluorescent dye" refers to a dye approved for medical use that is configured to reflect infrared light, such as indocyanine green (ICG). ICG has a peak spectral absorption in the near-infrared spectrum of approximately 800 nm. ICG emits fluorescence when illuminated by light between 750 nm and 950 nm. After the endoscopic system irradiates ICG with near-infrared light, it detects this fluorescence and images it to provide an image to a display that visually indicates both visible light and non-visible light, for example. For example, the endoscopic system can convert the non-visible light into a selected color and overlay the selected color representing the non-visible light on the visible light image.

[0025] An endoscope system can be equipped with one or more image sensors to image both white light (i.e., visible light) and infrared light. For example, some endoscopes are equipped with three - charge - coupled - device (3CCD) cameras. The 3CCD camera uses a prism to split the received light into three beams, one of which is directed to a red CCD, one to a green CCD, and one to a blue CCD. In some examples, the endoscope system is equipped with multiple image sensors, where each sensor includes pixels dedicated to a corresponding frequency band using a color array filter commonly known as a Bayer filter (see Figure 2 ). In other examples of the endoscope system, one image sensor is configured to capture both visible light and infrared light simultaneously. Other endoscope systems can be equipped with multiple image sensors that separately and simultaneously capture both visible light and infrared light.

[0026] For illustrative purposes, a description of a system for optimizing a hybrid video stream is provided in the context of endoscope system 100. However, it should be understood that the fluorescence image intensifier can be used in other applications, illustratively including exoscopes, tube scopes, videoscopes, and other systems having two or more types of illumination and one or more image sensors. Additionally, although the system is described for medical applications using fluorescent dyes, it should be understood that non - medical applications using other combinations of visible and invisible light can also benefit from the same principles.

[0027] Refer to Figure 1 , in some implementations, examples of endoscope system 100 include one or more light sources 110. The light source 110 emits both visible light (VL) 112a (e.g., white light) and non - visible light (NVL) 114a (e.g., infrared light, etc.). In some examples, the light source 110 alternates between emitting VL 112a and NVL light 114a. That is, in some examples, the light source 110 rapidly switches between emitting VL 112a and NVL light 114a. In other examples, the light source emits simultaneously. The VL 112a illuminates the surgical site of the system 100. The light source 110 can include one or more light - emitting diodes (LEDs) or any other suitable light - emitting device. Separate light sources can emit VL 112a and NVL light 114a respectively. In some examples, the light source 110 is included in the camera head unit 102.

[0028] The light 112a, 114a emitted by the light source 110 travels along the light guide 116 (e.g., an optical fiber) and illuminates or irradiates the target area 10 (e.g., the lumen of a patient) after leaving the light guide 116. The reflected VL 112b (i.e., the VL 112a reflected from the target area 10) and the emitted NVL light 114b (e.g., fluorescence (FL)) emitted by, for example, ICG irradiated by the NVL light 114a or any other form of invisible light are guided back to, for example, the dichroic prism 120 through the optical path 115. The dichroic prism 120 splits the received light into two beams of different wavelengths. That is, the dichroic prism 120 distributes the received light (which may include the reflected VL 112b and / or NVL 114b) to the image sensor 130. The image sensor 130 may include a VL image sensor 130a and an NVL image sensor 130b. For example, any reflected VL 112b (i.e., visible light) passing through the prism 120 can be guided to the VL image sensor 130a, while any NVL 114b passing through the prism 120 can be guided to the NVL image sensor 130b (i.e., light with a wavelength between 800 nm and 1200 nm). In some examples, the prism 120 and the image sensor 130 are also included in the camera head unit 102. Although a dichroic prism and two separate image sensors are illustrated, any means for capturing image data representing both the reflected VL 112b and the NVL 114b is within the spirit and scope of the appended claims.

[0029] The image sensor 130 can be a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD). It should be understood that any pixelated image sensor 130 known currently or developed in the future can be modified and adopted for use herein. In some implementations, the image sensor 130 includes a color filter array (CFA). Now referring Figure 2 , the image sensor 130 can include a CFA (sometimes referred to as a Bayer filter). The Bayer CFA includes a mosaic CFA for arranging red, green, and blue color filters on a grid of light sensors. As Figure 2 shown, the filter pattern is a commonly used 50% green, 25% red, and 25% blue color filter array. Thus, each pixel is filtered to record only one of the three colors, and various well-known demosaicking algorithms are used to obtain a full-color image. In some examples, the VL image sensor 130a and the NVL image sensor 130b are different sensors with the same or different resolutions. In other examples, the image sensor 130 is the same sensor. The same sensor (e.g., the same resolution, geometry, etc.) generally improves and facilitates the manufacturing, assembly, and alignment of the system 100. In still other examples, a single image sensor captures both the reflected VL 112b and the reflected NVL 114b.

[0030] Continuing reference Figure 1 , the sensor 130 sends the VL data 132a and the NVL data 132b to the camera control unit (CCU) 140. In some examples, the CCU 140 may be included in the camera head unit 102, while in other examples it is remote from the camera head unit 102. The CCU 140 includes computing resources 142 (e.g., data processing hardware) and storage resources 144 (e.g., memory hardware). In some implementations, the CCU 140 is physically configured at the system 100 (e.g., within the camera head unit 102) and communicates with the image sensor 130 in a wired manner. In other implementations, the CCU 140 communicates wirelessly with the image sensor 130 (e.g., via wireless, Bluetooth, etc.) and may be remote from the image sensor 130 and / or the system 100. In this case, the CCU 140 can correspond to any suitable computing device 1100 such as a desktop workstation, a laptop workstation, or a mobile device (e.g., a smartphone or a tablet) (see Figure 11 ). In still other implementations, the data 132 can be stored in the non-volatile storage of the system 100 (e.g., a thumb drive) and subsequently removed for processing at the data processing hardware 142 and the memory hardware 144 remote from the image sensor 130.

[0031] In some implementations, the VL image data 132a received by the CCU 140 includes data of a plurality of pixels in the RGB format. The RGB format or color model is an additive color model that represents all colors via three chrominance levels of three colors (red, green, and blue). Each pixel of the VL image data 132a will have a corresponding set of VL RGB values 134 (i.e., VL RGB red value 134R, VL RGB green value 134G, and VL RGB blue value 134B). Each set of VL RGB values 134 includes numerical values between a minimum VL RGB value and a maximum VL RGB value. The minimum and maximum values can depend on the image sensor 130. More specifically, these values can depend on the color bit depth of the image sensor 130. For example, when the image sensor 130 has a 12-bit bit depth, the minimum value can be 0 and the maximum value can be 4095. The VL RGB values 134 can be based on, for example, processing the pixel information from the image sensor 130 using a color filter array (e.g., a Bayer filter) ( Figure 2 ).

[0032] In some implementations, data processing hardware 142 executes (i.e., executes using instructions stored on storage resource 144) image replacer 150. Image replacer 150 receives NVL data 132b at intensity pixel value determiner 200. In some examples, intensity value determiner 200 determines invisible light values 210 (or NVL values) for respective pixels 136 of NVL image data 132b. For example, image data captured by an invisible light image sensor 130b can be represented as a grayscale image, where the value of each pixel of the image ranges from a minimum NVL value to a maximum NVL value. That is, in some examples, the more NVL detected or captured by a pixel, the larger the NVL value 210 will be (up to the maximum NVL value). For example, a pixel 136 having a minimum NVL value 210 (e.g., zero) can be represented as black on the grayscale image, while a pixel 136 having a maximum NVL value 210 (e.g., 4095 for 12-bit depth) can be represented as white on the grayscale image.

[0033] In some implementations, the NVL values 210 are passed to color mapper 300. Color mapper 300 maps or assigns or converts the NVL values 210 to a selected color map in RGB format to generate a set of invisible light RGB selected color values 310. Each set of NVL RGB selected color values 310 includes an NVL RGB red value 310R, an NVL RGB green value 310G, and an NVL blue value 210B. Although in the provided example the selected color is green only, any other color map can be selected, including blue only, red only, mixed monochromatic colors such as orange, yellow, purple, etc., or a multicolor that is commonly spoken of as a "heat map" look-up table (LUT).

[0034] Now referring to Figure 3 , three exemplary NVL values 210 (NVL) and the resulting sets of invisible light RGB selected color values 310 after color mapping are shown. Here, "R" represents the NVL RGB red value 310R, "G" represents the NVL RGB green value 310G, and "B" represents the NVL RGB blue value 310B. In this example, color mapper 300 maps each NVL value 210 to green. For example, when NVL is equal to 50, R is equal to 0, G is equal to 50, and B is equal to 0. When NVL is equal to 150, R is equal to 0, G is equal to 150, and B is equal to 0. Similarly, when NVL is equal to 250, R is equal to 0, G is equal to 250, and B is equal to 0. That is, in some implementations, color mapper 300 takes RGB values of all zeros and adds the NVL value 210 to the selected color (green in the shown example).

[0035] YCbCr is a color space that separates colors into a luminance component (Y), a blue-difference component (Cb), and a red-difference component (Cr). In some implementations, (such as Figure 1 depicted) the YCbCr converter 400 receives VL RGB values 134 and the NVL RGB selected color value set 310 (each in RGB format), and converts each value into VL YCbCr values 410 and NVL YCbCr values 412, respectively. The VL YCbCr values 410 each include a VL luminance (Y) value 410L, a VL blue-difference chrominance (Cb) value 410B, and a VL red-difference chrominance (Cr) value 410R. Similarly, the NVL YCbCr values 412 each include an NVL luminance value 412L, an NVL blue-difference chrominance value 412B, and an NVL red-difference chrominance value 412R. The conversion from the RGB color space to the YCbCr color space can be achieved by commonly known methods.

[0036] In some examples, ( Figure 1 depicted) the replacer 500 receives VL YCbCr values 410 and NVL YCbCr values 412 from the YCbCr converter 400. The replacer 500 can combine or mix or weight the VL blue-difference chrominance value 410B with the NVL blue-difference chrominance value 412B, and the VL red-difference chrominance value 410R with the NVL red-difference chrominance value 412R. The mixing or combination can be based on a weighting factor 510. Optionally, the weighting factor varies for each pixel based on the NVL value 210 (i.e., each pixel is weighted differently according to the relevant NVL value 210).

[0037] Now referring to Figure 4 , an exemplary graph 450 showing the replacement weighting function 512 is presented. Here, when the NVL value 210 is below the first replacement threshold 514, the weighting factor 510 is 0; and when the NVL value 210 is above the second replacement threshold 516, the weighting factor is 1. When the NVL value 210 is above the first replacement threshold 514 and below the second replacement threshold 516 (i.e., the NVL value 210 is between the two thresholds 514 and 516), the weighting factor 510 can be in linear proportion to the NVL value 210. Optionally, the weighting factor 510 can be in an exponential, logarithmic, etc. proportion. The first replacement threshold 514 and the second replacement threshold 516 can be adjusted to be between the minimum NVL value and the maximum NVL value to produce various desired results in the final enhanced image frame 170. In some examples, the replacer 500 weights the VL CbCr values 410B, 410R and the NVL CbCr values 412B, 412R using Equation 1 (generated below) to generate a weighted blue-difference chrominance value 520B (Cb w ) and a weighted red-difference chrominance value 520R (Cr w ), where w is the weighting factor.

[0038] (w)(Cb NVL Cr NVL )+(1-w)(Cb VL Cr VL )=Cb w Cr w (1)

[0039] Therefore, in some examples, when the NVL value 210 is lower than the first replacement threshold 514, the weighting factor 510 is zero, and Cb w Cr w is equal to Cb VL Cr VL (i.e., since the weighting factor makes the contribution of Cb NVL Cr NVL zero, the NVL image data is not used). When the NVL value 210 is higher than the second replacement threshold 516, the weighting factor 510 is 1, and Cb w Cr w is equal to Cb NVL Cr NVL (i.e., since the weighting factor makes the contribution of Cb VL Cr VL zero, Cb VL Cr VL is completely replaced by Cb NVL Cr NVL . Between the first replacement threshold 514 and the second replacement threshold 516, as the NVL value 210 increases, the weighting factor 510 increases the weight or contribution of the NVL CbCr values 412B, 412R to the Cb w Cr w value (while reducing the weight or contribution of the VL CbCr values 410B, 410R).

[0040] In some implementations, the replacer 500 combines the VL luminance value 410L with the weighted blue-difference chrominance value 520B and the red-difference chrominance value 520R to generate a replacement color light value 520 (i.e., the replacement YCbCr value or Y VL Cb w Cr w ). That is, the replacer 500 can replace the CbCr chrominance values of the VL YCbCr value 410 with the weighted CbCr values 520B, 520R while maintaining the original VL luminance value 410L.

[0041] Now refer to Figure 5, Tables 560a - c provide examples of the replacement values of the CbCr values by the replacement unit 500. Table 560a assumes that the NVL value 210 is 50, while the VL RGB red value 134R is 200, the VL RGB green value 134G is 150, and the VL RGB blue value 134B is 150. In this example, the depth is 8 bits, so the maximum value of each value is 255. Additionally, in this example, the first replacement threshold 514 is 100, and the second replacement threshold is 200. Still referring to Table 560a, after converting the VL RGB value 134 to YCbCr at (i.e., at the YCbCr converter 400), the VL luminance value 410L is 158, the VL blue - difference chrominance value 410B is 121, and the VL red - difference chrominance value 410R is 150. After converting the NVL RGB selected color value 310 to YCbCr, the NVL luminance value 412L is 41, the NVL blue - difference chrominance value 412B is 113, and the NVL red - difference chrominance value 412R is 110. That is, since the NVL value (i.e., 50) is lower than the first replacement threshold 514 (i.e., 100), the weighted CbCr values 520B, 520R are equal to the VL CbCr values 410B, 410R. Regardless of the NVL value 210, the replacement unit 500 combines the NVL luminance value 412L with the weighted CbCr values 520B, 520R to generate the final YCbCr value 520.

[0042] Table 560b assumes that the NVL value 210 is 150, which is higher than the first replacement threshold 514 (100 in this example) and lower than the second replacement threshold 516 (200 in this example). After conversion to YCbCr, the NVL YCbCr values include an NVL luminance value 412L of 92, an NVL blue - difference chrominance value 412B of 84, and an NVL red - difference chrominance value 412R of 73. Using the same VL YCbCr values 410 (i.e., Y = 158, Cb = 121, and Cr = 150), the weighted blue - difference chrominance value 520B is 103, and the weighted red - difference chrominance value 520R is 112. In this case, since the NVL value 210 is between the first replacement threshold 514 and the second replacement threshold 516, the weighted CbCr values 520B, 520R are a combination of the NVL value and the VL value based on the weighting factor 510. Again, regardless of the NVL value 210, the replacement unit 500 combines the NVL luminance value 412L with the weighted CbCr values 520B, 520R to generate the final YCbCr value 520.

[0043] Table 560c assumes that the NVL value 210 is 250, which is higher than the first replacement threshold 514 and the second replacement threshold 516 (100 and 200 respectively in this example). After conversion to YCbCr, the NVL YCbCr values include the NVL luminance value 412L of 142, the NVL blue-difference chrominance value 412B of 55, and the NVL red-difference chrominance value 412R of 36. Using the same VL YCbCr values 410 (i.e., Y = 158, Cb = 121, and Cr = 150), the weighted blue-difference chrominance value 520B is 55, and the weighted red-difference chrominance value 520R is 36. That is, since the NVL value 210 exceeds the second replacement threshold, the weighted CbCr 520B, 520R values are equal to the NVL CbCr values 412B, 412R (i.e., the VL CbCr values are completely replaced). Again, regardless of the NVL value 210, the replacer 500 combines the NVL luminance value 412L with the weighted CbCr values 520B, 520R to generate the final YCbCr value 520.

[0044] Thus, as Figure 5 shown in the table, in some implementations, when the NVL value 210 is below the first replacement threshold 514, the final YCbCr value 520 completely eliminates the NVL CbCr values 412B, 412R (i.e., the final CbCr value 520 is equal to the VL CbCr value 410). When the NVL value 210 is higher than the second replacement threshold 516 (the second replacement threshold 516 is greater than the first replacement threshold 514), the VL CbCr values 410B, 410R are completely replaced by the NVL CbCr values 412B, 412R (i.e., the final CbCr value 520 is equal to the NVL CbCr value 412). For NVL values 210 between the first replacement threshold 514 and the second replacement threshold 516, the replacer 500 can combine or mix or weight together the VL CbCr values 410B, 410R and the NVL CbCr values 412B, 412R. Regardless of the NVL value 210, the VL luminance value 410L is combined with the weighted CbCr values 520B, 520R. The image replacer 150 processes each pixel of the VL image data 132a and the NVL image data 132b to generate an enhanced image frame 170, which includes the VL luminance value 410L and the weighted CbCr values 520B, 520R for each pixel of the image to be displayed on the display 180. In some examples, the final YCbCr value 520 can be converted to the RGB format and / or additional filtering or processing can be performed before sending it to the display.

[0045] In some implementations, the CCU 140 simultaneously executes the image replacer 150 and the image adder 610. Now refer to Figure 6, both the image replacer 150 and the image adder 610 receive the VL image data 132a and the NVL image data 132b, and obtain the NVL value 210 of each pixel 136. The image adder 610 can add the NVL value 210 of each pixel to a selected color (e.g., green) of the corresponding VL RGB value 134. For example, when the selected color is green, each NVL value 210 can be directly added to the VL RGB green value 134G to generate an additive light value 620 (also referred to herein as an added RGB value). In some implementations, the added RGB value 620 is further processed or weighted. For example, Figure 7 An exemplary graph showing, for example, a weighting function or multiplier for removing low-level noise. In this example, the x-axis represents the NVL value 210 (from 0 up to the maximum allowable value). The y-axis represents a multiplier or weighting factor ranging from 0 to 1. The value of the multiplier is multiplied by the NVL value 210 to generate a modified NVL value 210 or an added RGB value 620. In this case, above the threshold, the multiplier is 1 and the full NVL value 210 is used, while below the threshold, the multiplier is zero and the NVL value 210 is also zero.

[0046] Now refer to Figure 8 , three exemplary tables 860a, 860b, and 860c show the added RGB values 620 in three examples reflecting the Figure 5 example in. Here, the bit depth is again 8 bits, so each RGB value has a maximum value of "255". Table 860a shows an NVL value of "50". As with tables 560a - c, the VL RGB value 134 is 200 for red, 150 for green, and 150 for blue. After the image adder 610 adds the NVL value 210 to the selected color (green in this example), the added RGB value 620 includes 200 for red (the same as the VL RGB red value 134R), 200 for green (the sum of "150" of the VL RGB green value 134G and "50" of the NVL value 210), and 150 for blue (the same as the VL RGB blue value 134B).

[0047] Table 860b shows the same VL RGB value 134 with the NVL value 210 being "150". Similar to Table 860a, the added RGB value 620 includes the red value "200" and the blue value "150". Here, when the sum of the VL RGB green value "150" and the NVL value 210 "150" equals 300 (exceeding the maximum value "255"), the added RGB green value is 255. Thus, in this case, adding the NVL value 210 to the VL RGB green value 134G causes the value to be clipped. That is, the actual value is lost due to exceeding the maximum value allowed by the bit depth. Table 860C shows the same VL RGB value 134 with the NVL value 210 being "250". Similar to the previous Tables 860a and 860b, the added RGB value 620 includes the red value "200" and the blue value "150". In this case, since the sum of the VL RGB green value "150" and the NVL value 210 "250" equals 400 (exceeding the maximum value "255"), the added RGB green value is again 255. Thus, although the NVL value 210 in Table 860c is greater than the NVL value 210 in Table 860b, due to clipping, these two values are the same in the added RGB value 620, and the difference in NVL is lost.

[0048] To mitigate clipping, the image replacer 150 can be executed simultaneously with the image adder 610. Referring back Figure 6 to, the image combiner 650 can receive the final YCbCr value 520 from the image replacer 150 and the added RGB value 620 from the image adder. In some examples, the image combiner 650 converts the final YCbCr value 520 from the image replacer into the RGB format to generate the replacement RGB value 630. Optionally, the image replacer 150 can generate the replacement RGB value 630 before sending it to the image combiner 650.

[0049] In some implementations, the image combiner 650 combines or mixes or weights the added RGB value 620 with the replacement RGB value 630 based on a combined weighting factor 910 ( Figure 9 ). In some examples, the image combiner 650 uses Equation 2 (provided below) to weight the added RGB value 620 with the replacement RGB value 630 using the weighting factor 910 to generate the final RGB value 660, where u represents the weighting factor 910, RGB add represents the added RGB value 620, and RGB rep represents the replacement RGB value 630.

[0050] (1 - u)(RGB add )+(u)(RGB rep ) = RGB final (2)

[0051] Now referring to Figure 9 FIG. 900, which illustrates an exemplary graph of a combined weighting function 920. Here, the x-axis is the combined weighting factor 910, which can be a value between 0 and 1. The y-axis of FIG. 900 is the NVL value 210 divided by the maximum allowable NVL value. The maximum value can be based on the color bit depth of the image sensor. For example, for an 8-bit depth, the maximum value will be 255, and for a 12-bit depth, the maximum value will be 4095. As the NVL value 210 increases (i.e., the ratio between the NVL value 210 and the maximum value approaches 1), the combined weighting factor 910 increases. In some implementations, when the NVL value 210 equals the maximum value, the combined weighting factor 910 equals 1, and when the NVL value 210 equals zero, the combined weighting factor 910 also equals zero.

[0052] Accordingly, using the combined weighting factor 910, the image combiner 650 combines or blends the added RGB value 620 with the replacement RGB value 630. As the added RGB value 620 approaches the maximum value 914 (i.e., the added RGB value approaches clipping), the replacement RGB value has an increasing weight, such that when the added RGB value 620 equals the maximum value 914, the final RGB value 660 equals the replacement RGB value 630. Similarly, as the added RGB value 620 decreases (i.e., approaches zero), the added RGB value 620 has an increasing weight, such that when the added RGB value 620 equals zero, the final RGB value 660 equals the added RGB value 620.

[0053] After processing and enhancing the image data 132a, 132b, the CCU 140 outputs the enhanced image frame 170 to the display 180. The enhanced image frame 170 includes an image based on the NVL image data 132b and the VL image data 132a. In some examples, the image frame 170 may be further processed (e.g., filtered, etc.) before being sent to the display. The display processes the image frame 170 to generate a visible image (i.e., a picture or video).

[0054] Accordingly, the provided endoscope system 100 can combine or blend visible light image data and invisible light image data (e.g., infrared image data) to generate an enhanced image frame that maps the invisible light to a selected color. The system 100 ensures that after combining or blending the image data using a variable weighting factor based on the intensity of the invisible light, the visible indication (i.e., the selected color overlay) of the invisible light image data is not too obvious, too bright, too scattered, or too faded.

[0055] Figure 10FIG. 0 is a flow chart of an exemplary operation 1000 of an enhanced fluorescence imaging system 100, where the enhanced fluorescence imaging system 100 includes a light source configured to emit invisible light and visible light, and an image sensor including a plurality of pixels configured to capture invisible light image data and visible light image data. The imaging system is configured to generate a video image onto a display. The system also includes data processing hardware in communication with the image sensor, and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include: at step 1002, determining an invisible value for each of the plurality of pixels. The invisible value is associated with the amount of invisible light captured by the image sensor.

[0056] At step 1004, the operation further includes applying a color map to each invisible value to associate the invisible value with a selected color to generate an invisible light selected color value. At step 1006, the operation includes weighting a visible light chromaticity value of the visible light image data with an invisible light chromaticity value of the invisible light selected color value to generate a weighted chromaticity value. At step 1008, the operation includes combining a brightness value of each pixel of the visible light image data with the weighted chromaticity value. The operation further includes: at step 1010, generating an RGB value based on the brightness value of the visible light image data and the weighted chromaticity value; and at operation of step 1012, sending the RGB value to the display.

[0057] Figure 11 FIG. 7 is a schematic diagram of an exemplary computing device 1100 (e.g., data processing hardware 142 and memory hardware 144) that can be used to implement the systems and methods described herein. For example, the computing device 1100 can perform tasks such as controlling the light source 110 (e.g., enabling and disabling the light source, switching between white light and near-infrared (NIR) light, etc.), configuring and communicating with the image sensor 130 (e.g., receiving image data), and implementing and executing one or more components 200, 300, 400, 500, etc. of the system 100. In some examples, the computing device 1100 sends the image data to the display 180. That is, using the data received from the image sensor 130, the computing device 1100 can store and execute instructions or operations to implement the components 200, 300, 400, 500, etc. The computing device 1100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the disclosure described and / or claimed herein.

[0058] The computing device 1100 (e.g., data processing hardware 142) includes a processor 1110, a memory 1120, a storage device 1130, a high-speed interface / controller 1140 connected to the memory 1120 and the high-speed expansion port 1150, and a low-speed interface / controller 1160 connected to the low-speed bus 1170 and the storage device 1130. Each of the components 1110, 1120, 1130, 1140, 1150, and 1160 is interconnected using various buses and can be mounted on a common motherboard or otherwise appropriately mounted. The processor 1110 can process instructions for execution within the computing device 1100, including instructions stored in the memory 1120 or on the storage device 1130 for displaying graphical information of a graphical user interface (GUI) on an external input / output device (such as a display coupled to the high-speed interface 1140). In other implementations, multiple processors and / or multiple buses can be appropriately used along with multiple memories and memory types. Additionally, multiple computing devices 1100 can be connected, where each device provides a part of the necessary operations (e.g., as a server group, a set of blade servers, or a multi-processor system).

[0059] The memory 1120 stores information non-temporarily within the computing device 1100. The memory 1120 can be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 1120 can be a physical device for storing programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by the computing device 1100. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs, etc.). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and magnetic disks or tapes.

[0060] The storage device 1130 can provide large-capacity storage for the computing device 1100. In some implementations, the storage device 1130 is a computer-readable medium. In various different implementations, the storage device 1130 can be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory, or other similar solid-state memory devices or an array of devices (including devices in a storage area network or other configurations). In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods (such as the methods described above, etc.). The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 1120, the storage device 1130, or the memory on the processor 1110, etc.

[0061] The high-speed controller 1140 manages the bandwidth-intensive operations of the computing device 1100, while the low-speed controller 1160 manages the lower bandwidth-intensive operations. This division of responsibilities is merely exemplary. In some implementations, the high-speed controller 1140 is coupled to the memory 1120, the display 1180 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 1150, which can accept various expansion cards (not shown). In some implementations, the low-speed controller 1160 is coupled to the storage device 1130 and the low-speed expansion port 1190. The low-speed expansion port 1190, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, etc. or a networking device such as a switch or a router, for example, via a network adapter.

[0062] As shown, the computing device 1100 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 1100a or be implemented multiple times as part of a laptop computer 1100b or a rack server system 1100c in a group of such servers 1100a.

[0063] The various implementations of the systems and technologies described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system, where the programmable system includes at least one programmable processor, which can be special-purpose or general-purpose and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device and to send data and instructions to a storage system, at least one input device, and at least one output device.

[0064] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level programming and / or object-oriented programming languages and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal for providing machine instructions and / or data to a programmable processor.

[0065] The processes and logical flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware) executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). By way of example, processors suitable for the execution of a computer program include both general and special purpose microprocessors, and any one or more processors in any kind of digital computer. In general, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing the instructions and one or more memory devices for storing the instructions and data. In general, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, by way of example including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0066] To provide interaction with a user, one or more aspects of the present invention may be implemented on a computer having a display device such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user, and a keyboard and an indicating device such as a mouse or trackball via which a user may provide input to the computer, optionally. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback such as, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form including, but not limited to, voice, speech, or tactile input. Additionally, the computer may interact with the user by sending and receiving documents relative to the devices used by the user (e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser).

[0067] While specific embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Additionally, although various aspects of the claimed subject matter have been described herein, these aspects need not be used in combination. Accordingly, the appended claims are intended to cover all such changes and modifications that fall within the scope of the claimed subject matter.

Claims

1. A fluorescence imaging system, which includes a light source configured to emit invisible light and visible light, and an image sensor. The image sensor includes a plurality of pixels configured to capture invisible light image data and visible light image data. The fluorescence imaging system is configured to generate a video image onto a display. The fluorescence imaging system includes: Data processing hardware that communicates with the image sensor; and Memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Determining an invisible light value for each of the plurality of pixels, the invisible light value being associated with the amount of invisible light captured by the image sensor; Applying a color map to each invisible light value to associate the invisible light value with a selected color, thereby generating an invisible light selected color value; Weighting the visible light chrominance value of the visible light image data with the invisible light chrominance value of the invisible light selected color value to generate a weighted chrominance value; and Combining the luminance value of each pixel of the visible light image data with the weighted chrominance value to enhance the video image, wherein weighting the visible light chrominance value of the visible light image data with the invisible light chrominance value of the invisible light selected color value includes: Converting the relevant visible light image data for each pixel into a visible light luminance value, a visible light blue-difference chrominance value, and a visible light red-difference chrominance value; Converting the invisible light selected color value into an invisible light luminance value, an invisible light blue-difference chrominance value, and an invisible light red-difference chrominance value; Weighting the visible light blue-difference chrominance value and the invisible light blue-difference chrominance value based on a weighting factor to generate a weighted blue-difference chrominance value; and Weighting the visible light red-difference chrominance value and the invisible light red-difference chrominance value based on the weighting factor to generate a weighted red-difference chrominance value.

2. The fluorescence imaging system according to claim 1, wherein, The color map is configured to transform each invisible light value into a selected color, the magnitude of the invisible light value being associated with the intensity of the selected color.

3. The fluorescence imaging system according to claim 1, wherein, The selected color is green.

4. The fluorescence imaging system according to claim 1, wherein, The weighting factor is based on the invisible light value of the relevant pixel.

5. The fluorescence imaging system according to claim 4, wherein, In the case where the invisible light value is below a first threshold, the weighted blue-difference chrominance value is equal to the visible light blue-difference chrominance value, and the weighted red-difference chrominance value is equal to the visible light red-difference chrominance value.

6. The fluorescence imaging system according to claim 5, wherein, In the case where the invisible light value is above a second threshold, the weighted blue-difference chrominance value is equal to the invisible light blue-difference chrominance value, and the weighted red-difference chrominance value is equal to the invisible light red-difference chrominance value.

7. The fluorescence imaging system according to claim 6, wherein, In the case where the invisible light value is between the first threshold and the second threshold, the weighted blue-difference chrominance value is between the visible light blue-difference chrominance value and the invisible light blue-difference chrominance value, and the weighted red-difference chrominance value is between the visible light red-difference chrominance value and the invisible light red-difference chrominance value.

8. The fluorescence imaging system according to claim 1, wherein, Each invisible light value is between an invisible light minimum value and an invisible light maximum value based on the pixel bit depth.

9. A fluorescence imaging system, which includes a light source configured to emit invisible light and visible light, and an image sensor. The image sensor includes a plurality of pixels configured to capture invisible light image data and visible light image data. The fluorescence imaging system is configured to generate a video image onto a display. The fluorescence imaging system includes: Data processing hardware that communicates with the image sensor; and Memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Determining an invisible light value for each of the plurality of pixels, the invisible light value being associated with the amount of invisible light captured by the image sensor; Add the invisible light value of each pixel to the selected color of the visible light image data to generate an additive light value; Apply a color mapping to each invisible light value to associate the invisible light value with the selected color, thereby generating an invisible light selected color value; Weight the visible light chromaticity value of the visible light image data and the invisible light chromaticity value of the invisible light selected color value to generate a weighted chromaticity value; Combine the luminance value of the visible light image data with the weighted chromaticity value to generate a combined luminance-chromaticity light value; Convert the combined luminance-chromaticity light value to a replacement color light value; and Weight the additive light value and the replacement color light value to generate a weighted color light value, thereby enhancing the video image.

10. The fluorescence imaging system according to claim 9, wherein, Adding the invisible light value to the selected color of the pixel of the visible light image data includes: For each pixel, determine the visible RGB value; For each pixel, determine the invisible light RGB value based on the invisible light image data; and For each pixel, add the invisible light RGB value to the visible light RGB value.

11. The fluorescence imaging system according to claim 10, wherein, Weighting the additive light value and the replacement color light value includes: weighting based on a weighting factor that is based on the invisible light value of each relevant pixel.

12. The fluorescence imaging system according to claim 11, wherein, The weighting factor is based on the invisible light value.

13. The fluorescence imaging system according to claim 11, wherein, The weighting factor is based on the additive light value.

14. The fluorescence imaging system according to claim 11, wherein, The weighting factor is based on the bit depth of the image sensor.

15. The fluorescence imaging system according to claim 10, wherein, The weight of each additive light value is inversely correlated with the relevant invisible light value.

16. A method for weighting image values of a fluorescence imaging system, the fluorescence imaging system including a light source configured to emit invisible light and visible light and an image sensor, the image sensor including a plurality of pixels configured to capture invisible light image data and visible light image data, the fluorescence imaging system being configured to generate a video image on a display, the method comprising: Determine the invisible light value for each pixel among the plurality of pixels, the invisible light value being associated with the amount of invisible light captured by the image sensor; Add the invisible light value of each pixel to the selected color of the visible light image data to generate an additive light value; Apply a color mapping to each invisible light value to associate the invisible light value with the selected color, thereby generating an invisible light selected color value; Weight the visible light chromaticity value of the visible light image data and the invisible light chromaticity value of the invisible light selected color value to generate a weighted chromaticity value; Combine the luminance value of the visible light image data with the weighted chromaticity value to generate a combined luminance-chromaticity light value; Convert the combined luminance-chromaticity light value to a replacement color light value; And Weight the additive light value and the replacement color light value to generate a weighted color light value.

Images