Method, processor, and medical fluorescence observation apparatus using a color conversion function with color dependency

Abstract

This application relates to an image processor and a computer-implemented image processing method for generating a digital output color image of an object using a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope. A digital white light color image (114) and a digital fluorescence color image (112) of the object (106) are acquired. The digital white light color image is recorded in a first imaging spectrum and includes a plurality of first pixels (150a). Each first pixel (150a) includes a first set ({R1, B1, G1}) of color space coordinates ({R1, B1, G1}) in a first set of color bands (R, G, B). The digital fluorescence color image (112) includes a plurality of second pixels (150b). Each second pixel (150b) includes a second set ({R2, B2, G2}) of color space coordinates (R2, G2, B2) in a second set of color bands. The second imaging spectrum overlaps with the fluorescence emission spectrum of at least one phosphor and is different from the first imaging spectrum (202). Both spectra overlap with the visible spectrum. A digital output color image (160) is generated from the digital white light color image and the digital fluorescence color image, and the digital output color image includes a plurality of output pixels (150c). The color (R * , G * , B * ) of the output pixel is calculated by applying a color conversion function (140) to an input merger ({R1, R2, G1, G2, B1, B2}) of the first set of color space coordinates of the first pixel and the second set of color space coordinates of the second pixel. The color conversion function is applied depending on the color space coordinates within the input merger. The digital white light image and the fluorescence image are processed as multi-spectrum images having more color bands and higher color accuracy. This makes it possible to apply different color conversion functions to different tissue types.

Description

Technical Field

[0001] The claimed subject matter relates to a computer-implemented image processing method and an image processor for generating a digital output color image of an object using a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope. Further, the claimed subject matter relates to a method for operating such a medical fluorescence observation device and to the medical fluorescence observation device itself.

Background Art

[0002] In existing medical fluorescence observation devices, fluorescence and anatomical images are recorded by a single camera. Such a setup may be used for fluorescence imaging of pPIX after administration of 5-ALA to reveal tumors. The selection of the optical filter is made such that the camera captures both the fluorescence at the red wavelength and a part of the excitation light at the blue wavelength. The excitation light reveals a part of the anatomical structure of the tissue, while the fluorescence indicates tumors. The optical filters used today are standardized for pPIX imaging. This setup may also be used for other fluorophores such as ICG.

[0003] This type of fluorescence imaging is particularly helpful in neurosurgical oncology, but still has significant drawbacks. The blue excitation light used to visualize the anatomical structure of the tissue has a very low intensity and provides poor visualization because the anatomical structure is shown only in a blue monochrome tone. In particular, under blue light, bleeding is very difficult to see or may not be visible at all. Therefore, this setup is not optimal for live surgical guidance.

[0004] Other medical fluorescence observation devices use a grayscale camera to record fluorescence and a color camera to record white light images. In such a setup, the fluorescence image and the white light image are digitally mixed and the fluorescence is marked with a pseudo-color. The drawback here is that this pseudo-color cannot accurately represent the color of the fluorescence as perceived by the human eye.

Summary of the Invention

Problems to be Solved by the Invention

[0005] From the above viewpoints, it is necessary to provide an apparatus and a method with improved representation of anatomical images.

Means for Solving the Problems

[0006] This need is addressed by an image processor for a medical fluorescence observation apparatus such as a fluorescence microscope or a fluorescence endoscope. The processor acquires a digital white light color image of an object recorded in a first imaging spectrum. The digital white light color image includes a plurality of first pixels, and each first pixel includes a first set of color space coordinates in a first set of color bands. The processor also acquires a digital fluorescence color image of the object recorded in a second imaging spectrum. The digital fluorescence color image includes a plurality of second pixels, and each second pixel includes a second set of color space coordinates in a second set of color bands. The second imaging spectrum overlaps with the fluorescence emission spectrum of at least one phosphor. The second imaging spectrum is different from the first imaging spectrum. Both the first and second imaging spectra overlap with the visible spectrum. The processor generates a digital output color image from the digital white light color image and the digital fluorescence color image. The digital output color image is configured to include a plurality of output pixels. The image processor is configured to calculate the color of the output pixels by applying a color conversion function to the input combination of the first set of color space coordinates of the first pixels and the second set of color space coordinates of the second pixels. The application of the color conversion function depends on the color space coordinates within the input combination.

[0007] Furthermore, this demand is addressed by a computer-implemented image processing method for a fluorescence observation apparatus such as a fluorescence microscope or a fluorescence endoscope, and the computer-implemented image processing method includes the following steps, namely, a step of obtaining a digital white light color image of an object recorded in a first imaging spectrum, wherein the digital white light color image includes a plurality of first pixels, and each first pixel includes a first set of color space coordinates in a first set of color bands; a step of obtaining a digital fluorescence color image of the object recorded in a second imaging spectrum, wherein the digital fluorescence color image includes a plurality of second pixels, and each second pixel includes a second set of color space coordinates in a second set of color bands, the second imaging spectrum overlaps with the fluorescence emission spectrum of at least one phosphor, the second imaging spectrum is different from the first imaging spectrum, and both the first and second imaging spectra overlap with the visible spectrum; and a step of generating a digital output color image from the digital white light color image and the digital fluorescence color image, wherein the digital white light color image includes a plurality of output pixels. The color of the output pixels is calculated by applying a color conversion function to the input combination of the first set of color space coordinates of the first pixels and the second set of color space coordinates of the second pixels, and the color conversion function is applied depending on the color space coordinates in the input combination.

[0008] The above-described image processor and computer-implemented method can perform different color conversions for different colors in an object. By making the application of the color conversion function depend on the input combination of two sets of color space coordinates, a much more accurate color conversion scheme becomes possible. The input combination includes more color bands than each of the digital white light color image and the digital fluorescence image, and thus includes more spectral information. The additional spectral information is a result of the first and second spectra being different.

[0009] The input merging of the first and second sets corresponds to the multispectral image of the object. By using the enhanced spectral resolution obtained by co-processing a digital white light color image and a digital fluorescence image as the multispectral image, individual color conversions can be applied to different tissue types.

[0010] The digital white light color image typically records the anatomical structure of the surgical scene by recording reflectance. The fluorescence color image can record the fluorescence of one or more phosphors, or, if no phosphor is used, may be used to record a reflected image of the anatomical structure. By using a color camera for both the fluorescence image and the white light image, a much more accurate color natural color output image can be provided.

[0011] The image processor may be configured to perform any of the above processing steps. The terms "image processor" and "data processing device" may be used synonymously.

[0012] The above solution means may be further improved by the following features, which may be added or combined independently of each other, and each feature has its own advantageous technical effect. Each of the following features may be used to improve one of the above methods and / or may be used to improve one of the above devices, regardless of whether a particular feature is mentioned in relation to the method only or in relation to the device only.

[0013] For example, a computer-implemented image processing method may include the step of demosaicking at least one of a digital white light color image and a digital fluorescence color image. By demosaicking, color artifacts may be avoided. Preferably, the demosaicking is performed before any color conversion function is applied.

[0014] The computer-implemented image processing method can further include a step of normalizing at least one of a digital white light color image and a digital fluorescence color image. Normalization facilitates the joint processing of the digital white light color image and the digital fluorescence color image.

[0015] According to one aspect, the computer-implemented image processing method can include a step of aligning a digital white light color image and a digital fluorescence color image. By this alignment, image features existing in both the digital white light color image and the digital fluorescence color image are represented in the same size and orientation in each image. After alignment, the digital white light color image and the digital fluorescence color image match each other. Thus, pixels at the same position in the two registered images are corresponding pixels.

[0016] In particular, the digital white light color image and the digital fluorescence color image may already be aligned in relation to each other when they are acquired by an image processor. Alternatively, the image processor may be configured to align the digital white light color image and the digital fluorescence color image in relation to each other.

[0017] Alignment should preferably be performed after demosaicing and before applying a color conversion function. In the aligned digital white light image and fluorescence image, a first pixel including a first set of color space coordinates and a second pixel including a second set of color space coordinates are corresponding pixels, and / or an output pixel and the first and second pixels are corresponding pixels.

[0018] In one embodiment, a fluorescence color camera can record a reflectance color image under the same illumination as a digital white light color image. Such a fluorescence color image can complement the reflected image recorded by the white light color camera. Thus, the fluorescence color camera is not necessarily limited to recording only fluorescence images. However, the fluorescence camera may be restricted to recording only images in a second imaging spectrum, for example, by changing one or more filters, if the hardware is not changed.

[0019] According to another advantageous embodiment of a computer-implemented image processing method and / or an image processor, the first imaging spectrum and the second imaging spectrum are complementary to each other, i.e., they do not overlap. In such a configuration, crosstalk between fluorescence and white light is avoided, while maintaining the fluorescence of at least one phosphor on the one hand and an accurate representation of the reflectance of the object on the other hand. Furthermore, by giving the digital fluorescence color image and the digital white light color image spectra that are complementary to each other respectively, there is no redundant special information, which facilitates the joint processing of the input merging of the two images to form a multi-spectral image.

[0020] The digital white light color image may be represented using at least three first color bands in a first color space. The digital fluorescence color image may be represented in a second color space including at least three second color bands. The first and second color spaces are preferably the same, whereby the first and second color bands are also the same. However, this does not necessarily have to be the case. The first and second color spaces may also be different.

[0021] The digital output color image may be generated in a third color space including at least three third color bands. The third color space may be the same color space as the first and / or second color spaces, or it may be a different color space.

[0022] Any of the first, second, and / or third color spaces may be an RGB color space including three color bands R, G, B, or any other color space such as HSV, LAB, CIELAB, XYZ, CMYK, multi-spectral color space or hyper-spectral color space, i.e., a color space using four or more color bands. The term "color" is used to specify coordinates in a color space, i.e., a set of color space coordinates in the color bands of a color space, where the color bands do not necessarily represent saturation, but can also represent other aspects of color such as saturation, brightness, or lightness as in the HSV, LUV, or LAB color spaces. In this regard, the term "color band" is used synonymously with color space dimension in this text.

[0023] According to aspects of a computer-implemented image processing method and / or an image processor, the first color band and the second color band are mapped onto a third color band of a digital output color image using a color conversion function. The color conversion function may be non-linear, but is preferably a linear conversion. The color conversion function can include a color conversion matrix or other types of linear functions. A color conversion matrix may be applied to the first color band and the second color band to map the first and second color bands onto the third color band. Preferably, each color band of the first (set of) color bands and each color band of the second (set of) color bands are input into the linear conversion as separate color bands. In particular, each color band of the first color band and each color band of the second color band may be input into the linear conversion simultaneously.

[0024] The color conversion function may be stored in the memory of an image processor or a medical fluorescence observation device. As will be further described below, two or more color conversion functions may be provided.

[0025] The color conversion matrix may have a dimension of numerical value X in one direction and a dimension of numerical value Y in another direction. The numerical value X may be the sum of the amount of the first color band, i.e., the number of color bands in the first set of color bands, and the amount of the second color band, i.e., the number of color bands in the second set of color bands. The numerical value Y may be the amount of color bands in the third color space. In other words, one dimension of the color conversion matrix may correspond to the sum of the dimensions of the color spaces of the digital white light color image and the digital fluorescence image. The other dimension of the color conversion matrix may correspond to the dimension of the color space of the digital output color image. For example, when the first, second, and third color spaces are each an RGB color space, the color conversion matrix has dimensions of 6×3 or 3×6 or 3×3.

[0026] Therefore, the third color band, i.e., the color band of the digital output color image, may result from a linear combination, particularly an addition, of the color bands of the digital white light color image and the digital fluorescence color image. Such a linear combination or addition can be regarded as a color conversion function because the two colors of the pixels in the digital white light color image and the digital fluorescence color image are mapped to the color of the pixels in the output image.

[0027] According to one aspect, a digital fluorescence color image and a digital white light color image are jointly processed as a multi-spectral image whose color bands are formed or constituted by complementary color bands of the digital fluorescence color image and the digital white light color image. This leads to improved color accuracy, especially when the fluorescence image is represented by reflected white light recorded in a second imaging spectrum and can thus complement the white light information in the white light color image. By using this procedure, the white light information is included in the color bands of the digital fluorescence color image and will thus be provided with a finer color granularity than the color granularity that the digital white light color image exactly provides. When the digital fluorescence color image is represented by fluorescence emission, processing the digital white light color image and the digital fluorescence color image as a multi-spectral image results in a faithful representation of the object under both reflected white light and fluorescence. In particular, when the digital fluorescence color image represents a reflected image of the object under the same illumination as the digital white light color image, additional spectral information is available.

[0028] According to another advantageous embodiment, the first imaging spectrum can include a first sub-band in a color band of a color space, and the second imaging spectrum can include a second sub-band in this color band, where the first sub-band and the second sub-band are preferably complementary to each other. The color space can be the color space of the digital white light color image, the color space of the digital fluorescence color image, or preferably the color space of the digital output color image. It is preferred that the sub-bands of one color band together complete each respective color band. This ensures that no useful spectral information is lost.

[0029] In two or more color bands of each respective color space, the first and second sub-bands may be present. The sub-bands in a color band may further be divided into a plurality of different wavelength bands that are distinct from each other.

[0030] Preferably, the second imaging spectrum includes IR wavelengths. This enables the capture of fluorescence emission in the near-infrared region and can add additional information regarding anatomical structures when digital fluorescence color images are not used for fluorescence capture. Further, the IR wavelengths can include spectral information that enables better discrimination between specific tissue types and their assigned specific color conversion functions.

[0031] The image processor may be configured to generate an output pixel by applying a color conversion function to both the first pixel and the second pixel simultaneously.

[0032] According to one embodiment, the image processor can include at least two different color conversion functions. The image processor may be configured to select one of at least two different color conversion functions depending on the input merge, or in this method, one of at least two different color conversion functions may be selected depending on the input merge.

[0033] The different color conversion functions may be used for different tissue types, where the tissue types include a predefined range of colors. The predefined range of colors can correspond to one or more regions in a color space, where each region is characterized by a set of color space coordinates. The color range may be determined for each tissue type in a calibration process.

[0034] To assign different color conversion functions to different tissue types, the image processor can include at least two sets of different target merges. Each of the at least two different color conversion functions can be assigned to a different one of the at least two sets of different target merges. The different target merges of one set correspond to the colors to be converted using the color conversion function assigned to that set. Each set of target merges can represent, for example, different tissue types, and each target merge within the set can correspond to the color assigned to this tissue type during calibration.

[0035] In one embodiment, the image processor may be configured to compare an input merge with target merges within a set of different target merges to determine whether the input merge is included in a predefined set of target merges.

[0036] In one embodiment, the image processor may be configured to select a color conversion function from at least two different sets of color conversion functions and to be assigned depending on the merging of a first set of color space coordinates of a first pixel and a second set of color space coordinates of a second pixel.

[0037] The processor may be configured to select a color conversion assigned to a set of target merges that includes the input merge. This color function may then be applied to the input merge.

[0038] In one embodiment, the image processor is the following, namely, - a predefined set of target merges in which at least one target merge corresponds to a color representing oxygenated blood; (such a set may represent a tissue containing oxygenated blood, such as arterial tissue); - a predefined set of target merges in which at least one target merge corresponds to a color representing non-oxygenated blood; (such a set may represent a tissue containing non-oxygenated blood, such as venous tissue) - a predefined set of target merges in which at least one target merge corresponds to a color representing living gray matter; - a predefined set of target merges in which at least one target merge corresponds to a color representing living white matter; and includes at least one of the above.

[0039] According to one embodiment, the image processor may be configured to be selectively operated in one of a first operation mode and a second operation mode. The first operation mode includes an image combining step, and the second operation mode includes a color conversion step and an image combining step. Here, the image processor is configured to generate a digital output color image by combining a digital white light color image and a digital fluorescence color image in the image combining step. Here, the image processor is configured to apply at least one color conversion function from a group including a first color conversion function applied to the digital white light color image and a second color conversion function applied to the digital fluorescence color image in the color conversion step.

[0040] A computer-implemented image processing method may be configured to be selectively executed in one of a first operation mode and a second operation mode. Here, in the first operation mode, an image combining step is executed, and in the second operation mode, a color conversion step and an image combining step are executed. Here, in the image combining step, a digital output color image is generated by combining a digital white light color image and a digital fluorescence color image. In the color conversion step, at least one color conversion function is applied from a group including a first color conversion function applied to the digital white light color image and a second color conversion function applied to the digital fluorescence color image.

[0041] By providing two selectable operation modes, it becomes possible for both to provide a digital output color image in the first operation mode corresponding to what is seen through the eyepiece of the medical observation device.

[0042] By providing two color conversion functions that act separately on a digital white light color image and a digital fluorescence color image respectively, it becomes possible to decouple the color conversion of the digital white light color image from the color conversion applied to the digital fluorescence color image. This allows each of these two images to be optimized independently before they are combined for the formation of the digital output color image for visual recognition. According to one aspect, the first and / or second color conversion function is configured to shift the (recorded) color to a (different) color within the color space in each of the digital white light color image and the digital fluorescence color image. This shift is predetermined by the color conversion function in relation to at least one of the amount and direction of the shift in the color space.

[0043] For example, the natural color of blood containing oxygen or blood not containing oxygen may be captured only incompletely in the first and second imaging spectra respectively. This may particularly apply when the reflection spectrum of the blood extends to wavelengths that are not recorded in the first or second imaging spectrum respectively. Therefore, when blood is recorded in a digital white light color image or a digital fluorescence color image, its color does not appear natural, making it more difficult for a user or even an automatic image processing device to recognize blood vessels. By using the first and / or second color conversion function, the (recorded) color, here the color of the blood, may be shifted to a more natural color in each of the digital white light color image or the fluorescence color image.

[0044] In another example, the recorded colors of oxygenated blood and deoxygenated blood may each be shifted differently by using different color conversion functions. Again, both of these colors are incompletely represented in the first and / or second imaging spectra. The recorded color of deoxygenated blood is further shifted towards blue, while the color of oxygenated blood is further shifted towards red. This means that in color space, the two recorded colors may require different amounts and / or directions of shift, and one color may be shifted differently in color space by a color conversion function in terms of at least one of amount and direction relative to another color.

[0045] Of course, the above examples apply to any other type of tissue, such as fluid tissues like lymph, and solid tissues like bone, muscle, and / or nerve tissue.

[0046] The converted color may correspond to a natural color, a pseudo-color such as a neon color, or a "hyper-real" color, in which case at least one of the color appearance parameters, specifically, hue, chroma, saturation, lightness, and luminance, is modified compared to the natural color to enhance visibility and contrast.

[0047] The natural or faithful color in this context corresponds to the color perceived by a CIE standard observer under a standard light source such as CIE illuminant A, B, or C, or other standardized white light illuminations.

[0048] The first and / or second color conversion functions may be configured to convert the colors in their respective digital white light or fluorescent color images into colors not located within their respective first and / or second imaging spectra. Thus, the color conversion functions may be used to correct for imaging defects resulting in a limited first and / or second imaging spectrum.

[0049] In another embodiment, the first and / or second color conversion function may be configured to shift the white point of the digital white light and / or the fluorescent color image to a predetermined position, particularly to another white point. The white point to which the predetermined white point is shifted can particularly correspond to a standard white point defined in a standard light source such as a CIE light source. This enables centering of the color space.

[0050] Furthermore, the first and / or second color conversion function may be configured to perform white balance adjustment or color balance adjustment on the respective digital white light and / or fluorescent color image. In color balance adjustment, the intensity of colors is adjusted so as to properly render at least some colors, particularly intermediate colors. In white balance adjustment, multiple colors are adjusted so that a white object appears white and is not colored.

[0051] According to another aspect, the first and / or second color conversion function may be configured to expand a continuous or fragmented region in the color space into a larger region in the respective digital white light and / or fluorescent color image. Such a color conversion function can better visualize the subtle differences between adjacent colors as the adjacent colors move apart. For example, when the region of the fluorescent color of a specific phosphor is expanded, the nuances of the fluorescence become more prominent. The expansion of the region may of course be accompanied by an appropriate color shift, so the expansion of the region occurs for natural or unnatural colors.

[0052] According to yet another aspect, the first and / or second color conversion function may be configured to shift all the colors in the color space of the respective digital white light and / or fluorescent color image.

[0053] Any of the above-described color conversion functions may be combined into a single color conversion function and / or any of the above-described color conversion functions may be sequentially applied to each of the digital white light and / or fluorescent color image.

[0054] More advantageously, the first and / or second color conversion functions are configured to be applied depending on the color of each pixel of the digital white light color image and / or the digital fluorescence color image. This generally allows different colors to be processed in different ways.

[0055] According to another aspect, a plurality of different first and / or second color conversion functions may be provided, and the processor may be configured to determine the first and / or second color conversion function among the plurality of different first and / or second color conversion functions applied to the pixel depending on the color of the pixel. The plurality of first and / or second color conversion functions may include any of the color conversion functions described above. According to this aspect, the target color dependency correction may be applied to each digital white light and / or fluorescence color image. For example, regions around some colors in a digital white light color image may be expanded to make the color differences more prominent. Another color, such as the color of bone and / or nerve, may be simply shifted to be more natural. Another color, and thus the tissue associated with this color, may be emphasized by being converted to a neon color.

[0056] The computer-implemented image processing method described above may be executed when executing a method for operating a medical fluorescence observation device. The image processor described above may be part of a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope, and in particular, may be part of a fluorescence microscope or a fluorescence endoscope configured for surgery such as neurosurgery, orthopedic surgery, and / or ophthalmic surgery. The fluorescence microscope may be, for example, a laboratory microscope used for biopsy.

[0057] A medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope may include an image processor configured to execute any of the above steps. Further, the medical fluorescence observation device may include a fluorescence color camera configured to record a digital fluorescence color image and a white light color camera configured to record a digital white light color image.

[0058] Since fluorescence may be of relatively low intensity, the fluorescence color camera may operate with a longer integration time.

[0059] A method for operating a medical fluorescence observation device may include recording a digital white light color image using a white light color camera and recording a digital fluorescence color image using a digital fluorescence color camera. The medical fluorescence observation device may include a digital fluorescence color camera and a white light color camera.

[0060] To reduce post-processing, the fluorescence color camera and the white light color camera may have overlapping coaxial and / or preferably the same field of view.

[0061] The medical fluorescence observation device may be a stereoscopic device or a monocular device. In one embodiment, a digital fluorescence color camera and a white light color camera may be provided in each stereoscopic channel. Alternatively, in a medical monocular fluorescence observation device, only a single white light color camera and a single digital fluorescence color camera may be provided.

[0062] In an alternative arrangement of a medical stereoscopic fluorescence observation device, the fluorescence color camera may be provided in one stereoscopic channel, while the white light color camera may be provided in the other stereoscopic channel. In such an arrangement, the fluorescence color camera may be configured to record a (white light) reflected image in a second imaging spectrum. In the case of a (white light) reflected image, this arrangement provides stereoscopic vision, but strictly speaking, this stereoscopic vision is limited to the overlapping portion of the first and second imaging spectra. However, this is hardly noticeable by a human observer. When fluorescence emission is recorded, the fluorescence color camera provides monocular vision based on the fluorescence emission of the object, while the white light color camera provides monocular vision based on the reflectivity of the object.

[0063] The medical fluorescence observation device can include an optical color separation assembly, which is configured to split the incident light into a first imaging spectrum and a second imaging spectrum. The color separation assembly can include optical elements such as a beam splitter, especially a dichroic beam splitter, and / or optical filters such as a fluorescence filter that blocks light other than fluorescence emission and a white light filter that blocks fluorescence emission.

[0064] The medical fluorescence observation device can further include an illumination assembly, which is preferably adjustable by including, for example, a plurality of LEDs or OLEDs of different colors or other light sources that emit light in a plurality of different spectral bands and can be individually turned on and off.

[0065] To facilitate post-processing and ensure that the recorded color space coordinates of the digital white light image and the digital fluorescence color image are comparable to each other, it is preferable that the digital fluorescence color image and the digital white light color image are recorded simultaneously and / or the same exposure time is used and / or the same gain and / or the same white balance adjustment and / or color correction are used. Moreover, the gain of the digital fluorescence color camera and the gain of the digital white light color camera may be maintained at a certain ratio, or may be automatically adapted otherwise. Among these possibilities, it is most preferable that the white light color camera and the fluorescence color camera are synchronized with respect to at least one of the exposure time and the gain. However, the fluorescence color camera may be operated with a longer integration time. This allows compensation for low fluorescence intensity. The white light color camera and / or the fluorescence color camera are preferably CCD or CMOS cameras. The fluorescence camera and the white light color camera are preferably the same.

[0066] According to another aspect, a set of phosphor dependencies of different color conversion functions may be provided, and may be stored, for example, in an image processor or a medical fluorescence device. Each set of phosphor dependencies represents a different combination of the first and second imaging spectra, i.e., a different filter set used in the color separation assembly, which is required when different phosphors or combinations of different phosphors are used. Examples of phosphors used in this context are 5-ALA / pPIX, fluorescein, and ICG. Each of these phosphors requires excitation at a different wavelength and emits fluorescence at a different wavelength. Thus, in a surgical environment where ICG is used, the first imaging spectrum and the second imaging spectrum are different from those in a surgical environment where 5-ALA / pPIX is used and require different color conversion matrices. Thus, the sub-bands of the first and second imaging spectra in the color bands of each color space are different for each case of the phosphor.

[0067] The set of phosphor dependencies may be automatically or manually selected from a plurality of such sets depending on the phosphor used or the setting of the color separation assembly. For example, a medical fluorescence observation device may be configured to automatically detect the configuration or setting of the color separation assembly and select a color conversion matrix depending on this setting. For example, when the optical filter of the color separation assembly for using ICG is replaced for the use of 5-ALA / pPIX, the medical fluorescence observation device can automatically select the color conversion matrix for this 5-ALA / pPIX.

[0068] The subject matter claimed also relates to a computer-readable medium and a computer program including instructions for causing a computer to perform computer-implemented image processing in any of the above-described embodiments.

[0069] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0070] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent descriptions of corresponding methods, where a block or apparatus corresponds to a step or a feature of a step. Similarly, aspects described in the context of a step also represent descriptions of corresponding blocks or items or features of corresponding apparatuses.

[0071] In the following, the present invention will be illustratively described with reference to embodiments and drawings. The combinations of features shown in these embodiments should not be regarded as limiting. For example, features of an embodiment having the above-described technical effects that are not required for a particular application may be omitted. Conversely, if features described above that are not part of the embodiments described below are required for a particular application for the technical effects associated with this particular feature, they may be added.

[0072] Throughout this specification and the drawings, the same reference numerals are used for elements that correspond to each other with respect to function and / or structure.

Brief Description of the Drawings

[0073]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0074] Exemplary embodiments of the present invention will be described with reference first to FIG. 1.

[0075] In FIG. 1, a medical fluorescence observation device 100 is schematically shown. This medical fluorescence observation device 100 may be a microscope or an endoscope. The main difference between a microscope and an endoscope is that in an endoscope (not shown), the object 106 is observed through an optical fiber that brings the object 106 to be inspected close to the vicinity of the object 106 to be inspected, for example, by insertion into the body, while in a microscope, the objective lens 174 is directed towards the object. The medical fluorescence observation device in FIG. 1 is a microscope, but the following description also applies to an endoscope. The medical fluorescence observation device 100 may be a medical fluorescence observation device used in surgery. The medical fluorescence observation device 100 may also be a medical fluorescence observation device such as a laboratory microscope used in a laboratory. The object 106 to be inspected may consist of or include a biological tissue 107.

[0076] The object 106 may include one or more phosphors 116, 118. At least one phosphor may be a phosphor naturally contained in the object. For example, bone and blood contain phosphors. At least one phosphor may be added to the object 106, for example, by injecting it into the biological tissue 107. Examples of phosphors that may be added to the object 106 are ICG, fluorescein, and / or 5-ALA. 5-ALA is synthesized into pPIX / pPIX intracellularly.

[0077] The medical fluorescence observation device 100 is a fluorescence device. This means that the medical fluorescence observation device is configured to observe, record, and preferably also excite the fluorescence of one or more phosphors 116, 118.

[0078] The medical fluorescence observation device 100 may be a stereoscopic device as exemplified in FIG. 1. Thus, this device can include two identical sub-assemblies 101L and 101R for each of the two stereoscopic channels. Since the two sub-assemblies 101L, 101R are identical in function and structure, the following description focuses on the right sub-assembly 101R, but is equally applicable to the left stereoscopic channel 101L.

[0079] Alternatively, the medical fluorescence observation device 100 may be a monocular device. In this case, only one of the two sub-assemblies 101L, 101R may be present. Therefore, the following description is equally applicable to the medical monocular fluorescence observation device 100.

[0080] The medical fluorescence observation device 100 may be used to generate at least one digital white light color image 114, which represents a reflected image of the object 106 over the entire visible light range. This visible light range or visible spectrum includes wavelengths from about 310 nm to about 1100 nm, or from about 380 nm to about 750 nm, or from about 450 nm to about 700 nm. When a fluorescence spectrum or multiple phosphors are used, the fluorescence spectra of the phosphors 116, 118 are preferably omitted from the spectrum recorded in the digital white light color image 114. This ensures that only the reflected light is included in the digital white light color image 114 when fluorescence is present. For example, when 5-ALA / pPIX is used as the phosphor, the fluorescence spectrum from about 625 nm to about 650 nm may not be recorded in the digital white light color image 114.

[0081] When light of a specific wavelength is used to excite fluorescence, spectra containing or consisting of these wavelengths may not be recorded or represented in the digital white light color image 114. For example, when 5-ALA / pPIX is used as a phosphor, fluorescence may be excited by irradiating the object 106 with wavelengths between approximately 380 nm and approximately 450 nm. For fluorescein, ICG, and other phosphors, although different from 5-ALA / pPIX in terms of excitation and emission spectra, they are applied within known ranges.

[0082] By not recording the fluorescence excitation spectrum and the fluorescence emission spectrum, the digital white light color image 114 represents the reflectance of the object 106, that is, a white light image of the object 106 that would be seen by a human observer. Therefore, the digital white light color image 114 may be regarded as a true color image.

[0083] The digital imaging system 102 may further be used to generate a digital fluorescence color image 112 of the object 106. The digital fluorescence color image 112 represents the fluorescence emission of one or more phosphors 116, 118. Therefore, preferably, the digital fluorescence color image 112 does not record wavelengths outside the emission spectrum or the emission spectra of one or more phosphors.

[0084] Both the digital white light color image 114 and the digital fluorescence color image 112 are color images. These images are recorded using at least three color bands, i.e., the primary colors of the color space. For example, the digital white light color image 114 and the digital fluorescence color image 112 may be recorded in the RGB color space using three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescence color image 112 may be recorded in different color spaces respectively, and / or may be represented as multi-spectral or hyperspectral color images. The digital white light color image 114 and the digital fluorescence color image 112 do not have to be recorded in the same color space, but this is preferred.

[0085] The digital white light color image 114 and the digital fluorescence image 112 include pixels 150. In a color space such as the RGB color space, each color of the pixel 150 is represented by a triplet of three integer values, where each integer value indicates the intensity of one of the primary colors R, G, B. For example, the highest intensity red may be indicated by the triplet {255, 0, 0}. The highest intensity green may be indicated by {0, 255, 0}, and the highest intensity blue may be indicated by {0, 0, 255}. Thus, the RGB color space is a three-dimensional space, and the CMYK color space will be a four-dimensional space. Color can be regarded as a point within the color space having color space coordinates such as {0, 0, 255}. A multi-spectral or hyperspectral color space having n color bands will consequently be an n-dimensional color space accordingly, and each color will be represented by n sets of values.

[0086] The spectrum recorded and represented in the digital white light color image 114, i.e., the first imaging spectrum, and the spectrum recorded in the digital fluorescence color image 112, i.e., the second imaging spectrum, are preferably complementary to each other, i.e., they do not overlap except for unavoidable filter losses. Preferably, both of them represent the complete visible light spectrum.

[0087] More specifically, the medical observation device 100 can include a digital imaging system 102 for generating a digital fluorescence color image 112 and a digital white light color image 114. This digital imaging system 102 can include a white light color camera 110 and a fluorescence color camera 111.

[0088] The white light color camera 110 is configured to record a digital white light color image 114. In particular, the white light color camera 110 may be configured to generate a stream of digital white light color images 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images across the entire visible spectrum of the wavelengths shown above. The white light color camera 110 may be a CCD, CMOS, or multi - spectrum or hyperspectrum camera.

[0089] The fluorescence color camera 111 is configured to record a digital fluorescence image 112. In particular, the fluorescence camera 111 may be configured to generate a stream of digital fluorescence color images 112 in the form of a digital video stream. The fluorescence color camera 111 may be configured to record the digital fluorescence color image 112 only in the fluorescence spectra of at least one phosphor 116, 118. The fluorescence camera 111 may be configured to record digital fluorescence images only in one or more narrow optical bands. These narrow bands should overlap with the fluorescence spectra of one or more phosphors 116, 118 where fluorescence is recorded. Preferably, the fluorescence spectra of the phosphor 116 and the second phosphor 118 are at least partially, preferably completely separated, whereby the fluorescence camera 111 can record light in two separate fluorescence bands spaced apart from each other.

[0090] The fluorescence color camera 111 may be a CCD, CMOS, or multi - spectral or hyperspectral camera. The white - light color camera 110 and the fluorescence color camera 111 are preferably of the same type, but this is not necessarily required. Since fluorescence usually has a very low intensity, the fluorescence color camera 111 can have a longer integration time.

[0091] The respective fields of view 184 of the cameras 110, 111 are preferably aligned or even made to coincide coaxially. Thus, the cameras 110, 111 preferably provide the same field of view 184 at the same viewing point and focal length. This results in the same representation of the object 106 in the images 112, 114 generated by the different cameras 110, 111. Both cameras 110, 111 can use the same objective lens 174.

[0092] If the coincidence of the viewing point and the field of view cannot be generated optically, it may be generated by image processing by applying a coincidence or alignment routine to the digital images 112, 114, as will be further explained below. Alignment can also be performed when the cameras 110, 111 have the same viewing point and field of view.

[0093] Preferably, the two cameras 110, 111 are operated synchronously. In particular, the exposure times may be synchronized. Thus, the medical fluorescence observation device 100 may be configured to generate a digital white - light color image 114 and a digital fluorescence image 112 simultaneously.

[0094] Preferably, the gains of the two cameras 110, 111 are synchronized, i.e., the two cameras 110, 111 are adjusted simultaneously. Moreover, the ratio of the gain applied in camera 110 to the gain applied in camera 111 may be constant even when the gain is changed. Gamma correction and color adjustment or white balance may be switched off or maintained constant.

[0095] Due to any of the above features, the comparison, joint processing and / or combination of the two images 112, 114 is facilitated.

[0096] In order to separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescence color image 112, i.e., to separate the reflection spectrum from the fluorescence spectrum, the medical observation device 100 can include an optical color separation assembly 176. This color separation assembly 176 can include optical elements such as a beam splitter 192 which may be dichroic. The color separation assembly 176 can further or alternatively include an optical white light filter 188 and / or an optical fluorescence filter 190.

[0097] The fluorescence filter 190 is preferably configured to transmit light in the fluorescence spectra of one or more phosphors 116, 118 and block light outside the fluorescence spectra.

[0098] The fluorescence filter 190 can be configured as a band-pass filter including one or more pass bands. Each pass band should overlap with the fluorescence emission spectra of the respective phosphors 116, 118 for which fluorescence is to be recorded. Since the fluorescence filter 190 is in the optical path between the beam splitter 192 and the fluorescence camera 111, only the wavelengths of the pass bands of the fluorescence filter 190 are transmitted to the fluorescence camera 111.

[0099] The white light filter 188 is preferably configured to block light within the fluorescence spectra of one or more phosphors 116, 118. The white light filter 188 may also be configured to block light within the fluorescence excitation spectra.

[0100] The white light filter 188 is preferably configured as a band-blocking filter whose stop band corresponds to or at least includes the pass band of the fluorescence filter 190. The white light filter 188 is located in the optical path between the beam splitter 192 and the white light camera 110. Therefore, the white light camera 110 records only wavelengths outside the blocking band of the white light filter 188, and thus also records wavelengths outside the pass band of the fluorescence filter 190.

[0101] Either the white light filter 188 or the fluorescence filter 190 may be an adjustable filter.

[0102] When the beam splitter 192 is a dichroic beam splitter, at least one of the filters 188, 190 may be omitted because in this case the optical spectrum filtering is already integrated in the dichroic beam splitter. Then, the above description of the pass band and the blocking band should also apply mutatis mutandis to the dichroic beam splitter 192.

[0103] Therefore, the white light color camera 110 records a digital white light color image 114 in a reflection spectrum, which is a first imaging spectrum different from the second imaging spectrum recorded as a fluorescence spectrum by the fluorescence camera. The wavelengths included in the first and second imaging spectra are determined by the filter settings of the color separation assembly 176.

[0104] The medical fluorescence observation device 100 can further include an illumination assembly 178, which is preferably configured to illuminate the object 106 through an objective lens 174 through which the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 may be configured to selectively generate white light, i.e., light that is uniformly distributed across the entire visible spectrum, and fluorescence excitation light that includes only light of a wavelength that excites the fluorescence of at least one phosphor 116, 118. The illumination light generated by the illumination assembly 178 may be supplied to the objective lens 174 using an illumination beam splitter 180.

[0105] The illumination filter 179 may be provided depending on the phosphor and the excitation spectrum specific to its fluorescence. For example, when 5-ALA / pPIX is used as the phosphor, the illumination filter has a transmittance of 90% to 98% up to a wavelength of 425 nm, a transmittance of between 0.5% and 0.7% between wavelengths of 450 nm and 460 nm, a transmittance not exceeding 0.1% between wavelengths of 460 nm and 535 nm, and a transmittance that is substantially zero when the wavelength exceeds 535 nm.

[0106] Instead of or in addition to the illumination filter 179, the illumination assembly 178 can include an adjustable light source that includes, for example, a number of LEDs or OLEDs of different colors.

[0107] The medical fluorescence observation device 100 can further include an image processor 170. This image processor 170 can be a hardware module such as a microprocessor or a software module. The image processor 170 may be a combination of both a hardware module and a software module, for example, by using a software module configured to be executed on a specific processor such as a vector processor, a floating-point graphics processor, a parallel processor, and / or a plurality of processors. The image processor 170 may be part of a general-purpose computer 186 such as a PC.

[0108] The image processor 170 is configured to acquire a digital white light color image 114 and a digital fluorescence image 112. For example, the image processor 170 may be configured to acquire the digital white light color image 114 and the digital fluorescence image 112 from the memory 194 and / or directly from the cameras 110, 111. The memory 194 may be part of the image processor 170 or may be resident elsewhere in the medical fluorescence observation device 100.

[0109] The image processor 170 is further configured to calculate a digital output color image 160 from the digital white light color image 114 and the digital fluorescence image 112. The digital output color image 160 is a color image represented in a color space. The color space of the digital output color image may be different from the color space of either the digital white light color image 114 or the digital fluorescence image 112. However, preferably, the color space of the digital output color image 160 is the same as the color space of the digital white light color image 114 and the digital fluorescence image 112.

[0110] The image processor 170 includes at least two operation modes configured to operate selectively. For example, a user (not shown) can manually select which of the two operation modes to execute. A selector device 165 may be provided to select an operation mode from at least two available operation modes. This selector device 165 may be a mechanical device such as a dial or a switch, or may be a button or other element on a graphical user interface displayed on a display such as the displays 132, 182, which will be described in more detail below.

[0111] As shown in FIG. 1, the image processor 170 can include a first operation mode A, a second operation mode B, and optionally a third operation mode C. Alternatively, the image processor 170 may be configured to operate in only a single operation mode, for example, one of modes A, B, and C.

[0112] In the first operation mode A, the image processor 170 is preferably configured to generate a digital output color image 160 that represents what would be seen through the eyepiece 104. To achieve this, in operation mode A, the image processor 170 may be configured to combine the digital fluorescence color image 112 and the digital white light color image 114 to generate the digital output color image 160. In particular, the image processor 170 may be configured to additionally combine or add the values of the digital fluorescence color image 112 and the digital white light color image 114 at each corresponding pixel 150 in the digital fluorescence color image 112 and the digital white light color image 114 in each color band of their color spaces.

[0113] In the second operation mode B, the image processing device 170 is first configured to apply the color conversion function 140 to at least one of the digital white light color image 114 and the digital fluorescence color image 112, and then to generate the digital output color image 160 from the digital white light color image 114 and the digital fluorescence color image 112. Thus, in the second operation mode B, at least one of the digital white light color image 114 and the digital fluorescence color image 112 is color-converted before being combined. By switching between the first operation mode A and the second operation mode B, the user can switch between a perspective that faithfully renders what is seen through the eyepiece and a perspective that can emphasize different features of the object 106.

[0114] The color conversion function 140 may be any type of function, such as a one-dimensional or n-dimensional interpolation function. However, preferably, the color conversion function 140 is a linear conversion function. In particular, the color conversion function 140 may be a color conversion matrix 142. One dimension of the color conversion matrix 142 may be the sum of the number of color bands in the digital fluorescence color image 112 and the number of color bands in the digital white light color image 114. Another dimension of the color conversion matrix 142 may correspond to the number of color bands in the digital output color image 160 in the other direction of the matrix.

[0115] In another embodiment, the color conversion matrix 142 may have a first dimension corresponding to the number of color bands depending on the side to which one of the two color conversion matrices is applied in the digital white light color image 114 or the digital fluorescence color image 112, and a second dimension corresponding to the number of color bands in the digital output color image 160.

[0116] When all the images 112, 114 are in the RGB color space, the dimensions of the color conversion matrix 142 are preferably 3×3 matrix or 6×3 or 3×6 matrix when the color conversion matrix is configured to operate simultaneously on the digital white light color image and the digital fluorescence color image.

[0117] The second operation mode B itself may include different sub-modes. Each sub-mode may apply different color conversion functions 140 to at least one of the digital white light color image 114 and the digital fluorescence color image 112 before the digital fluorescence color image 112 is combined with the digital white light color image 114. In FIG. 1, as an example, three sub-modes B-I, B-II, B-III are shown. Of course, more or fewer than three sub-modes may be provided. The different color conversion functions 140 may be stored in the memory 194.

[0118] For example, the first color conversion function 140a may be applied only to the digital white light color image 114, and the second color conversion function 140b may be applied only to the digital fluorescence color image 112. The first and second color conversion functions 140a, 140b may be applied independently of each other to separate the color conversion of the digital white light color image 114 from the color conversion of the digital fluorescence color image 112. For example, only one of the first and second color conversion functions 140a, 140b may be applied to their respective images 112, 114, or both may be applied to their respective images.

[0119] In one embodiment, the user may be selectable between different first color conversion functions 140a and / or different second color conversion functions 140b.

[0120] For example, in operation mode B-I, a first color conversion function 140a configured to convert the colors in the digital white light color image 114 to colors not located within the second imaging spectrum may be used. In this case, the natural color of the object 106, or a part of the object that was partially filtered and thus not properly recorded in the digital white light color image 114 due to distortion by the color separation assembly 176, can be converted by the first color conversion function 140a to a corresponding color closer to the natural color.

[0121] In another additional or alternative example, the first color conversion function 140a may be used in operation mode B-I, or in an additional optional operation mode that converts the colors in the digital white light color image 114 to pseudo colors or different hues. This can highlight the colors associated with a particular type of tissue. For example, the colors of nerves, arterial blood, and / or venous blood may all be converted to different pseudo colors.

[0122] By using the selector device 165, a user can quickly perform different assignments of pseudo-colors to different types of tissues. In one operation mode, the color of arterial blood may be converted to neon red, in another operation mode, the color of venous blood may be converted to neon blue, and in yet another operation mode, the color of nerve tissue may be converted to neon yellow. Other operation modes can use combinations of these modes. When the first color conversion function 140a assigns different hues instead of pseudo-colors, contrast may be enhanced. For example, the first color conversion function 140a can spread the red color in the digital white light color image 114 over a wider range, and thus make smaller changes in blood oxygenation more noticeable to the user.

[0123] In yet another additional or alternative example, the first color conversion function may be used, for example, in operation mode B-I or an additional operation mode, for color balance adjustment and / or white balance adjustment of the digital white light color image 114. In white balance adjustment, the overall mixture of colors is changed, but intermediate colors, i.e., gray, black, and white, are maintained as intermediate colors. In color balance adjustment, colors including intermediate colors are adjusted to correspond to the intermediate colors.

[0124] In yet another additional or alternative example, the first color conversion function 140a may be used, for example, in operation mode B-I or an additional operation mode, to shift the white point of the digital white light color image 114 to a predetermined position within the color space. Adjusting the white point allows for adjustments for different illuminations and filter settings of the color separation assembly 176.

[0125] According to one example, at least one of the above-described first color conversion functions 140a may be applied to the digital white light color image 114. According to another example, at least two of the above-described first color conversion functions 140a may be sequentially applied to the digital white light color image 114. Alternatively or additionally, two or more of the above-described color conversion functions 140a may be combined into a single color conversion function 140a.

[0126] The descriptions of the various first color conversion functions 140a above are also applicable mutatis mutandis to the various second color conversion functions 140b. The only difference between these two functions is that the second color conversion 140b operates on the digital fluorescence image 112 and is configured to operate on different colors.

[0127] In operation mode B-II, for example, one or more color conversion functions 140b may be applied only to the digital fluorescence image, and the color conversion function 140a is not applied.

[0128] In operation mode B-III, for example, one or more color conversion functions 140a may be applied to the digital white light color image, and one or more color conversion functions 140b may be applied to the digital fluorescence color image 112.

[0129] In an optional third operation mode C, it may be assumed that a third color conversion function 140c is applied simultaneously to the digital white light color image 114 and the digital fluorescence color image 112, where the application of the color conversion function 140c results directly in the digital output color image 160.

[0130] If another type of phosphor having different excitation wavelengths and / or fluorescence wavelengths is used, different color conversion functions 140a, 140b, 140c need to be provided. This is because it is necessary to adapt the combination of filter settings in the color separation assembly 176, and thus the first and second imaging spectra, to the excitation wavelength and / or fluorescence wavelength.

[0131] Thus, in one embodiment, the medical observation device 100 or the processor 170 may be configured to store a plurality of different sets 143 of the color conversion functions 140a, 140b, and / or 140c. Each different set 143 includes one or more of the color conversion functions 140a, 140b, and / or 140c described above. Each different set 143 corresponds to the use of a different phosphor and thus represents different filter settings of the color separation assembly 176. For example, the first set 143a may be used for ICG as a phosphor, while the second set 143b may be used for 5-ALA / pPiX as a phosphor. Of course, additional sets may be adapted for use with additional phosphors or phosphor combinations.

[0132] The medical fluorescence observation device 100 may be adjusted for different phosphors by reconfiguring the color separation assembly 176, for example, by replacing optical elements such as the filters 190 and / or 188 or the dichroic beam splitter 180. Selection of the appropriate set 143 of the color conversion functions 140 may be performed automatically, for example, if the medical observation device 100 is configured to automatically detect the settings of the color separation assembly 176.

[0133] Alternatively or additionally, the medical observation device 100 may include a filter setting selector device 168 that allows a user to manually select the set 143 of the color conversion functions 140 to be applied to the digital white light color image 114 and / or the digital fluorescence color image. The filter setting selector device 168 may be part of a mechanical and / or graphical user interface. For example, if the filter setting selector device 168 is operated to be in position a, the set 143a is selected. At position b, the set 143b is selected, and so on.

[0134] The digital output color image 160 may be displayed on a display 132 integrated with the medical fluorescence observation device 100. For example, the display 132 may be integrated with the eyepiece or ocular lens 104 of the medical fluorescence observation device 100. The display 132 can also display a graphical user interface for operating the medical observation device 100.

[0135] The medical fluorescence observation device 100 can include a direct optical path 134 from the object 106 through the objective lens 174 to the ocular lens 104. In such a case, the display may be a translucent display 132 positioned within the direct optical path 134, or may be projected into the direct optical path 134. A beam splitter 136 may be provided for splitting light between the optical ocular lens 104 and the digital imaging system 102.

[0136] Alternatively, the medical fluorescence observation device 100 does not have a direct optical path 134, but can only display an image from the integrated display 132. As a further alternative, the medical fluorescence observation device may not have any display at all.

[0137] The medical fluorescence observation device 100 can include an output interface 172 to which one or more (external) displays 182 may be connected. For this purpose, the output interface 172 can include standardized connectors and data transmission protocols such as USB, HDMI, DVI, DisplayPort, Bluetooth, etc. The external display may be a monitor, 3D goggles, an eyepiece, etc. Any combination of external displays may be connected to the output interface 172. Any of the displays 182 can display a graphical user interface for operating the medical observation device 100.

[0138] The computer 186 or the image processor 170 is connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission lines may be wired, wireless, or partially wired and partially wireless. The computer 186 and / or the image processor 170 is not actually integrated into the medical fluorescence observation device 100, but may be located physically separated from the digital imaging system 102. For this purpose, the digital imaging system 102 and the computer 186 and / or the image processor 170 may be connected to a network such as a LAN, WLAN, or WAN to which at least one display 182 is also connected.

[0139] According to a variant, the medical fluorescence observation device 100 may be a stereoscope, but may include only two cameras for each stereoscopic channel. In one stereoscopic channel, a fluorescence color camera 111 is used and is configured to selectively record the white light reflectance as well, while in the other stereoscopic channel, a white light color camera 110 is used. In such an arrangement, when fluorescence is not used, a stereoscopic white light color image is provided, and when fluorescence is used, a monocular white light color image and a monocular fluorescence color image are provided. The above description and the following description are equally applicable to this configuration.

[0140] FIG. 2 shows an example of the first operation mode A.

[0141] Reference numeral 200 is, respectively, a quantitative example of the first imaging spectrum 202 recorded by the digital white light camera 110 and / or represented in the digital white light color image 114. The intensity I over the wavelength / color λ is shown normalized. The first imaging spectrum 202 preferably extends over at least the visible spectrum 212.

[0142] As a mere example, the color space in which the first imaging spectrum 202 is recorded is an RGB color space having three primary colors or color bands 204, 206, and 208. One primary color 204 is blue, another primary color 206 is green, and the third primary color 208 is red. The sensitivities of the sensors of the white light color camera 110 in the different primary colors 204, 206, 208 are adjusted so that the sensitivity across the entire visible spectrum 212 is as constant as possible.

[0143] If a color space other than RGB is used, the number, position, and / or width of the color bands may be different. Otherwise, there is no fundamental difference from the RGB color space.

[0144] The first imaging spectrum 202 does not include the fluorescence excitation light and fluorescence emission spectrum of at least one phosphor 116, 118. Thus, the first imaging spectrum 202 may include at least one cutoff band 210 where the fluorescence coincides with the fluorescence emission of at least one phosphor recorded by the fluorescence color camera 111. This cutoff band 210 is generated, for example, by the white light filter 188. The number, width, and / or position of the cutoff band 210 depend on the number and type of phosphors observed in the object 106.

[0145] Reference numeral 220 indicates the second imaging spectrum 222 recorded by the fluorescence color camera 111 and / or represented in the digital fluorescence color image 112. As a mere example, the color space in which the second imaging spectrum 222 is recorded is also the RGB color space. The sensitivities of the sensors of the fluorescence color camera 111 in the different primary colors 204, 206, 208 are adjusted so that the sensitivity across the entire visible spectrum 212 is as constant as possible.

[0146] The spectra 202, 222 do not have to be recorded in the same color space, but this is preferred.

[0147] The second imaging spectrum 222 can include one or more passbands 224. The number, position, and / or width of the passbands depend on the number and type of phosphors used. At least one passband 224 preferably corresponds to at least one blocking band 210. At least one passband is generated, for example, by the fluorescence filter 190.

[0148] The first imaging spectrum 202 and the second imaging spectrum 222 are complementary to each other. They preferably complement each other to cover the whole or most of the visible spectrum 212.

[0149] Each passband 224 of the second imaging spectrum 222 preferably overlaps with the fluorescence emission spectra 226, 228 of the phosphors 116, 118 where the fluorescence is recorded and overlaps with one or more primary colors 204, 206, 208 of the color spectrum. For example, the fluorescence emission spectrum 226 of one phosphor 116 can overlap with all three primary colors 204, 206, 208. The sensor of the fluorescence camera 111 that records the color band 204 records only a small portion 230 of the fluorescence emission spectrum 226. The sensor of the fluorescence camera 111 that records the color band 206 records most of the fluorescence emission spectrum 226, but the sensor of the fluorescence camera 111 that records the color band 208 also records only a small portion 236 of the fluorescence emission spectrum 226.

[0150] The fluorescence spectrum 228, if present in a particular case, is recorded only by the fluorescence camera 111 that records the color band 208, in contrast.

[0151] In its simplest form, the combination 240 of the digital white light color image 114 and the digital fluorescence color image 112 is an additive combination. For example, the values in each color band of the digital white light color image 114 and the fluorescence color image 112 are added at each corresponding pixel 150. The corresponding pixel 150 may be located at the same position in both images 112, 114 if the images 112, 114 are aligned and of the same size. If both images 112, 114 are RGB images, the values in the RGB color bands of pixel 150 in the digital white light color image are {R1, G1, B1}, and the values in the RGB color bands of the corresponding pixel 150 in the digital fluorescence color image are {R2, G2, B2}, then the values in the RGB color bands of the corresponding pixel 150 in the digital output color image are {R1 + R2, G1 + G2, B1 + B2}. As a result of this combination 240, a color transformation occurs. This is because the color of the pixels in the output image is different from the color of the pixels in images 112, 114. Therefore, the combination 240 is an example of the color transformation function 140. This combination 240 may be implemented using the color transformation matrix 142.

[0152] Reference numeral 250 indicates a qualitative representation of the spectrum 252 of the digital output image 160. In short, the spectrum 252 of the digital output image 160 is the addition of the first imaging spectrum 202 and the second imaging spectrum 222.

[0153] Alternatively, the individual spectra 202 and 222 are processed as a single composite multi-spectral spectrum having numerical values of color bands corresponding to the sum of the color bands in the first imaging spectrum 202 and the second imaging spectrum 222. In other words, the digital fluorescence color image 112 and the digital white light color image 114 are co-processed as a single (virtual) multi-spectral image. Instead of adding the color space coordinates in each color band of images 112, 114, a combined set is formed and processed while keeping the color bands of images 112, 114 separate.

[0154] This corresponds to operation mode C (Figure 1), in which color conversion functions 140, 140c are applied to both the digital white light color image 114 and the digital fluorescence color image 112 to obtain a digital output color image 160. The spectrum of this digital output color image 160 is indicated by reference numeral 250 in Figure 2.

[0155] Referring to Figure 3, this will be described below. For simplicity, Figure 3 shows the spectrum of Figure 2 in the RGB color space, with only a single passband 224 or stopband 210 shown.

[0156] The digital white light color image 114 recorded in the first imaging spectrum 202 includes signals R1, G1, and B1, each representing the intensity I of its respective color band. Due to the stopband 210, the signal G1 results from two separated wavelength bands. However, the sensor that records G1 cannot distinguish between these two wavelength bands.

[0157] The digital fluorescence color image 112 recorded in the second imaging spectrum 222 includes signals R2, G2, B2 in each color band.

[0158] Since the first and second imaging spectra 202, 222 are complementary to each other, each color band is divided into two signals, each signal arriving from a different camera 110, 111 and representing or equivalent to a sub-band of the color band. The color band 204 is divided into R1 and R2 respectively. The color band 206 is divided into G1 and G2 respectively, and the color band 208 is divided into B1 and B2 respectively. Preferably, there is no overlap between the various signals within the color band.

[0159] Therefore, even when the first and second imaging spectra 202, 222 are recorded in the same color space using the same type of color camera, a multi-spectral image of the object 106 results, which in this case consists of six color bands R1, R2, G1, G2, B1, B2. This is schematically shown by reference numeral 300. It should be noted that this is independent of whether the fluorescence camera actually records fluorescence emission or reflectance.

[0160] In other words, at least one color band 204, 206, 208 of the color space of the at least digital output color image 160 is subdivided into two sub-bands R1, R2, G1, G2, B1, B2, where one of the two sub-bands is included in the digital fluorescence color image 112 and the other of the two sub-bands is included in the digital white light color image 114. Preferably, the two sub-bands within a color band do not overlap. They are preferably complementary. Most preferably, together they at least substantially complete their respective color bands. The sub-bands can include or consist of two separate spectral bands. Each sub-band may be regarded as a color band in itself.

[0161] The color conversion function 140 reflects the subdivision of the color band into sub-bands determined by the blocking band 210 and the pass band 224. Since the width and / or position of the sub-bands determines how much light is collected by the respective cameras 110, 111 in that sub-band, the color conversion function 140 needs to be adjusted for each different filter setting of the color separation assembly 176.

[0162] Therefore, the digital white light image 114 and the digital fluorescence image 112 may be processed together as an image consisting of the sum of the color bands in the two images 112 and 114. This provides improved color resolution.

[0163] For example, the color conversion function 140c may be applied in the form of a linear transformation using the color conversion matrix 142 to generate the digital output color image 160.

[0164] When the digital output color image 160 is represented in the RGB color space, a 6×3 or 3×6 color conversion matrix may be applied to the combined digital fluorescence image 112 and white light image 114 or their component signals R * , G * , B * in order to achieve. The matrix coefficients C11, C12, …, C63 may be determined experimentally.

[0165] In FIG. 3, from the set of color space coordinates {R1, G1, B1} in the color bands R, G, B of the (first) pixel of the digital white light color image 114 and the set of color space coordinates {R2, G2, B2} in the preferably corresponding (second) pixel of the digital fluorescence color image 112, an input merge or, in synonymous terms, an input merge set {R1, R2, G1, G2, B1, B2} is formed. This input merge set corresponds to the pixel of the multispectral image. Next, the color conversion matrix is applied to the input merge. No additional memory is required to form the input merge. The input merge may be logically generated, for example, by combining pointers to the color space coordinates of the first and second pixels. Of course, the color space coordinates of these two pixels may also be copied to the memory that physically stores the input merge.

[0166] Next, the operation mode B will be described.

[0167] FIG. 4 shows a schematic reflection spectrum 400 of the biological tissue of the object 106, where the term tissue includes both fluid tissue and solid tissue. For example, the reflection spectrum 400 may correspond to the reflection spectrum of blood containing oxygen.

[0168] The spectral sensitivities R, G, and B of the R, G, and B sensors of the sample RGB color camera are also shown in FIG. 4.

[0169] In human perception, tissues such as oxygenated blood are perceived as having natural colors, i.e., colors with red, under white light illumination such as a CIE light source. Therefore, although the reflection spectrum 400 spreads widely over the visible light spectrum, it is perceived as a single natural color. In the first imaging spectrum 202 where the digital white light color image 114 is recorded, the wavelength λ in the cutoff band 210 is not recorded. Therefore, the reflection spectrum 400 represented in the digital white light color image 114 does not faithfully represent the natural color of the tissue.

[0170] The same applies to the second imaging spectrum 222 and the digital fluorescence color image 112, where the reflection spectrum 400 also causes distortion.

[0171] FIG. 5 shows a schematic diagram of the CIE1932 color space diagram, where color 500 is shown. Color 500 can represent the natural color perceived by a standard observer when viewing a tissue having the reflection spectrum 400 (FIG. 4) illuminated by, for example, a CIE light source A, B, or C or another standard light source.

[0172] Since the wavelengths within the cutoff band 210 are not recorded, color 500 is represented in the digital white light color image 114 as the (recorded) color 502. To represent color 500 more faithfully, the recorded color 502 is converted to color 500 by the color conversion function 140. This color conversion function 140 may be the first or second color conversion function 140a, 140b. For example, the color conversion may be a shift or translation in the LMS color space or tristimulus coordinates.

[0173] The color conversion function 140 may be determined by color calibration using known colors, such as color cards, known filter settings, and known lighting conditions. The color 500 may be a wavelength that is within the cutoff band 210 and thus does not exist within the first imaging spectrum 202. Along with the shift from 502 to 500, the entire color space may be shifted.

[0174] Although the color 500 into which the color 502 is converted by the color conversion function is not a natural color but a pseudo color or a color that is close to the natural color but visually different, it may be beneficial in some applications. This can offset the type of tissue associated with this color from other tissues or fluorescent colors close to the color 500 and help emphasize this type of tissue to the trained eye. For example, the color 500 in the case of oxygenated blood may be a "hyper-real" color, such as a red that is brighter than the natural color of oxygenated blood, or a pseudo color of red such as neon red. Of course, such a conversion can be similarly performed for any other color or tissue type.

[0175] Due to the cutoff band 210, the position of the natural white point 504 is shifted to the recorded white point 506. Thus, another color conversion function 140, such as the color conversion function 140c, can shift the recorded white point 506 to the natural white point 504 or near it. In this regard, the white point is treated in the same manner as any color.

[0176] The shift of a single color can correspond to a shift of the entire color space, i.e., all colors are shifted by the same amount.

[0177] A shift of the entire color space by the same amount and in the same direction may be used to adjust the white balance or color balance of the digital white light color image 114.

[0178] Alternatively, different colors or sets of colors may be transformed or shifted to be different in relation to at least one of the amount and direction of the shift. For example, the recorded color 508 of blood without oxygen may be shifted, for example, to a color 510 in a further blue range. Such a shift from color 508 to color 510 can be made different from the shift from color 502 to color 500. Thus, instead of transforming the entire color space, the color conversion function 140 can be configured such that a color-dependent transformation is generated. Different (recorded) colors 502, 504, 508 may have different color conversion functions 140 applied to them.

[0179] Another example of the color conversion function 140, for example 140d, can expand the region 512 within the color space to a wider region 514. Within this region 514, the distance between colors becomes longer than within region 512, so that the color differences become more prominent. The application of such a color conversion function may be color-dependent. In this case, only a predefined color region 512 is expanded. Since this color conversion function 140 can include a color shift, not only is region 514 expanded, but it also moves to different regions of the color space.

[0180] Any of the above color conversion functions 140 can only transform a selected subset of the color appearance parameters of the color. These color appearance parameters are hue, chroma, saturation, lightness, and luminance. For example, when converting color 502 to color 500, only the hue may be changed, or only the saturation and lightness may be changed by the color conversion function 140.

[0181] Any combination of the above color conversion functions 140a may be applied. Although the term "image" has been used in the above description, it should be understood that the color conversion function 140 is applied to the image at the pixel level, that is, to each pixel 150 of the digital white light color image 114. The color of the pixel 150 can determine which color conversion function 140 or which combination of color conversion functions 140 is applied. This can be done automatically by the processor 170.

[0182] FIG. 8 shows an example of color-dependent color conversion. The digital white light color image 114 and the digital fluorescence color image 112 may both be reflected images of the object 106 in this example. However, this is not necessarily the case, and the digital fluorescence image 112 may instead be a fluorescence image or a reflected image containing the fluorescence of the phosphor.

[0183] In the images 112, 114, three different types of tissues 852a, 852b, 852c are shown. Of course, there may be more or fewer types of tissues in the images 112, 114. The various types of tissues may not be clearly visible in one or both of the images 112, 114. For example, the tissue 852a may be an arterial blood vessel composed of oxygenated blood, the tissue 852b may be a venous blood vessel composed of non-oxygenated blood, or, if fluorescence is recorded in the digital fluorescence image, it may be a tumor, and 852c may be a background, for example, in a brain surgery, it may be nerve tissue.

[0184] The pixel 150a1 of the digital white light color image 114 is located in the region of the tissue 852a. The color space coordinates of the pixel 150a1 are {R 1,1 , G 1,1 , B 1,1} when the RGB color space is assumed only for the purpose of explanation. The pixel 150a2 of the digital white light color image 114 is located in the region of the tissue 852b. The color space coordinates of the pixel 150a2 are {R 1,2 , G 1,2 , B 1,2}. The pixel 150a3 of the digital white light color image 114 is located in the region of the tissue 852c. The color space coordinates of the pixel 150a3 are {R 1,3 , G 1,3 , B 1,3}. These color space coordinates are recorded in the first imaging spectrum 202.

[0185] In the digital fluorescence image 112, pixel 150b1 is located within tissue 852a and is preferably the corresponding pixel to pixel 150a1. Pixel 150b2 is located within tissue 852b and is preferably the corresponding pixel to pixel 150a2. Pixel 150b3 is located within tissue 852c and is preferably the corresponding pixel to pixel 150a3. These corresponding pixels are preferably located at corresponding positions in images 112, 114, or at corresponding positions of patterns previously identified in images 112, 114 using a pattern recognition algorithm.

[0186] The color space coordinates of pixel 150b1, assuming here also the RGB color space, are {R 2,1 , G 2,1 , B 2,1}. The color space coordinates of pixel 150b2 are {R 2,2 , G 2,2 , B 2,2}. The color space coordinates of pixel 150b3 are {R 2,3 , G 2,3 , B 2,3}. Since these color space coordinates are those recorded in the second imaging spectrum 222, they contain spectral information different from the color space coordinates of the corresponding pixels 150a - 150c in the digital white light color image. If the color spaces of images 112, 114 are different, they may be converted to a common color space.

[0187] As shown in FIG. 3, an input merger (set) 846 of color space coordinates may be formed at each of pixels 150a1 - 150a3 and 150b1 - 150b3. Thus, the color space coordinates of pixels 150a1 and 150b1, {R 1,1 , G 1,1 , B 1,1} and {R 2,1 , G 2,1 , B 2,1} form the input merger {R 1,1 , R 2,1 , G 1,1 , G 2,1 , B 1,1 , B 2,1}. The color space coordinates of pixels 150a2 and 150b2, {R1,2 , G 1,2 , B 1,2} and {R 2,2 , G 2,2 , B 2,2} are the input merges {R 1,2 , R 2,2 , G 1,2 , G 2,2 , B 1,2 , B 2,2} to form. Color space coordinate pixels 150a3 and 150b3, {R 1,3 , G 1,3 , B 1,3} and {R 2,3 , G 2,3 , B 2,3} are the input merges {R 1,3 , R 2,3 , G 1,3 , G 2,3 , B 1,3 , B 2,3} to form. Note that each merge 846 substantially corresponds to the input merge {R1, R2, G1, G2, B1, B2} in FIG. 2. Each different input merge 846 corresponds to a different color.

[0188] A set 143 including at least two, i.e., a plurality of color conversion functions 140, may be provided, for example, stored in an image processor 170. The set 143 can include, for example, different color conversion functions 140-I, 140-II, 140-III shown in FIG. 5. Instead of a set, a single color conversion function 140 may be provided. In one embodiment, these color conversion functions 140-I, 140-II, 140-III can correspond to color conversion functions 140a, 140b, 140c.

[0189] If only a single color conversion function 140 is provided, its application can depend on the input merge 846 in FIG. 8, i.e., the color space coordinates contained therein. For example, if the input merge belongs to a predetermined set 856 of target merges 858 that have been or were assigned to a particular color conversion function 140, then this particular color conversion function 140 is applied to the input merge. Otherwise, the color conversion function 140 is not applied. In this case, the color conversion function 140 may only be applied if the input merge 846 represents a color that is typical for organization 852a but not for organization 852b or organization 852c. This is because the color conversion function 140 has been or was assigned to the color representing organization 852a.

[0190] If two or more color conversion functions 140 are provided, e.g., color conversion functions 140-I, 140-II, 140-III, which of the color conversion functions 140-I, 140-II, 140-III is applied, or whether any color conversion function is applied in all cases, can depend on the input merge 846, i.e., the set of color space coordinates of the input merge. Each set 856 is assigned to a different color conversion function.

[0191] For example, if the input merge 846, or more precisely, the target merge 858 corresponding to the input merge 846, is included within a predefined set 856 of target merges 858, the color conversion function 140-I assigned to this particular set 856 may be applied. If the input merge 846 is included within a second predefined set of target merges 858, preferably not overlapping with the first set, the color conversion function 140-II assigned to this set may be applied. If the input merge 846 is included within a third predefined set 856 of target merges, a third color conversion function 140-III may be applied, or the color conversion function may not be applied. Alternatively, if the input merge 846 is not included within any set 856, none may be applied, or a specific color conversion function may be applied. Thus, tissue 852a can undergo a different color conversion than tissue 852b and / or tissue 852c, as described with reference to colors 500, 508, 504 in FIG. 5, each of which may be represented by a different input merge.

[0192] Each set 856 represents a different group of colors. Each set 856 may in particular be a specific type of tissue indicating the colors included in this set. For example, one set 856 can include a range of brighter reds for representing oxygenated blood, another set 856 can include a range of bluish-reds for representing non-oxygenated blood, another set 856 can include a range of grayish-pinks for representing living gray brain matter, another set 856 can include a range of whitish-pinks for representing living white brain matter, and so on. Any number of sets and combinations may be used. However, the target merge 858 should be included in only one set 856 in order to assign color conversion functions and input merges 846 one-to-one.

[0193] The target merge 858 may be determined empirically in a calibration process.

[0194] At least one set 856 of the target merges 856 may be stored in the image processor 170, for example, as a look-up table, or may be constituted by the image processor 170.

[0195] The input merge {R 1,1 ,R 2,1 ,G 1,1 ,G 2,1 ,B 1,1 ,B 2,1} to which the color conversion function 140-I is assigned and applied results in converting the input merge into the color space coordinates {R * 1,G * 1,B * 1} assigned to the pixel 150c1 in the digital output color image 160. The output pixel 150c1 is preferably the corresponding pixel for at least one of the pixels 150a1 and 150b1. The input merge {R 1,2 ,R 2,2 ,G 1,2 ,G 2,2 ,B 1,2 ,B 2,2} to which the color conversion function 140-II is assigned and applied results in converting the input merge into the color space coordinates {R * 2,G * 2,B * 2} assigned to the pixel 150c2 in the digital output color image 160. The output pixel 150c2 is preferably the corresponding pixel for at least one of the pixels 150a2 and 150b2. The input merge {R 1,3 ,R 2,3 ,G 1,3 ,G 2,3 ,B 1,3 ,B 2,3} to which the color conversion function 140-III is assigned and applied results in converting the input merge into the color space coordinates {R * 3,G * 3,B *It is converted to 3}. The output pixel 150c1 is a corresponding pixel to at least one of the pixels 150a3 and 150b3. If the color conversion function is not assigned to the input merge and as a result, color conversion is not applied to this input merge, then as described with reference to FIG. 2, the color space coordinates within the merge that are in the same color band may be calculated by adding the color space coordinates within the merge.

[0196] Since the input merge includes many more color bands than each of the images 112, 114, the accuracy of the merge-dependent color conversion is consistent with the accuracy of the tissue type detection. In practice, the tissue type-dependent color conversion may be integrated into the color conversion process by accurately calibrating different predetermined sets 856 of the target merge 858 that determine which color conversion function is used.

[0197] FIG. 6 shows a schematic diagram of an imaging method that may be implemented as a computer-implemented method, for example, executed on the image processor 170.

[0198] In optional step 600, for example, using the white light color camera 110, a digital white light color image 114 is recorded. In optional step 602, for example, using the fluorescence camera 111, a digital fluorescence color image 112 is recorded. Steps 600 and 602 are optional. This is because these images 112, 114 may also be acquired from the memory. As described above, these cameras 110, 111 should be synchronized with respect to the exposure time and locked to each other with respect to the gain, that is, maintained at the same gain ratio. The gamma may be set to a fixed value, and preferably all automatic color adjustments are turned off.

[0199] In optional step 604, each of the images 112, 114 may be demosaicked.

[0200] In optional step 606, one or both of the images 112 or 114 may be aligned such that the same image features are represented geometrically identically in each of the images 112, 114 with respect to position, size, and orientation. After alignment, each pixel in the digital white light color image 114 has a corresponding pixel in the digital fluorescence color image 112. Preferably, these corresponding pixels are located at the same position in each of the images 112, 114.

[0201] In one operating mode, for example operating mode B, but optionally also in operating mode C, the color conversion is performed in an optional color conversion step 608. This step 608 is omitted in operating mode A, for example. In step 608, a color conversion function 140 as described above or a combination of color conversion functions 140 is applied to at least one of the digital white light color image 114 and the digital fluorescence image 112.

[0202] In the image combination step 610, the digital white light color image 114 and the digital fluorescence color image 112 are combined for the generation of the digital output color image 160. This can be done in all operating modes A, B, C. As described above, the images 112, 114 may simply be added for combination, as shown in FIG. 2, for example. Such addition may be performed by adding the color space coordinates of two images located within the same color band. If a pixel in the digital white light color image 114 has color space coordinates {R1, G1, B1} in a certain color space and the corresponding pixel in the digital fluorescence color image 112 has color space coordinates {R2, G2, B2}, the color space coordinates {R * , G * , B *} of the corresponding pixel in the digital output color image may be calculated as R * = R1 + R2, G * = G1 + G2, B * = B1 + B2.

[0203] In operation mode C, as shown in FIG. 3, for the generation of the digital output color image 160, the color conversion function 140 may be applied to both the digital white light color image 114 and the digital fluorescence image 112 in step 610. In step 610, as described with reference to FIG. 8, a color conversion function may be selected from a predetermined set of color conversion functions depending on the input combination and applied to the input combination.

[0204] In step 612, post-processing may be performed. For example, the digital output color image 160 may be homogenized, its contrast may be enhanced, and / or a color space conversion such as from RGB to sRGB and / or gamma correction may be performed. It is important here to note that the color conversion in steps 608 and / or 610 is not gamma correction.

[0205] In step 614, the digital output image 160 is displayed.

[0206] Some embodiments relate to a fluorescence microscope that includes a microscope, particularly a system as described in connection with one or more of FIGS. 1-6. Alternatively, the microscope may be part of a system as described in connection with one or more of FIGS. 1-6, or may be connected to a system as described in connection with one or more of FIGS. 1-6.

[0207] FIG. 7 shows a schematic diagram of a system 700 configured to implement the method described herein. System 700 includes a microscope 710 and a computer system 720. The microscope 710 is configured to image and is connected to the computer system 720. The computer system 720 is configured to implement at least a portion of the method described herein. The computer system 720 may be configured to execute a machine learning algorithm. The computer system 720 and the microscope 710 may be separate entities or may be integrated within a single common housing. The computer system 720 may be part of the central processing system of the microscope 710 and / or the computer system 720 may be part of a sub-component of the microscope 710, such as a sensor, actuator, camera, or illumination unit of the microscope 710.

[0208] The computer system 720 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system distributed across various locations such as local clients and / or one or more remote server farms and / or data centers, comprising one or more processors and one or more storage devices). The computer system 720 may include any circuit or combination of circuits. In one embodiment, the computer system 720 may include one or more processors, which can be of any kind. As used herein, a processor may be, for example, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other kind of processor or processing circuit, but is not limited thereto. Other kinds of circuits that may be included in the computer system 720 may be custom circuits, application specific integrated circuits (ASICs), etc., for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 720 may include one or more storage devices that may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives for handling removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 720 may include a display device, one or more speakers and a keyboard and / or a mouse, a trackball, a touch screen, a controller that may include a voice recognition device, or any other device that enables a user of the system to input information into the computer system 720 and receive information from the computer system 720.

[0209] Some or all of the steps may be performed by, or using, a hardware device such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the extremely important steps may be performed by such a device.

[0210] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is executable by a non-transitory recording medium, which is, for example, a digital recording medium such as a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, in which electronically readable control signals are stored that cooperate (or are capable of cooperating) with a programmable computer system to implement each method. Thus, the digital recording medium may be computer-readable.

[0211] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so that any of the methods described herein are implemented.

[0212] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to implement any of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0213] Another embodiment includes a computer program stored on a machine-readable carrier for implementing any of the methods described herein.

[0214] Thus, in other words, embodiments of the present invention are computer programs having program code for implementing any of the methods described herein when the computer program is executed on a computer.

[0215] Thus, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) including a stored computer program for implementing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium to be recorded is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein including a processor and a recording medium.

[0216] Thus, another embodiment of the present invention is a data stream or signal sequence representing a computer program for implementing any of the methods described herein. The data stream or signal sequence may be configured to be transferred, for example, via a data communication connection such as the Internet.

[0217] Another embodiment includes a processing means, for example, a computer or a programmable logic device configured or adapted to implement any of the methods described herein.

[0218] Another embodiment includes a computer having an installed computer program for implementing any of the methods described herein.

[0219] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0220] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to implement any of the methods described herein. Generally, advantageously, the methods are implemented by any hardware device.

Description of Reference Numerals

[0221] 100 Medical fluorescence observation device 101L Stereoscopic subassembly for left channel 101R Stereoscopic subassembly for right channel 102 Digital imaging system 104 Eyepiece 106 Object to be inspected 107 Biological tissue 110 White light color camera 111 Fluorescence color camera 112 Digital fluorescence color image 114 Digital white light color image 116 Phosphor 118 Second phosphor 132 Integrated / internal display 134 Direct optical path 136 beam splitter 140 color conversion function 140a~140d Different color conversion functions 140-I,140-II,140-III Different color conversion functions 142 color conversion matrix 143 set of color conversion functions 143a~143c Different sets of color conversion functions 150 pixel 150a1~150a3 Pixels in a digital white light color image 150b1~150b3 Pixels in a digital fluorescence color image 150c1~150c3 Pixels in a digital output color image 160 digital output color image 165 selector device 168 filter setting selector 170 image processor 172 output interface 174 objective lens 176 color separation assembly 178 illumination assembly 179 illumination filter 180 illumination beam splitter 182 display 184 field of view 186 computer 188 white light filter 190 fluorescence filter 192 dichroic beam splitter 194 memory 196 data transmission line 200 spectrum 202 first imaging spectrum 204 primary color or color band 206 primary color or color band 208 primary color or color band 210 blocking band 212 visible spectrum or visible light range 220 spectrum 222 Second imaging spectrum 224 Passband 226 Fluorescent emission spectrum of phosphor 228 Fluorescent emission spectrum of another phosphor 230 Part of the fluorescent emission spectrum in color band 204 232 Part of the fluorescent emission spectrum in color band 206 236 Part of the fluorescent emission spectrum in color band 208 240 Image combination 250 Spectrum 300 Multispectral image from digital white light and fluorescent color images 400 Reflection spectrum 500 Converted color 502 Recorded color 504 Natural white point 506 Recorded white point 508 Recorded color 510 Converted color 512 Recorded color area 514 Target color area 600 Recording of digital white light color image 602 Recording of digital fluorescent color image 604 Demosaicing process 606 Alignment 608 Color conversion step 610 Image combination step 612 Post-processing 614 Display 700 System 710 Microscope 720 Computer system 846 Input merging 852a~852b Tissue type 856 Set of different target mergings 858 Target merging λ Wavelength I Intensity A First operation mode of the processor B Second operation mode of the processor Coefficients of the color conversion matrix C11, C12, …, C63 Example of RGB color space R, G, B, R1, G1, B1, R2, G2, B2, R * , G * , B * (Intensity) signals in each color band or sub-band of the color space

Claims

1. An image processor (170) for a medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, - The image processor (170) is configured to acquire a digital white light color image (114) of an object (106) recorded in a first imaging spectrum (202), and the digital white light color image (114) includes a plurality of first pixels (150a), and each first pixel (150a) includes a first set of color space coordinates ({R1, B1, G1}) in a first set of color bands (R, G, B), - The image processor (170) is configured to acquire a digital fluorescence color image (112) of the object (106) recorded in a second imaging spectrum (222), and the digital fluorescence color image (112) includes a plurality of second pixels (150b), and each second pixel (150b) includes a second set of color space coordinates (R2, G2, B2) in a second set of color bands (R, G, B), The second imaging spectrum (222) overlaps with the fluorescence emission spectrum (226) of at least one phosphor (116), the second imaging spectrum (222) is different from the first imaging spectrum (202), and both the first and second imaging spectra (202, 222) overlap with the visible spectrum (212), - The image processor (170) is configured to generate a digital output color image (160) from the digital white light color image (114) and the digital fluorescence color image (112), and the digital output color image (160) includes a plurality of output pixels (150c), The image processor (170) calculates the color (R ＊ , G ＊ , B ＊ ) of the output pixel (150c) by applying a color conversion function (140) to an input merge (846, {R1, R2, G1, G2, B1, B2}) of the first set of the color space coordinates ({R1, B1, G1}) of the first pixel (150a) and the second set of the color space coordinates ({R2, G2, B2}) of the second pixel (150b), and is configured to: The application of the color conversion function (140) depends on the color space coordinates (R1, R2, G1, G2, B1, B2) of the input combination ({R1, R2, G1, G2, B1, B2}), Image processor (170).

2. The first imaging spectrum (202) and the second imaging spectrum (222) are complementary to each other, The image processor (170) according to Claim 1.

3. The digital white light color image (114) and the digital fluorescence color image (112) are aligned in relation to each other, and the first pixel (150a) including the first set of the color space coordinates and the second pixel (150b) including the second set of the color space coordinates are corresponding pixels, and the output pixel (150c) and the first and the second pixels are corresponding pixels. The image processor (170) according to claim 1 or 2.

4. The color conversion function (140) includes a color conversion matrix (142), one dimension of the color conversion matrix (142) corresponds to the total amount of color bands in the first and the second sets, and another dimension of the color conversion matrix (142) corresponds to the amount of color bands in the digital output color image (160). The image processor (170) according to any one of claims 1 to 3.

5. The image processor (170) includes at least two different color conversion functions (140, 140-I, 140-II, 140-III), and the image processor is adapted to select one of the at least two different color conversion functions depending on the input merging (846). The image processor (170) according to any one of claims 1 to 4.

6. The image processor includes at least two sets (856) of different target mergings (858), and each of the at least two different color conversion functions (140, 140-I, 140-II, 140-III) is assigned to a different set (856) among the at least two sets (856) of the different target mergings (858). The image processor (170) according to claim 5.

7. The image processor is configured to apply a color conversion function assigned to a set (856) including a target merging (858) corresponding to the input merging (846) among the at least two sets (856) of the different target mergings (858) among the at least two color conversion functions. The image processor (170) according to claim 6.

8. The image processor is as follows, that is, - at least one target merging is a predetermined set of target mergings corresponding to a color representing blood containing oxygen, and - at least one target merging is a predetermined set of target mergings corresponding to a color representing blood not containing oxygen. - at least one target merge corresponds to a predetermined set of target merges corresponding to the color representing living gray matter, - at least one target merge corresponds to a predetermined set of target merges corresponding to the color representing living white matter, - at least one target merge corresponds to a predetermined set of target merges corresponding to the color representing bone, - at least one target merge corresponds to a predetermined set of target merges corresponding to the color representing nerve tissue, including at least one of: The image processor (170) according to claim 6 or 7.

9. At least two images of the group including the digital white light color image (114), the digital fluorescence color image (112), and the digital output color image (160) are represented in the same color space. The image processor (170) according to any one of claims 1 to 8.

10. A medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, wherein the medical fluorescence observation device (100) includes the image processor (170) according to any one of claims 1 to 9, a fluorescence color camera (111) configured to record the digital fluorescence color image (112), a white light color camera (110) configured to record the digital white light color image (114), The medical fluorescence observation device (100) comprising.

11. The medical fluorescence observation device (100) is a surgical fluorescence microscope. The medical fluorescence observation device (100) according to claim 10.

12. A computer-implemented image processing method for a fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, wherein the computer-implemented image processing method - obtaining a digital white light color image (114) of an object (106) recorded in a first imaging spectrum (202), the digital white light color image (114) including a plurality of first pixels (150a), each first pixel (150a) including a first set of color space coordinates (R1, B1, G1) in a first set of color bands (R, G, B), - Obtaining the digital fluorescence color image (112) of the object (106) recorded in the second imaging spectrum (222), wherein the digital fluorescence color image (112) includes a plurality of second pixels (150b), and each second pixel (150b) includes a second set of color space coordinates (R2, G2, B2) in a second set of color bands (R, G, B) ({R2, B2, G2}); including; the second imaging spectrum (222) overlaps with the fluorescence emission spectrum (226) of at least one phosphor (116), the second imaging spectrum (222) is different from the first imaging spectrum (202), and both the first and second imaging spectra (202, 222) overlap with the visible spectrum (212); The computer-implemented image processing method; - Generating a digital output color image (160) from the digital white light color image (114) and the digital fluorescence color image (112), wherein the digital output color image (160) includes a plurality of output pixels (150c); By applying a color conversion function (140) to the input combination ({R1, R2, G1, G2, B1, B2}) of the first set ({R1, B1, G1}) of the color space coordinates of the first pixel (150a) and the second set ({R2, G2, B2}) of the color space coordinates of the second pixel (150b), the color (R ＊ , G ＊ , B ＊ ) of the output pixel (150c) is calculated, The color conversion function (140) is applied depending on the color space coordinates (R1, R2, G1, G2, B1, B2) within the input combination ({R1, R2, G1, G2, B1, B2}); Method. [

13. ] A computer program product or computer-readable medium, including instructions that cause a computer to execute the method according to Claim 12 when the program is executed by the computer. [

14. ] A method for operating a medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the method including: - Recording a digital fluorescence color image (112) in a second imaging spectrum (222) using a fluorescence color camera (111); - Recording a digital white light color image (114) in a first imaging spectrum (202) using a white light color camera (110); The method including. [

15. ] The method includes: - Recording the digital fluorescence color image (112) as a reflected image of the object (106); - Recording the digital white light color image (114) as a reflected image of the object (106); further including; The method according to Claim 14.

Images