Method of using two color images and a color camera for fluorescence and white light, a processor, and a medical fluorescence observation device

Abstract

This application relates to an image processor (170) and a computer-implemented image processing method for generating a digital output color image (160) of an object (106) using a medical fluorescence observation device (100) such as a microscope or an endoscope. The application further relates to a medical fluorescence observation device (100) including such an image processor (170) and a method for operating such a device including the above computer-implemented method. Two selectable operation modes (A, B) are provided. In a first operation mode (A), the digital output color image (160) is generated by simply combining a digital white light color image (114) and a digital fluorescence image (112), whereby the digital output color image (160) appears in the field of view through an eyepiece. In a second operation mode (B), a color conversion function (140, 140a, 140b) is applied to at least one of the digital white light color image (114) and the digital fluorescence color image (112) to enhance features introduced by an optical filter and / or correct color distortion.

Description

Technical Field

[0001] The claimed subject matter relates to a computer-implemented image processing method and an image processor for generating a digital output color image of an object using a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope. Further, the claimed subject matter relates to a method for operating such a medical fluorescence observation device and to the medical fluorescence observation device itself.

Background Art

[0002] In existing medical fluorescence observation devices, fluorescence and anatomical images are recorded by a single camera. Such a setup may be used for fluorescence imaging of pPIX after administration of 5-ALA to reveal tumors. The selection of the optical filter is such that the camera captures both the fluorescence at the red wavelength and a part of the excitation light at the blue wavelength. The excitation light reveals a part of the anatomical structure of the tissue, while the fluorescence indicates the tumor. The optical filters used today are standardized for pPIX imaging.

[0003] This type of fluorescence imaging is particularly helpful in neurosurgical oncology, but still has significant drawbacks. The blue excitation light used to visualize the anatomical structure of the tissue has a very low intensity and provides poor visualization because the anatomical structure is shown only in a blue monochromatic tone. In particular, under blue light, bleeding is very difficult to see or may even be impossible to see. Therefore, this setup is not optimal for live surgical guidance.

[0004] Other medical fluorescence observation devices use a grayscale camera to record fluorescence and a color camera to record white light images. In such a setup, the fluorescence image and the white light image are digitally mixed and the fluorescence is marked with a pseudo-color. The drawback here is that this pseudo-color does not accurately represent the color of the fluorescence as perceived by the human eye.

Summary of the Invention

Problems to be Solved by the Invention

[0005] From the above viewpoints, there is a need to provide an apparatus and a method that offer improved and more natural representations of anatomical images and fluorescence. At the same time, the user needs to be able to compare the improved representation with the actual scene.

Means for Solving the Problems

[0006] This need is addressed by providing an image processor for a medical fluorescence observation apparatus such as a fluorescence microscope or a fluorescence endoscope. The processor is configured to acquire a digital white light color image of an object recorded in a first imaging spectrum and a digital fluorescence color image of the object recorded in a second imaging spectrum. The second imaging spectrum overlaps with the fluorescence emission spectrum of at least one phosphor. The second imaging spectrum is different from the first imaging spectrum. Both the first and second imaging spectra overlap with the visible spectrum. The image processor is configured to be selectively operated in one of a first operation mode and a second operation mode. The first operation mode includes an image combining step. The second operation mode includes a color conversion step and an image combining step. The image processor is configured to generate a digital output color image by combining the digital white light color image and the digital fluorescence color image in the image combining step. The image processor is configured to provide at least one color conversion function from a group including a first color conversion function applied to the digital white light color image and a second color conversion function applied to the digital fluorescence color image in the color conversion step.

[0007] This need is also addressed by a computer-implemented image processing method for a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope. The computer-implemented image processing method includes the following steps: obtaining a digital white light color image of an object recorded in a first imaging spectrum; and obtaining a digital fluorescence color image of the object recorded in a second imaging spectrum. The second imaging spectrum overlaps with the fluorescence emission spectrum of at least one phosphor. The second imaging spectrum is different from the first imaging spectrum. Both the first and second imaging spectra overlap with the visible spectrum. The method is configured to be selectively executed in one of a first operation mode and a second operation mode. In the first operation mode, an image combining step is executed. In the second operation mode, a color conversion step and an image combining step are executed. In the image combining step, a digital output color image is generated by combining the digital white light color image and the digital fluorescence color image. In the color conversion step, at least one color conversion function is applied from a group including a first color conversion function applied to the digital white light color image and a second color conversion function applied to the digital fluorescence color image.

[0008] By providing two selectable operation modes, it becomes possible for both to provide a digital output color image in a first operation mode corresponding to what is seen through the eyepiece of a medical observation device.

[0009] The digital white light color image typically records the anatomical structure of the surgical scene. The fluorescence color image can record the fluorescence of one or more phosphors, but can also be used to provide additional information about the anatomical structure when the fluorescence is not used for excitation respectively.

[0010] By providing two color conversion functions that act separately on a digital white light color image and a digital fluorescence color image respectively, it becomes possible to decouple the color conversion of the digital white light color image from the color conversion applied to the digital fluorescence color image. This makes it possible to optimize each of these two images independently for visual recognition before they are combined for the formation of the digital output color image.

[0011] Furthermore, by using a color camera for both the fluorescence image and the white light image, a much higher color accuracy and faithful color output image can be provided.

[0012] The terms "image processor", "processor" and "data processing device" are used synonymously.

[0013] The above solution means may be further improved by the following features, which may be added or combined independently of each other, and each feature has its own advantageous technical effect. Each of the following features may be used to improve one of the above methods and / or to improve one of the above devices, regardless of whether a particular feature is mentioned in relation to the method only or in relation to the device only.

[0014] A computer-implemented image processing method may include a step of demosaicking at least one of a digital white light color image and a digital fluorescence color image. Demosaicking may avoid color artifacts.

[0015] A computer-implemented image processing method may further include a step of normalizing at least one of a digital white light color image and a digital fluorescence color image. Normalization facilitates the joint processing of the digital white light color image and the digital fluorescence color image.

[0016] The computer-implemented image processing method can further include the step of aligning at least one of a digital white light color image and a digital fluorescence color image. By this alignment, image features existing in both the digital white light color image and the digital fluorescence color image are represented in the same size and orientation in each image. After alignment, the digital white light color image and the digital fluorescence color image match each other.

[0017] The image processor may be configured to execute any of the above processing steps.

[0018] In one embodiment, the fluorescence color camera can record an additional digital white light color image, and such a digital white light color image can complement the digital white light color image recorded by the white light color camera. Therefore, the fluorescence color camera is not necessarily limited to recording only fluorescence images. However, the fluorescence camera may be restricted to recording only images in the second imaging spectrum, for example, by replacing one or more filters, if the hardware is not changed.

[0019] According to another advantageous embodiment of the computer-implemented image processing method and / or the image processor, the first imaging spectrum and the second imaging spectrum are complementary to each other. In such a configuration, crosstalk between fluorescence and white light is avoided, while maintaining an accurate representation of the fluorescence of at least one phosphor on the one hand and the reflectance of the object on the other hand. Furthermore, by giving the digital fluorescence color image and the digital white light color image spectra that are complementary to each other, respectively, joint processing of the two images for forming a multi-spectral image becomes easier.

[0020] The digital white light color image may be represented using at least three first color bands in a first color space.

[0021] The digital fluorescence color image may be represented in a second color space including at least three second color bands. The first and second color spaces are preferably the same, whereby the first and second color bands are also the same. However, this is not necessarily the case. The first and second color spaces may also be different.

[0022] The digital output color image may be generated in a third color space including at least three third color bands. The third color space may be the same color space as the first and / or second color spaces, or may be a different color space.

[0023] Any of the first, second, and / or third color spaces may be an RGB color space including three color bands R, G, B, or may be any other color space such as a multi-spectral color space or a hyper-spectral color space, i.e., a color space using four or more color bands. Each color band is represented by color space coordinates. Even for the same color, if the color spaces are different, the color space coordinates are also different. Each pixel of the digital white light color image, the digital fluorescence color image, and the digital output color image includes color space coordinates representing the color of each pixel.

[0024] According to one embodiment of a computer-implemented image processing method and / or an image processor, the first color band and the second color band are mapped onto or converted into the third color band of the digital output color image using a color conversion function. The color conversion function may be a non-linear function, but is preferably a linear conversion function.

[0025] The linear conversion can include a color conversion matrix or other types of linear functions. To map the first and second color bands onto the third color band, the color conversion matrix may be applied to the first color band and the second color band. Preferably, each color band of the first color band and each color band of the second color band are input into the linear conversion as separate color bands. In particular, each color band of the first color band and each color band of the second color band may be input into the linear conversion simultaneously.

[0026] The color conversion matrix or linear function can be resident in the memory of an image processor or a medical fluorescence observation device.

[0027] The color conversion matrix can have a dimension of the number X in one direction × the number Y in another direction. The number X may be the sum of the amount of the first color band, i.e., the number of color bands within the first color band, and the amount of the second color band, i.e., the number of color bands within the second color band. The number Y may be the amount of color bands in the third color space. For example, when the first, second, and third color spaces are each an RGB color space, the color conversion matrix has dimensions of 6×3 or 3×6 or 3×3. In particular, one dimension of the color conversion matrix may correspond to the sum of the dimensions of the color spaces of the digital white light color image and the digital fluorescence color image. Another dimension of the color conversion matrix may correspond to the dimension of the color space of the digital output color image.

[0028] Therefore, the third color band, i.e., the color band of the digital output color image, can result from a linear combination, particularly an addition, of the color bands of the digital white light color image and the digital fluorescence color image.

[0029] According to one aspect, the digital fluorescence color image and the digital white light color image are co-processed as a multi-spectrum image whose color bands are formed or constituted by complementary color bands of the digital fluorescence color image and the digital white light color image. The color space coordinates of this multi-spectrum image may correspond to the merger of the set of color space coordinates of the digital white light color image and the set of color space coordinates of the digital fluorescence color image. The multi-spectrum image may be physically created in an image processor or purely logically created by co-processing the color space coordinates of the digital white light color image and the digital fluorescence color image while keeping them separated.

[0030] Processing as a multi - spectral image leads to improved color representation, especially when the fluorescence image is represented by reflected white light recorded in a second imaging spectrum and thus can complement the white - light information in the white - light color image. By using this procedure, the white - light information is included in the color band of the digital fluorescence color image and will thus have a finer color granularity than that provided by the digital white - light color image. When the digital fluorescence color image is represented by fluorescence emission, processing the digital white - light color image and the digital fluorescence color image as a multi - spectral image results in a faithful representation of the object under both reflected white light and fluorescence.

[0031] According to another advantageous embodiment, the first imaging spectrum can include a first sub - band in a color band of a color space, and the second imaging spectrum can include a second sub - band in this color band, where the first sub - band and the second sub - band are preferably complementary to each other. The color space can be the color space of the digital white - light color image, the color space of the digital fluorescence color image, or preferably the color space of the digital output color image. Having such a subdivided color band facilitates the linear transformation for mapping the color bands of the fluorescence and white - light color images to the color band of the digital output color image. It is preferred that the sub - bands of one color band together complete each respective color band. This ensures that no useful spectral information is lost.

[0032] Naturally, in two or more color bands of each color space, the first and second sub - bands may be present. The more sub - bands are used in different color bands, the more accurate the spectral information recorded in each white - light or fluorescence color image will be. The sub - bands within a color band itself may be divided into a plurality of different wavelength bands that are distinct from each other.

[0033] Preferably, the second imaging spectrum includes IR wavelengths. This enables the capture of fluorescence emission in the near-infrared region and can add additional information regarding anatomical structures when digital fluorescence color images are not used for fluorescence capture.

[0034] According to one aspect, the first and / or second color conversion functions are configured to shift the (recorded) color to a (different) color in a predefined color space. The predefined color space may be the color space of the digital white light color image and the digital fluorescence image, or it may be a different color space. The shift is predetermined by the color conversion function in relation to at least one of the amount and direction of the shift in the color space where the color is represented by color space coordinates.

[0035] For example, the natural color of oxygenated or deoxygenated blood may be captured only incompletely in the first and second imaging spectra, respectively. This is because the reflection spectrum of blood may contain wavelengths that are not recorded in the first or second imaging spectrum, respectively. Therefore, when blood is recorded in a digital white light color image or a digital fluorescence image, its color cannot be seen naturally, making it more difficult to recognize blood vessels. By using the first and / or second color conversion functions, the (recorded) color may be shifted to a more natural color in the respective digital white light color image or fluorescence color image.

[0036] In another example, the recorded colors of oxygenated blood and deoxygenated blood may be shifted differently in a predefined color space, such as the color space of a digital white light color image or a digital fluorescence color image. Here too, both of these colors are incompletely represented in the first or second imaging spectrum. The recorded color of deoxygenated blood may be further shifted towards blue, while the color of oxygenated blood is further shifted towards red. This may require different amounts and / or directions of shift for the two recorded colors, and one color may be shifted differently from another within the color space in relation to at least one amount and direction by a color conversion function.

[0037] Naturally, the above examples apply to any other type of tissue, such as fluid tissues like lymph, and solid tissues like bone, muscle, and / or nerve tissue.

[0038] The converted colors may correspond to natural colors, pseudo-colors such as neon colors, or "hyper-real" colors, in which case at least one of the color appearance parameters, specifically, at least one of hue, chroma, saturation, lightness, and luminance, is modified compared to the natural color to enhance visibility and contrast.

[0039] The natural or faithful color in this context corresponds to the color perceived by a CIE standard observer under a standard light source such as CIE light source A, B, or C, or any other standardized white light illumination.

[0040] The first and / or second color conversion functions may be configured to convert the colors within each digital white light or fluorescence color image into colors that do not lie within the respective first and / or second imaging spectra. Thus, the color conversion functions may be used to correct for imaging defects that result in a limited first and / or second imaging spectrum.

[0041] In another embodiment, the first and / or second color conversion function may be configured to shift the white point of the digital white light and / or the fluorescent color image to a predetermined position, particularly to another white point. The white point to which the predetermined white point is shifted can particularly correspond to a standard white point defined in a standard light source such as a CIE light source. This enables centering of the color space.

[0042] Furthermore, the first and / or second color conversion function may be configured to perform white balance adjustment or color balance adjustment on the respective digital white light and / or fluorescent color image. In color balance adjustment, the intensity of colors is adjusted so as to properly render at least some colors, particularly intermediate colors. In white balance adjustment, multiple colors are adjusted so that a white object appears white and is not colored.

[0043] According to another aspect, the first and / or second color conversion function may be configured to expand a continuous or fragmented region in the color space into a larger region in the respective digital white light and / or fluorescent color image. Such a color conversion function can better visualize the subtle differences between adjacent colors as the adjacent colors move apart. For example, when the region of the fluorescent color of a specific phosphor is expanded, the nuances of the fluorescence become more prominent. The expansion of the region may of course be accompanied by an appropriate color shift, so the expansion of the region occurs for natural or unnatural colors.

[0044] According to yet another aspect, the first and / or second color conversion function may be configured to shift all the colors in the color space of the respective digital white light and / or fluorescent color image.

[0045] Any of the above-described color conversion functions may be combined into a single color conversion function and / or any of the above-described color conversion functions may be sequentially applied to each of the digital white light and / or fluorescent color image.

[0046] Furthermore, it would be even more advantageous if the first and / or second color conversion functions were configured to be applied depending on the color of each pixel of the digital white light color image and / or the digital fluorescence color image. This generally allows different colors to be processed in different ways.

[0047] According to another aspect, a plurality of different first and / or second color conversion functions may be provided, and the processor may be configured to determine or select a first and / or second color conversion function from among the plurality of different first and / or second color conversion functions applied to the pixel depending on the color of the pixel. The plurality of first and / or second color conversion functions can include any of the color conversion functions described above. According to this aspect, the desired color dependency correction may be applied to each digital white light and / or fluorescence color image. For example, regions around some colors in a digital white light color image may be expanded to make the color differences more prominent. Another color, such as the color of bone and / or nerve, may be simply shifted to be more natural. Another color, and thus the tissue associated with this color, may be emphasized by being converted to a neon color.

[0048] The above computer-implemented image processing method may be executed when executing a method for operating a medical fluorescence observation device. The above-described image processor may be part of a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope, and in particular, may be part of a fluorescence microscope or a fluorescence endoscope configured for surgery such as neurosurgery, orthopedic surgery, and / or ophthalmic surgery. The fluorescence microscope may be, for example, a laboratory microscope used for biopsy.

[0049] A medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope can include an image processor configured to execute any of the above-described steps. Further, the medical fluorescence observation device can include a fluorescence color camera configured to record a digital fluorescence color image and a white light color camera configured to record a digital white light color image.

[0050] Since fluorescence may sometimes be of relatively low intensity, the fluorescence color camera may be operated with a longer integration time.

[0051] A method for operating a medical fluorescence observation device can include the step of recording a digital white light color image using the white light color camera and the step of recording a digital fluorescence color image using the digital fluorescence color camera. The medical fluorescence observation device can include a digital fluorescence color camera and a white light color camera.

[0052] To reduce post-processing, the fluorescence color camera and the white light color camera can have overlapping coaxial and / or preferably the same field of view.

[0053] The medical fluorescence observation device can be a stereoscopic device or a monocular device. In one embodiment, the digital fluorescence color camera and the white light color camera may be provided in each stereoscopic channel. Alternatively, in a medical monocular fluorescence observation device, only a single white light color camera and a single digital fluorescence color camera may be provided.

[0054] In an alternative arrangement configuration of the medical stereoscopic fluorescence observation device, the fluorescence color camera may be provided in one stereoscopic channel, while the white light color camera may be provided in the other stereoscopic channel. In such an arrangement configuration, the fluorescence color camera may be configured to record a (white light) reflected image in the second imaging spectrum. In the case of the (white light) reflected image, this arrangement configuration provides stereoscopic vision, but strictly speaking, this stereoscopic vision is limited to the overlapping part of the first and second imaging spectra. However, this is hardly noticed by a human observer. When fluorescence emission is recorded, the fluorescence color camera provides monocular vision based on the fluorescence emission of the object, while the white light color camera provides monocular vision based on the reflectance of the object.

[0055] The medical fluorescence observation device can include an optical color separation assembly configured to split the light incident thereon into a first imaging spectrum and a second imaging spectrum. The color separation assembly can include optical elements such as a beam splitter, particularly a dichroic beam splitter, and / or optical filters such as a fluorescence filter that blocks light other than fluorescence emission and a white light filter that blocks fluorescence emission.

[0056] The medical fluorescence observation device can further include an illumination assembly that is preferably adjustable by including, for example, a plurality of LEDs or OLEDs of different colors or other light sources that emit light in a plurality of different spectral bands and can be individually turned on and off.

[0057] To facilitate post-processing and ensure that the recorded intensity values of digital white light images and digital fluorescence color images are comparable to each other, it is preferable that the digital fluorescence color image and the digital white light color image are recorded simultaneously, and / or the same exposure time is used, and / or the same gain and / or the same white balance adjustment and / or color correction are used. Furthermore, the gain of the digital fluorescence color camera and the gain of the digital white light color camera may be maintained at a certain ratio, or may be automatically adapted otherwise. Among these possibilities, it is most preferable that the white light color camera and the fluorescence color camera are synchronized with respect to at least one of the exposure time and the gain. However, the fluorescence color camera may be operated with a longer integration time. This enables compensation for low fluorescence intensity. The white light color camera and / or the fluorescence color camera are preferably CCD or CMOS cameras. The fluorescence camera and the white light color camera are preferably identical.

[0058] According to another aspect, the medical fluorescence observation device can include a memory storing a plurality of different color conversion matrices. Each color conversion matrix represents a different combination of the first and second imaging spectra, i.e., a different filter set used in the color separation assembly, which is required when different phosphors or combinations of different phosphors are used. Examples of phosphors used in this context are 5-ALA / pPIX, fluorescein, and ICG. Each of these phosphors requires excitation at a different wavelength and emits fluorescence at a different wavelength. Thus, in a surgical environment where ICG is used, the first and second imaging spectra are different from those in a surgical environment where 5-ALA / pPIX is used and require different color conversion matrices. Therefore, the sub-bands of the first and second imaging spectra in the color bands of each color space are different for each case of the phosphor.

[0059] The color conversion matrix may be automatically or manually selected from a plurality of color conversion matrices depending on the phosphor used. For example, a medical fluorescence observation device may be configured to automatically detect the configuration or setting of a color separation assembly and select a color conversion matrix depending on this setting. For example, when the optical filter of the color separation assembly is replaced for the use of 5-ALA / pPIX, the medical fluorescence observation device can automatically select the color conversion matrix for this 5-ALA / pPIX. Of course, this selection can also be made manually, or the automatic selection may be manually overwritten.

[0060] The subject matter recited in the claims also relates to a computer-readable medium and a computer program that include instructions for causing a computer to perform computer-implemented image processing in any of the above-described embodiments.

[0061] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0062] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or apparatus corresponds to a step or a feature of a step. Similarly, aspects described in the context of a step also represent a description of the corresponding block or item or feature of the corresponding apparatus.

[0063] Hereinafter, the present invention will be exemplarily described with reference to embodiments and drawings. The combinations of features shown in these embodiments should not be regarded as limiting. For example, features that are not required in a particular application of an embodiment having the above-described technical effects may be omitted. Conversely, in some of the embodiments described below, features not described above may be added if they are required for the technical effects associated with this particular feature in a particular application.

[0064] Throughout this specification and the drawings, the same reference numerals are used for elements that correspond to each other with respect to function and / or structure.

Brief Description of the Drawings

[0065]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0066] Exemplary embodiments of the present invention will be described with reference first to FIG. 1.

[0067] Figure 1 schematically shows a medical fluorescence observation apparatus 100. The medical fluorescence observation apparatus 100 may be a fluorescence microscope or a fluorescence endoscope. The main difference between a microscope and an endoscope is that in an endoscope (not shown), an object 106 is observed through an optical fiber that brings the object 106 to be inspected close to the object 106, for example, by insertion into the body, while in a microscope, the objective lens 174 is directed at the object. The medical fluorescence observation apparatus in Figure 1 is a microscope, but the following description also applies to an endoscope. The medical fluorescence observation apparatus 100 may be a medical fluorescence observation apparatus used in a surgical operation. The medical fluorescence observation apparatus 100 may also be a medical fluorescence observation apparatus such as a laboratory microscope used in a laboratory. The object 106 to be inspected may consist of or include a biological tissue 107.

[0068] The object 106 may include one or more phosphors 116, 118. At least one phosphor may be a phosphor naturally contained in the object. For example, bone and blood contain phosphors. At least one phosphor may be added to the object 106, for example, by injecting it into the biological tissue 107. Examples of phosphors that may be added to the object 106 are ICG, fluorescein, and / or 5-ALA. 5-ALA is synthesized into pPIX intracellularly.

[0069] The medical observation apparatus 100 is a fluorescence apparatus. This means that the medical fluorescence observation apparatus is configured to observe, record, and preferably also excite the fluorescence of one or more phosphors 116, 118.

[0070] The medical fluorescence observation apparatus 100 may be a stereoscopic apparatus as exemplarily shown in Figure 1. Thus, the apparatus may include two identical sub-assemblies 101L and 101R for each of the two stereoscopic channels. Since the two sub-assemblies 101L, 101R are identical in function and structure, the following description focuses on the right sub-assembly 101R, but is equally applicable to the left stereoscopic channel 101L.

[0071] The medical fluorescence observation device 100 may alternatively be a monocular device. In this case, only one of the two sub-assemblies 101L, 101R may be present. Therefore, the following description also applies equally to the medical monocular fluorescence observation device 100.

[0072] The medical fluorescence observation device 100 may be used to generate at least one digital white light color image 114, which represents a reflected image of the object 106 over the entire visible light range. This visible light range or visible spectrum includes wavelengths from about 310 nm to about 1100 nm, or from about 380 nm to about 750 nm, or from about 450 nm to about 700 nm. When a fluorescence spectrum or multiple phosphors are used, the fluorescence spectra of the phosphors 116, 118 are preferably omitted from the spectrum recorded in the digital white light color image 114. This ensures that only the reflected light is included in the digital white light color image 114 when fluorescence is present. For example, when 5-ALA / pPIX is used as a phosphor, the fluorescence spectrum from about 625 nm to about 650 nm may not be recorded in the digital white light color image 114.

[0073] When light of a specific wavelength is used to excite fluorescence, a spectrum including or consisting of these wavelengths may also not be recorded or represented in the digital white light color image 114. For example, when 5-ALA / pPIX is used as a phosphor, fluorescence may be excited by irradiating the object 106 with wavelengths between about 380 nm and about 450 nm. For fluorescein, ICG and other phosphors, although different from 5-ALA / pPIX in terms of excitation and emission spectra, known ranges apply.

[0074] By not recording the fluorescence excitation spectrum and the fluorescence emission spectrum, the digital white light color image 114 represents the reflectance of the object 106, that is, the white light image of the object 106 that would be seen by a human observer. Therefore, the digital white light color image 114 may be regarded as a true color color image.

[0075] The digital imaging system 102 may further be used to generate a digital fluorescence color image 112 of the object 106. The digital fluorescence color image 112 represents the fluorescence emission of one or more phosphors 116, 118. Therefore, preferably, the digital fluorescence color image 112 does not record wavelengths outside the emission spectrum or the emission spectra of one or more phosphors.

[0076] Both the digital white light color image 114 and the digital fluorescence color image 112 are color images. These images are recorded using at least three color bands, i.e., the primary colors of the color space. For example, the digital white light color image 114 and the digital fluorescence color image 112 may be recorded in the RGB color space using three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescence color image 112 may be recorded in different color spaces respectively, and / or may be represented by multi-spectral or hyperspectral color images. The digital white light color image 114 and the digital fluorescence color image 112 do not necessarily have to be recorded in the same color space, but this is preferred.

[0077] The digital white light color image 114 and the digital fluorescence image 112 include pixels 150. Each pixel includes color space coordinates, which represent the color of the pixel within the color space. Even for the same color, the color space coordinates are different if the color spaces are different.

[0078] In a color space such as the RGB color space, each color of pixel 150 is represented by a triplet of three integer values, where each integer value indicates the intensity of one of the primary colors R, G, and B. The triplet corresponds to the color space coordinates.

[0079] For example, the highest intensity red may be indicated by the triplet [255, 0, 0]. The highest intensity green may be indicated by [0, 255, 0], and the highest intensity blue may be indicated by [0, 0, 255]. Thus, the RGB color space is a three-dimensional space with three color space coordinates. In comparison, the CMYK color space would be a four-dimensional space. Color can be regarded as a point within the color space pointed to by a vector such as [0, 0, 255]. A multi-spectrum or hyperspectrum color space with n color bands would consequently be an n-dimensional color space, and each color would be represented by n sets of color space coordinates.

[0080] The spectrum recorded in the digital white light color image 114 and the spectrum recorded in the digital fluorescence color image 112 are preferably complementary to each other, that is, they do not overlap.

[0081] More specifically, the digital imaging system 100 can include a digital imaging system 102 for generating the digital fluorescence color image 112 and the digital white light color image 114. This digital imaging system 102 can include a white light color camera 110 and a fluorescence color camera 111.

[0082] The white light color camera 110 is configured to record the digital white light color image 114. In particular, the white light color camera 110 may be configured to generate a stream of the digital white light color image 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images over the entire visible spectrum of the wavelengths shown above. The white light color camera 110 may be a CCD, CMOS, or multi-spectrum or hyperspectrum camera.

[0083] The fluorescence color camera 111 is configured to record a digital fluorescence image 112. In particular, the fluorescence camera 111 may be configured to generate a stream of digital fluorescence color images 112 in the form of a digital video stream. The fluorescence color camera 111 may be configured to record the digital fluorescence color image 112 only with the fluorescence spectra of at least one phosphor 116, 118. The fluorescence camera 111 may be configured to record the digital fluorescence image only in one or more narrow optical bands. These narrow bands should overlap with the fluorescence spectra of one or more phosphors 116, 118 where the fluorescence is recorded. Preferably, the fluorescence spectra of the phosphor 116 and the second phosphor 118 are at least partially, preferably completely separated, whereby the fluorescence camera 111 can record light in two separate fluorescence bands spaced apart from each other.

[0084] The fluorescence color camera 111 may be a CCD, CMOS, or multi - spectral or hyperspectral camera. The white - light color camera 110 and the fluorescence color camera 111 are preferably of the same type, but this is not necessarily required. Since fluorescence usually has a very low intensity, the fluorescence color camera 111 can have a longer integration time.

[0085] The respective fields of view 184 of the cameras 110, 111 are preferably aligned or even coincide coaxially. Thus, the cameras 110, 111 preferably provide the same field of view 184 at the same viewing point and focal length. This results in the same representation of the object 106 in the images 112, 114 generated by the different cameras 110, 111. Both cameras 110, 111 can use the same objective lens 174.

[0086] When the coincidence of the viewpoints and the fields of view cannot be optically generated, it may be generated by image processing by applying a coincidence or alignment routine to the digital images 112, 114 as further described below.

[0087] Preferably, the two cameras 110, 111 are operated synchronously. In particular, the exposure times may be synchronized. Thus, the medical fluorescence observation apparatus 100 may be configured to simultaneously generate a digital white light color image 114 and a digital fluorescence image 112.

[0088] Preferably, the gains of the two cameras 110, 111 are synchronized, that is, the two cameras 110, 111 are adjusted simultaneously. Moreover, the ratio between the gain applied in camera 110 and the gain applied in camera 111 may be constant even when the gain is changed. Gamma correction and color adjustment or white balance may be switched off or may be maintained constant.

[0089] By any of the above, the comparison, joint processing and / or combination of the two images 112, 114 are facilitated.

[0090] An optical color separation assembly 176 may be provided to separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescence color image 112, that is, to separate the reflection spectrum from the fluorescence spectrum. This color separation assembly 176 can include optical elements such as a dichroic beam splitter 192 which may be dichroic. The color separation assembly 176 can further or alternatively include an optical white light filter 188 and / or an optical fluorescence filter 190.

[0091] The fluorescence filter 190 is preferably configured to transmit light in the fluorescence spectrum of one or more phosphors 116, 118 and block light outside the fluorescence spectrum.

[0092] The fluorescence filter 190 may be configured as a band - pass filter including one or more pass - bands. Each pass - band should overlap with the fluorescence emission spectrum of each phosphor 116, 118 for which fluorescence is to be recorded. Since the fluorescence filter 190 is in the optical path between the beam splitter 192 and the fluorescence camera 111, only the wavelengths of the pass - band of the fluorescence filter 190 are transmitted to the fluorescence camera 111.

[0093] The white - light filter 188 is preferably configured to block light within the fluorescence spectra of one or more phosphors 116, 118. The white - light filter 188 may also be configured to block light within the fluorescence excitation spectrum.

[0094] The white - light filter 188 is preferably configured as a band - stop filter whose stop - band corresponds to the pass - band of the fluorescence filter 190 or at least includes its pass - band. The white - light filter 188 is located in the optical path between the beam splitter 192 and the white - light camera 110. Thus, the white - light camera 110 records only the wavelengths outside the blocking band of the white - light filter 188 and, therefore, also records the wavelengths outside the pass - band of the fluorescence filter 190.

[0095] Either the white - light filter 188 or the fluorescence filter 190 may be an adjustable filter.

[0096] When the beam splitter 192 is a dichroic beam splitter, at least one of the filters 188, 190 may be omitted since in this case the optical - spectrum filtering is already integrated in the dichroic beam splitter. Then, the above description of the pass - band and the blocking band should be mutatis mutandis applicable to the dichroic beam splitter 192.

[0097] Thus, the white light color camera 110 records the digital white light color image 114 in a reflection spectrum, which is a first imaging spectrum different from the second imaging spectrum recorded as a fluorescence spectrum by the fluorescence camera. The wavelengths included in the first and second imaging spectra are determined by the filter settings of the color separation assembly 176.

[0098] The medical fluorescence observation device 100 can further include an illumination assembly 178, which is preferably configured to illuminate the object 106 through the objective lens 174 through which the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 may be configured to selectively generate white light, i.e., light uniformly distributed over the entire visible spectrum, and fluorescence excitation light including only light of wavelengths that excite the fluorescence of at least one phosphor 116, 118. The illumination light generated by the illumination assembly 178 may be supplied to the objective lens 174 using an illumination beam splitter 180.

[0099] The illumination filter 179 may be provided depending on the phosphor and its specific excitation spectrum of fluorescence. For example, when 5-ALA / pPIX is used as the phosphor, the illumination filter may have a transmittance of 90% - 98% up to a wavelength of 425 nm, a transmittance between 0.5% - 0.7% between wavelengths of 450 nm - 460 nm, a transmittance not exceeding 0.1% between wavelengths of 460 nm - 535 nm, and a substantially zero transmittance exceeding a wavelength of 535 nm.

[0100] Instead of or in addition to the illumination filter 179, the illumination assembly 178 can include an adjustable light source including, for example, a number of different colored LEDs or OLEDs.

[0101] The medical fluorescence observation device 100 can further include an image processor 170. This image processor 170 can be a hardware module such as a microprocessor or a software module. The image processor 170 can be a combination of both a hardware module and a software module, for example, by using a software module configured to be executed on a specific processor such as a vector processor, a floating-point graphic processor, a parallel processor, and / or a plurality of processors. The image processor 170 can be a part of a general-purpose computer 186 such as a PC.

[0102] The image processor 170 is configured to acquire a digital white light color image 114 and a digital fluorescence image 112. For example, the image processor 170 can be configured to acquire the digital white light color image 114 and the digital fluorescence image 112 from the memory 194 and / or directly from the cameras 110, 111. The memory 194 can be a part of the image processor 170 or can be resident at some other location within the medical fluorescence observation device 100.

[0103] The image processor 170 is further configured to calculate a digital output color image 160 from the digital white light color image 114 and the digital fluorescence image 112. The digital output color image 160 is a color image represented in a color space. The color space of the digital output color image can be different from the color space of either the digital white light color image 114 or the digital fluorescence image 112. However, preferably, the color space of the digital output color image 160 is the same as the color space of the digital white light color image 114 and the digital fluorescence image 112.

[0104] The image processor 170 includes at least two operation modes configured to operate selectively. For example, a user (not shown) can manually select which of the two operation modes to execute. A selector device 165 may be provided to select an operation mode from at least two available operation modes. This selector device 165 may be a mechanical device such as a dial or a switch, or may be a button or other element on a graphical user interface displayed on a display such as displays 132, 182 which will be described in more detail below.

[0105] As shown in FIG. 2, the image processor 170 can include a first operation mode A and a second operation mode B.

[0106] In the first operation mode A, the image processor 170 is preferably configured to generate a digital output color image 160 representing what would be seen through the eyepiece 104. To achieve this, in operation mode A, the image processor 170 may be configured to combine the digital fluorescence color image 112 and the digital white light color image 114 to generate the digital output color image 160. In particular, the image processor 170 may be configured to additionally combine or add the color space coordinates of the digital fluorescence color image 112 and the digital white light color image 114 at each corresponding pixel 150 in the digital fluorescence color image 112 and the digital white light color image 114 in each color band of their color spaces.

[0107] In the second operation mode B, the image processor 170 is configured to first apply the color conversion function 140 to at least one of the digital white light color image 114 and the digital fluorescence color image 112, and then generate a digital output color image 160 from the digital white light color image 114 and the digital fluorescence color image 112. Thus, in the second operation mode B, at least one of the digital white light color image 114 and the digital fluorescence color image 112 is color-converted before being combined. By switching between the first operation mode A and the second operation mode B, the user can switch between a perspective that faithfully renders what is seen through the eyepiece and a perspective that can emphasize different features of the object 106.

[0108] The color conversion function 140 may be any type of function, such as a one-dimensional or n-dimensional interpolation function. However, preferably, the color conversion function 140 is a linear conversion function. In particular, the color conversion function 140 may be a color conversion matrix 142. One dimension of the color conversion matrix 142 may be the sum of the number of color bands in the digital fluorescence color image 112 and the number of color bands in the digital white light color image 114. Another dimension of the color conversion matrix 142 can correspond to the number of color bands in the digital output color image 160 in the other direction of the matrix.

[0109] In another embodiment, the color conversion matrix 142 can have a first dimension corresponding to the dimension of the color space of the digital white light color image 114 or the digital fluorescence color image 112 on the side to which one of the two color conversion matrices is applied, and a second dimension corresponding to the dimension of the color space of the digital output color image 160.

[0110] If all the images 112, 114 are in the RGB color space, the dimensions of the color conversion matrix 142 are preferably 3×3 matrix or 6×3 or 3×6 matrix when the color conversion matrix is configured to operate simultaneously on the digital white light color image and the digital fluorescence color image.

[0111] The second operation mode B itself can include different sub - modes. Each sub - mode can apply a different color conversion function 140 to at least one of the digital white - light color image 114 and the digital fluorescence color image 112 before the digital fluorescence color image 112 is combined with the digital white - light color image 114. In FIG. 2, as an example, three sub - modes B - I, B - II, B - III are shown. Of course, more or fewer than three sub - modes may be provided. The different color conversion functions 140 may be stored in the memory 194.

[0112] For example, the first color conversion function 140a may be applied only to the digital white - light color image 114, and the second color conversion function 140b may be applied only to the digital fluorescence color image 112. The first and second color conversion functions 140a, 140b may be applied independently of each other to separate the color conversion of the digital white - light color image 114 from the color conversion of the digital fluorescence color image 112. For example, only one of the first and second color conversion functions 140a, 140b may be applied to their respective images 112, 114, or both may be applied to their respective images.

[0113] In one embodiment, the user may be able to select between different first color conversion functions 140a and / or different second color conversion functions 140b.

[0114] For example, in operation mode B - I, a first color conversion function 140a configured to convert the colors in the digital white - light color image 114 to colors not located within the second imaging spectrum may be used. In this case, the natural color of the object 106, or a part of the object that was partially filtered and thus not properly recorded in the digital white - light color image 114 because it was distorted by the color - separation assembly 176, can be converted to a corresponding color closer to the natural color by the first color conversion function 140a.

[0115] In another additional or alternative example, the first color conversion function 140a may be used in operation mode B-I or in an additional optional operation mode that converts the colors in the digital white light color image 114 to pseudo colors or different hues. This can emphasize the colors associated with a particular type of tissue. For example, the colors of nerves, arterial blood, and / or venous blood may all be converted to different pseudo colors.

[0116] By using the selector device 165, the user can quickly perform different assignments of pseudo colors for different types of tissue. In one operation mode, the color of arterial blood may be converted to neon red, in another operation mode, the color of venous blood may be converted to neon blue, and in yet another operation mode, the color of nerve tissue may be converted to neon yellow. Other operation modes can use combinations of these modes. If the first color conversion function 140a assigns different hues instead of pseudo colors, the contrast may be emphasized. For example, the first color conversion function 140a can spread the red color in the digital white light color image 114 over a wider range, and thus make smaller changes in blood oxygenation more noticeable to the user.

[0117] In yet another additional or alternative example, the first color conversion function may be used for color balance adjustment and / or white balance adjustment of the digital white light color image 114, for example, in operation mode B-I or in an additional operation mode. In white balance adjustment, the overall mixture of colors is changed, but intermediate colors, i.e., gray, black, and white, are maintained as intermediate colors. In color balance adjustment, the colors including intermediate colors are adjusted to correspond to the intermediate colors.

[0118] In yet another additional or alternative example, the first color conversion function 140a may be used to shift the white point of the digital white light color image 114 to a predetermined position within the color space, for example, in operation mode B-I or an additional operation mode. Adjusting the white point enables adjustments for different illuminations and filter settings of the color separation assembly 176.

[0119] According to one example, at least one of the first color conversion functions 140a described above may be applied to the digital white light color image 114. According to another example, at least two of the first color conversion functions 140a described above may be sequentially applied to the digital white light color image 114. Alternatively or additionally, two or more of the color conversion functions 140a described above may be combined into a single color conversion function 140a.

[0120] The descriptions of the various first color conversion functions 140a above are also applicable mutatis mutandis to the various second color conversion functions 140b, and the only difference between these two functions is that the second color conversion 140b operates on the digital fluorescence image 112 and is configured to operate on different colors.

[0121] In operation mode B-II, for example, one or more color conversion functions 140b may be applied only to the digital fluorescence image, and the color conversion function 140a is not applied.

[0122] In operation mode B-III, for example, one or more color conversion functions 140a may be applied to the digital white light color image, and one or more color conversion functions 140b may be applied to the digital fluorescence color image 112.

[0123] An optional third operation mode C may be envisioned in which a third color conversion function 140c is applied simultaneously to the digital white light color image 114 and the digital fluorescence color image 112, where the application of the color conversion function 140c results directly in the digital output color image 160.

[0124] When another type of phosphor having different excitation wavelengths and / or fluorescence wavelengths is used, different color conversion functions 140a, 140b, 140c need to be provided. This is because the combination of filter settings in the color separation assembly 176, and thus the first and second imaging spectra, need to be adapted to the excitation wavelength and / or fluorescence wavelength.

[0125] Thus, in one embodiment, the medical observation device 100 or the processor 170 may be configured to store a plurality of different sets 143 of the color conversion functions 140a, 140b, and / or 140c. Each different set 143 includes one or more of the color conversion functions 140a, 140b, and / or 140c described above. Each different set 143 corresponds to the use of a different phosphor and thus represents different filter settings of the color separation assembly 176. For example, the first set 143a may be used for ICG as the phosphor, while the second set 143b may be used for 5-ALA / pPiX as the phosphor. Naturally, additional sets may be used to adapt to additional phosphors or combinations of phosphors.

[0126] The medical fluorescence observation device 100 may be adjusted for different phosphors by reconfiguring the color separation assembly 176, for example, by replacing optical elements such as the filters 190 and / or 188 or the dichroic beam splitter 180. The selection of the appropriate set 143 of the color conversion function 140 may be performed automatically, for example, if the medical observation device 100 is configured to automatically detect the settings of the color separation assembly 176.

[0127] Alternatively or additionally, the medical observation device 100 can include a filter setting selector device 168 that enables a user to manually select a set 143 of color conversion functions 140 to be applied to the digital white light color image 114 and / or the digital fluorescence color image. The filter setting selector device 168 can be part of a mechanical and / or graphical user interface. For example, if the filter setting selector device 168 is operated to be in position a, set 143a is selected. At position b, set 143b is selected, and so on.

[0128] The digital output color image 160 may be displayed on a display 132 integrated with the medical fluorescence observation device 100. For example, the display 132 may be integrated with an eyepiece or an ocular lens 104 of the medical fluorescence observation device 100. The display 132 can also display a graphical user interface for operating the medical observation device 100.

[0129] The medical fluorescence observation device 100 can include a direct optical path 134 from the object 106 through the objective lens 174 to the ocular lens 104. In such a case, the display may be a translucent display 132 positioned within the direct optical path 134 or projected into the direct optical path 134. A beam splitter 136 may be provided to split light between the optical ocular lens 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed towards the ocular lens 104.

[0130] Alternatively, the medical fluorescence observation device 100 does not have a direct optical path 134 but can only display an image from the integrated display 132. As a further alternative, the medical fluorescence observation device may not have any display at all.

[0131] The medical fluorescence observation device 100 can include an output interface 172 to which one or more (external) displays 182 may be connected. For this purpose, the output interface 172 can include standardized connectors and data transmission protocols such as USB, HDMI, DVI, DisplayPort, Bluetooth, etc. The external display may be a monitor, 3D goggles, eyeglasses, etc. Any combination of external displays may be connected to the output interface 172. Any of the displays 182 can display a graphical user interface for operating the medical observation device 100.

[0132] The computer 186 or the image processor 170 is connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission line may be wired, wireless, or partially wired and partially wireless. The computer 186 and / or the image processor 170 are not actually integrated into the medical fluorescence observation device 100 but may be located physically separated from the digital imaging system 102. For this purpose, the digital imaging system 102 and the computer 186 and / or the image processor 170 may be connected to a network such as a LAN, WLAN, or WAN to which at least one display 182 is also connected.

[0133] According to a variant, the medical fluorescence observation device 100 may be stereoscopic, but may include only two cameras, one for each stereoscopic channel. In one stereoscopic channel, a fluorescence color camera 111 is used and is configured to selectively record the white light reflectance as well, while in the other stereoscopic channel, a white light color camera 110 is used. In such an arrangement, when fluorescence is not used, a stereoscopic white light color image is provided, and when fluorescence is used, a monocular white light color image and a monocular fluorescence color image are provided. The above description and the following description are equally applicable to this configuration.

[0134] FIG. 2 shows an example of the first operation mode A.

[0135] Reference numeral 200 is a quantitative example of the first imaging spectrum 202 recorded by the digital white light camera 110 and / or represented in the digital white light color image 114. The intensity I over the wavelength / color λ is shown normalized. The first imaging spectrum 202 preferably extends over at least the visible spectrum 212.

[0136] As a mere example, the color space in which the first imaging spectrum 202 is recorded is an RGB color space having three primary colors or color bands 204, 206 and 208. One primary color 204 is blue, another primary color 206 is green, and the third primary color 208 is red. The sensitivities of the sensors of the white light color camera 110 in the different primary colors 204, 206, 208 are adjusted so that the sensitivity over the entire visible spectrum 212 is as constant as possible.

[0137] If a color space other than RGB is used, the number, position and / or width of the color bands may be different. Otherwise, there is no fundamental difference from the RGB color space.

[0138] The first imaging spectrum 202 does not include the fluorescence excitation light and fluorescence emission spectra of at least one phosphor 116, 118. Thus, the first imaging spectrum 202 may include at least one blocking band 210 where the fluorescence coincides with the fluorescence emission of at least one phosphor whose fluorescence is recorded by the fluorescence color camera 111. This blocking band 210 is generated, for example, by the white light filter 188. The number, width and / or position of the blocking band 210 depends on the number and type of phosphors observed in the object 106.

[0139] Reference numeral 220 shows, respectively, a second imaging spectrum 222 recorded by the fluorescence color camera 111 and / or represented in the digital fluorescence color image 112. As a mere example, the color space in which the second imaging spectrum 222 is recorded is also the RGB color space. The sensitivities of the sensors of the fluorescence color camera 111 in different primary colors 204, 206, 208 are adjusted so that the sensitivities over the entire visible spectrum 212 are as constant as possible as a result.

[0140] The spectra 202, 222 do not have to be recorded in the same color space, although this is preferred.

[0141] The second imaging spectrum 222 can include one or more passbands 224. The number, position and / or width of the passbands depend on the number and type of phosphors used. At least one passband 224 preferably corresponds to at least one blocking band 210. At least one passband is generated, for example, by the fluorescence filter 190.

[0142] The first imaging spectrum 202 and the second imaging spectrum 222 are complementary to each other. They preferably complement each other so as to cover the whole or most of the visible spectrum 212.

[0143] Each passband 224 of the second imaging spectrum 222 preferably overlaps with the fluorescence emission spectra 226, 228 of the phosphors 116, 118 in which fluorescence is recorded and can overlap with one or more primary colors 204, 206, 208 of the color spectrum. For example, the fluorescence emission spectrum 226 of one phosphor 116 can overlap with all three primary colors 204, 206, 208. The sensor of the fluorescence camera 111 that records the color band 204 records only a small portion 230 of the fluorescence emission spectrum 226. The sensor of the fluorescence camera 111 that records the color band 206 records most of the fluorescence emission spectrum 226, but the sensor of the fluorescence camera 111 that records the color band 208 also records only a small portion 236 of the fluorescence emission spectrum 226.

[0144] The fluorescence spectrum 228, if present in a particular case, is recorded only by the fluorescence camera 111 that records the color band 208, in contrast.

[0145] In its simplest form, the combination 240 of the digital white light color image 114 and the digital fluorescence color image 112 is an additional combination. For example, the color space coordinates in each color band of the digital white light color image 114 and the fluorescence color image 112 are added at each corresponding pixel 150. The corresponding pixel 150 may be a pixel 150 that is located at the same position in both images 112, 114 when the images 112, 114 are aligned and of the same size. If both images 112, 114 are RGB images, the color space coordinates in the RGB color bands of the pixel 150 in the digital white light color image are {R1, G1, B1}, and the color space coordinates in the RGB color bands of the corresponding pixel 150 in the digital fluorescence color image are {R2, G2, B2}, then the color space coordinates in the RGB color bands of the corresponding pixel 150 in the digital output color image may be calculated as {R1 + R2, G1 + G2, B1 + B2}.

[0146] The reference numeral 250 indicates a qualitative representation of the spectrum 252 of the digital output image 160. In short, the spectrum 252 of the digital output image 160 is the addition of the first imaging spectrum 202 and the second imaging spectrum 222.

[0147] Alternatively, the individual spectra 202 and 222 are processed as a single composite multi - spectral spectrum having numerical values of color bands corresponding to the sum of the color bands in the first imaging spectrum 202 and the second imaging spectrum 222. In other words, the digital fluorescence color image 112 and the digital white light color image 114 are co - processed as a single (virtual) multi - spectral image.

[0148] This corresponds to operation mode C (Figure 1), in which color conversion functions 140, 140c are applied to both the digital white light color image 114 and the digital fluorescence color image 112 to obtain a digital output color image 160. The spectrum of this digital output color image 160 is indicated by reference numeral 250 in Figure 2.

[0149] Referring to Figure 3, this will be described below. For simplicity, Figure 3 shows the spectrum of Figure 2 in the RGB color space, with only a single passband 224 or stopband 210 shown.

[0150] The digital white light color image 114 recorded in the first imaging spectrum 202 consists of signals R1, G1, and B1, each representing the intensity I of its respective color band. Due to the stopband 210, the signal G1 results from two separated wavelength bands. However, the sensor that records G1 cannot distinguish between these two wavelength bands.

[0151] The digital fluorescence color image 112 recorded in the second imaging spectrum 222 includes signals R2, G2, B2 in each color band.

[0152] Since the first and second imaging spectra 202, 222 are complementary to each other, each color band is divided into two signals, each signal arriving from a different camera 110, 111 and representing or equivalent to a sub-band of the color band. Color band 204 is divided into R1 and R2, respectively. Color band 206 is divided into G1 and G2, respectively, and color band 208 is divided into B1 and B2, respectively. Preferably, there is no overlap between the various signals within the color band.

[0153] Therefore, even when the first and second imaging spectra 202, 222 are recorded in the same color space using the same type of color camera, a multi-spectral image of the object 106 results, which in this case consists of six color bands R1, R2, G1, G2, B1, B2. This is schematically shown by reference numeral 300. It should be noted that this is independent of whether the fluorescence camera actually records fluorescence emission or reflectance.

[0154] In other words, at least one color band 204, 206, 208 of the color space of the at least digital output color image 160 is subdivided into two sub-bands R1, R2 and / or G1, G2 and / or B1, B2, where one of the two sub-bands is included in the digital fluorescence color image 112 and the other of the two sub-bands is included in the digital white light color image 114. Preferably, the two sub-bands within a color band do not overlap. They are preferably complementary. Most preferably, together they at least substantially complete their respective color bands. A sub-band can include or consist of two separated spectral bands. Each sub-band may itself be regarded as a color band.

[0155] The color conversion function 140 reflects the subdivision of the color band into sub-bands determined by the stop band 210 and the pass band 224. Since the width and / or position of the sub-band determines how much light is collected by the respective cameras 110, 111 in that sub-band, the color conversion function 140c needs to be adjusted for each different filter setting of the color separation assembly 176.

[0156] Accordingly, the digital white light image 114 and the digital fluorescence image 112 may be processed together as an image consisting of the sum of the color bands in the two images 112 and 114. This provides improved color resolution. The color space coordinates (set) {R1, R2, G1, G2, B1, B2} of this multispectral image correspond to the union (set) of the color space coordinates set {R1, G1, B1} of the digital white light color image 114 and the color space coordinates set {R2, G2, B2} of the digital fluorescence color image 112.

[0157] The color conversion function 140c may be applied to this combined set in the form of a linear transformation using the color conversion matrix 142 to generate the digital output color image 160.

[0158] When the digital output color image 160 is represented in the RGB color space, a 6×3 or 3×6 color conversion matrix may be applied to the combined digital fluorescence image 112 and white light image 114 or their component signals R2, G2, B2, R1, G1, B1 respectively to achieve the RGB signals R * , G * , B * . The matrix coefficients C11, C12, …, C63 may be determined experimentally.

[0159] Next, operation mode B will be described.

[0160] FIG. 4 shows a schematic reflection spectrum 400 of the biological tissue of the object 106, where the term tissue includes both fluid tissue and solid tissue. For example, the reflection spectrum 400 may correspond to the reflection spectrum of blood containing oxygen.

[0161] The spectral sensitivities R, G, B of the R, G, B sensors of the sample RGB color camera are also shown in FIG. 4.

[0162] In human perception, tissues such as oxygenated blood are perceived as having natural colors, i.e., red, under white light illumination such as CIE light sources. Therefore, although the reflection spectrum 400 spreads widely over the visible light spectrum, it corresponds to the natural color. In the first imaging spectrum 202 where the digital white light color image 114 is recorded, the wavelength λ in the cutoff band 210 is not recorded. Therefore, the reflection spectrum 400 represented in the digital white light color image 114 does not faithfully represent the natural color of the tissue.

[0163] The same applies to the second imaging spectrum 222 and the digital fluorescence color image 112, where the reflection spectrum 400 also causes distortion.

[0164] Figure 5 shows a schematic diagram of the CIE1932 color space or any other tristimulus or uniform color space, where color 500 is shown. Color 500 can represent the natural color perceived by a standard observer when viewing a tissue having a reflection spectrum 400 (Figure 4) illuminated by, for example, CIE light source A, B, or C or another standard light source.

[0165] Since the wavelengths within the cutoff band 210 are not recorded, color 500 is represented in the digital white light color image 114 as (recorded) color 502. To represent color 500 more faithfully, the recorded color 502 is converted to color 500 by the color conversion function 140. This color conversion function 140 may be the first or second color conversion function 140a, 140b. For example, the color conversion may be a shift or translation in the LMS color space, uniform color space, tristimulus color space, or even the RGB color space.

[0166] The color conversion function 140 may be determined by color calibration using known colors, such as color cards, known filter settings, and known lighting conditions. Color 500 may be a wavelength within the cutoff band 210 and thus not present within the first imaging spectrum 202. Along with the shift from 502 to 500, the entire color space may be shifted.

[0167] If the color 500 to which color 502 is converted is not a natural color but a pseudo-color or a color that is close to a natural color but is visually different, the type of tissue associated with this color can be better offset from other tissues or fluorescent colors that are spatially or perceptually close to color 500, and in some applications, this may be beneficial to highlight this type of tissue to the eyes of a skilled person. For example, color 500 in the case of oxygenated blood may be a "hyper-real" red that is brighter than the natural color of oxygenated blood, or a pseudo-color of red such as neon red. Of course, this can be similarly implemented for any other color or tissue.

[0168] For the blocking band 210, the position of the natural white point 504 may be shifted to the recorded white point 506. Thus, another color conversion function 140, such as color conversion function 140d, can shift the recorded white point 506 to or near the natural white point 504. In this regard, the white point is treated in the same way as any color.

[0169] A shift of a single color can correspond to a shift of the entire color space, i.e., all colors are shifted by the same amount.

[0170] A shift of the entire color space by the same amount and in the same direction may be used for white balance adjustment or color balance adjustment of the digital white light color image 114.

[0171] Alternatively, different colors may be converted or shifted so as to be different in relation to at least one of the amount and direction of the shift. For example, the recorded color 508 of blood that does not contain oxygen may be further shifted, for example, to a color 510 in the blue range. Such a shift from color 508 to color 510 can be made different from the shift from color 502 to color 500. Thus, instead of converting the entire color space, the color conversion function 140 can be configured such that a color-dependent conversion is generated. Different color conversion functions 140 may be applied for different (recorded) colors 502, 504, 508.

[0172] Another example of the color conversion function 140 can expand the area 512 in the color space to a wider area 514. Within this area 514, since the distance between colors becomes longer than within the area 512, the difference in colors becomes more prominent. The application of such a color conversion function may be color-dependent. In this case, only the predetermined color area 512 is expanded. Since this color conversion function 140 can include a color shift, not only is the area 514 expanded, but it also moves to different areas of the color space.

[0173] Any of the above color conversion functions 140 can only convert a selected subset of the color appearance parameters of the color. These color appearance parameters are hue, chroma, saturation, lightness, and luminance. For example, when converting color 502 to color 500, only the hue may be changed, or only the saturation and lightness may be changed by the color conversion function 140.

[0174] Any combination of the above color conversion functions 140a may be applied. Although the expression "image" is used in the above description, it should be understood that the color conversion function 140 is applied to an image at the pixel level, that is, to each pixel 150 of the digital white light color image 114. The color of the pixel 150 can determine which color conversion function 140 or which combination of color conversion functions 140 is applied. This can be automatically performed by the processor 170.

[0175] FIG. 6 shows a schematic diagram of an imaging method that may be implemented as a computer-implemented method executed, for example, on an image processor 170.

[0176] In optional step 600, for example using a white light color camera 110, a digital white light color image 114 is recorded. In optional step 602, for example using a fluorescence camera 111, a digital fluorescence color image 112 is recorded. Steps 600 and 602 are optional because these images 112, 114 may also be retrieved from memory. As described above, these cameras 110, 111 should be synchronized with respect to exposure time and locked to each other with respect to gain, i.e., maintained at the same gain ratio. Gamma may be set to a fixed value and all automatic color adjustments are preferably turned off.

[0177] In optional step 604, each of the images 112, 114 may be demosaicked.

[0178] In optional step 606, one or both of the images 112 or 114 may be aligned so that the same image features are geometrically identical within each of the images 112, 114 with respect to size and orientation.

[0179] In one operating mode, for example operating mode B, but optionally also in operating mode C, color conversion is performed in color conversion step 608. Step 608 is omitted in operating mode A. In step 608, a color conversion function 140 as described above or a combination of color conversion functions 140 is applied to at least one of the digital white light color image 114 and the digital fluorescence image 112.

[0180] In the image combining step 610, the digital white light color image 114 and the digital fluorescence color image 112 are combined for generating the digital output color image 160 in all operation modes A, B, and C. As described above, the images 112, 114 may simply be added for combination, as shown, for example, in FIG. 2. In operation mode C, as shown in FIG. 3, for generating the digital output color 160, in step 419, the color conversion function 140 may be applied to both the digital white light color image 114 and the digital fluorescence image 112.

[0181] In step 612, post-processing may be performed. For example, the digital output color image 160 may be equalized, its contrast may be enhanced, and / or a color space conversion such as from RGB to sRGB and / or gamma correction may be performed. It is important to note that the color conversion in steps 608 and / or 610 is not gamma correction.

[0182] In step 614, the digital output image 160 is displayed.

[0183] Some embodiments relate to a fluorescence microscope including a microscope, in particular, a system as described in connection with one or more of FIGS. 1 - 6. Alternatively, the microscope may be part of a system as described in connection with one or more of FIGS. 1 - 6, or may be connected to a system as described in connection with one or more of FIGS. 1 - 6.

[0184] FIG. 7 shows a schematic diagram of a system 700 configured to implement the method described herein. The system 700 includes a microscope 710 and a computer system 720. The microscope 710 is configured to image and is connected to the computer system 720. The computer system 720 is configured to implement at least a portion of the method described herein. The computer system 720 may be configured to execute a machine learning algorithm. The computer system 720 and the microscope 710 may be separate entities or may be integrated within a single common housing. The computer system 720 may be part of the central processing system of the microscope 710 and / or the computer system 720 may be part of a sub-component of the microscope 710, such as a sensor, actuator, camera, or illumination unit of the microscope 710.

[0185] The computer system 720 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system distributed across various locations such as local clients and / or one or more remote server farms and / or data centers, comprising one or more processors and one or more storage devices). The computer system 720 may include any circuit or combination of circuits. In one embodiment, the computer system 720 may include one or more processors, which can be of any type. As used herein, a processor may mean any kind of computing circuit, such as, for example, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, but is not limited thereto. Other types of circuits that may be included in the computer system 720 may be custom circuits, application specific integrated circuits (ASICs), etc., and may be, for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 720 may include one or more storage devices that may include one or more storage elements suitable for specific applications, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives for handling removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 720 may include a display device, one or more speakers and a keyboard and / or a mouse, a trackball, a touch screen, a controller that may include a voice recognition device, or any other device that enables a user of the system to input information into the computer system 720 and receive information from the computer system 720.

[0186] Some or all of the steps may be performed by a hardware device (or using a hardware device) such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the critically important steps may be performed by such a device.

[0187] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is executable by a non-transitory recording medium, which is a digital recording medium, such as an electronic-readable control signal stored therein that cooperates (or is capable of cooperating) with a programmable computer system to implement each method, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory. Thus, the digital recording medium may be computer-readable.

[0188] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system such that any of the methods described herein are implemented.

[0189] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which code is operative to implement any of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0190] Another embodiment includes a computer program stored on a machine-readable carrier for implementing any of the methods described herein.

[0191] Thus, in other words, embodiments of the present invention are computer programs having program code for implementing any of the methods described herein when the computer program is executed on a computer.

[0192] Thus, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) including a stored computer program for implementing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein including a processor and a recording medium.

[0193] Thus, another embodiment of the present invention is a data stream or signal sequence representing a computer program for implementing any of the methods described herein. The data stream or signal sequence may be configured to be transferred, for example, via a data communication connection such as the Internet.

[0194] Another embodiment includes processing means, for example, a computer or programmable logic device configured or adapted to implement any of the methods described herein.

[0195] Another embodiment includes a computer having an installed computer program for implementing any of the methods described herein.

[0196] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0197] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to implement any of the methods described herein. Generally, and advantageously, the methods are implemented by any hardware device.

Explanation of Reference Numerals

[0198] 100 Medical fluorescence observation device 101L Stereoscopic subassembly for left channel 101R Stereoscopic subassembly for right channel 102 Digital imaging system 104 Eyepiece 106 Object to be inspected 107 Biological tissue 110 White light color camera 111 Fluorescence color camera 112 Digital fluorescence color image 114 Digital white light color image 116 Phosphor 118 Second phosphor 132 Integrated / internal display 134 Direct optical path 136 Beam splitter 140 Color conversion function 140a~140d Different color conversion functions 142 Color conversion matrix 143 Set of color conversion functions 143a~143c Different sets of color conversion functions 150 Pixel 160 Digital output color image 165 Selector device 168 Filter setting selector 170 Image processor 172 Output interface 174 Objective lens 176 Color separation assembly 178 Lighting assembly 179 Lighting filter 180 Lighting beam splitter 182 Display 184 Field of view 186 Computer 188 White light filter 190 Fluorescent filter 192 Dichroic beam splitter 194 Memory 196 Data transmission line 200 Spectrum 202 First imaging spectrum 204 Primary color or color band 206 Primary color or color band 208 Primary color or color band 210 Blocking band 212 Visible spectrum or visible light range 220 Spectrum 222 Second imaging spectrum 224 Passband 226 Fluorescent emission spectrum of phosphor 228 Fluorescent emission spectrum of another phosphor 230 Part of the fluorescent emission spectrum in color band 204 232 Part of the fluorescent emission spectrum in color band 206 Part of the fluorescence emission spectrum in the 236 color band 208 240 Image combination 250 Spectrum 300 Multispectral images from digital white light and fluorescence images 400 Reflection spectrum of tissue 500 Natural color 502 Recorded color 504 Natural white point 506 Recorded white point 508 Recorded color 510 Converted color 512 Recorded color area 514 Target color area 600 Recording of digital white light color image 602 Recording of digital fluorescence color image 604 Demosaicing 606 Alignment 608 Color conversion step 610 Image combination step 612 Post-processing 614 Display 700 System 710 Microscope 720 Computer system λ Wavelength I Intensity A First operation mode of the processor B Second operation mode of the processor C11, C12, …, C63 Coefficients of the color conversion matrix Example of RGB color space R, G, B, R1, G1, B1, R2, G2, B2, R * , G * , B * Color space coordinates

Claims

1. An image processor (170) for a medical fluorescence observation device (100) such as a fluorescence microscope or fluorescence endoscope, The aforementioned image processor (170) - A digital white light color image (114) of the object (106) recorded in the first imaging spectrum (202) is acquired. - The system is configured to acquire a digital fluorescence color image (112) of the object (106) recorded in the second imaging spectrum (222), The second imaging spectrum (222) overlaps with the fluorescence emission spectrum (226) of at least one phosphor (116), and unlike the first imaging spectrum (202), both the first and second imaging spectra (202, 222) overlap with the visible spectrum (212). The image processor (170) is configured to be operated selectively in one of a first operation mode (A) and a second operation mode (B), the first operation mode including an image merging step (610), and the second operation mode including a color conversion step (608) and an image merging step (610). The image processor (170) is configured to generate a digital output color image (160) by combining the digital white light color image (114) and the digital fluorescent color image (112) in the image combining step (610). In the color conversion step (608), the image processor (170) - A first color conversion function (140a) applied to the digital white light color image (114), - A second color conversion function (140b) applied to the digital fluorescent color image (112), The group is configured to apply at least one color conversion function (140, 140a, 140b) from the group that includes the following: Image processor (170).

2. The first imaging spectrum (202) and the second imaging spectrum (222) are complementary to each other. The image processor (170) according to claim 1.

3. The colors of the digital white light color image (114), the digital fluorescent color image (112), and the digital output color image (160) are represented within the same color space. The image processor (170) according to claim 1 or 2.

4. The image processor (170) is configured to additively combine the color space coordinates (R, G, B) of corresponding pixels (150) of the digital white light color image (114) and the digital fluorescent color image (112) in the image combining step (610). The image processor (170) according to claim 1.

5. The first color conversion function (140a) is configured to convert the colors (502, 504, 508) recorded in the digital white light color image (114) to colors (500, 506, 510) that are not located in the first imaging spectrum (202), and / or The second color conversion function (140b) is configured to convert the colors (502, 504, 508) in the digital fluorescence color image (112) to colors (500, 506, 510) that are not located in the second imaging spectrum (222). The image processor (170) according to claim 1.

6. The first color conversion function (140a) is configured to color balance the digital white light color image (114). The image processor (170) according to claim 1.

7. The first color conversion function (140a) is configured to shift the recorded colors (502, 504, 508) of the digital white light color image (114) to predetermined colors (500, 506, 510) in the color space, and / or The second color conversion function (140b) is configured to shift the colors (502, 504, 508) of the digital fluorescent color image (112) to predetermined colors (500, 506, 510) in the color space. The image processor (170) according to claim 1.

8. The image processor (170) is configured to shift different recorded colors (502, 504, 508) by different amounts and / or directions in a predetermined color space. The image processor (170) according to claim 7.

9. The first and / or second color conversion functions (140a, 140b) are configured to convert a region (512) of recorded colors (502, 504, 508) to a larger region (514) in a predetermined color space. The image processor (170) according to claim 1.

10. A medical fluorescence observation device (100) such as a fluorescence microscope or fluorescence endoscope, wherein the medical fluorescence observation device (100) is The image processor (170) according to claim 1, A fluorescent color camera (111) configured to record a digital fluorescent color image (112), A white light color camera (110) configured to record a digital white light color image (114), A medical fluorescence observation device (100) including [a specific component].

11. The fluorescent color camera (111) has a longer integration time than the white light color camera (110). A medical fluorescence observation device (100) according to claim 10.

12. The aforementioned medical fluorescence observation device (100) is a surgical fluorescence microscope. A medical fluorescence observation device (100) according to claim 10 or 11.

13. A computer-based image processing method for a fluorescence observation device (100) such as a fluorescence microscope or fluorescence endoscope, wherein the computer-based image processing method comprises the following steps, namely: - A step of acquiring a digital white light color image (114) of the object (106) recorded in the first imaging spectrum (202), - A step of acquiring a digital fluorescence color image (112) of the object (106) recorded in the second imaging spectrum (222), Includes, The second imaging spectrum (222) overlaps with the fluorescence emission spectrum (226) of at least one phosphor (116), and unlike the first imaging spectrum (202), both the first and second imaging spectra (202, 222) overlap with the visible spectrum (212). The computer-implemented image processing method is configured to be executed selectively in either a first operation mode or a second operation mode. In the first operation mode (A), the image merging step (610) is performed, and in the second operation mode (B), the color conversion step (608) and the image merging step (610) are performed. In the image merging step (610), a digital output color image (160) is generated by merging the digital white light color image (114) and the digital fluorescent color image (112). In the aforementioned color conversion step (608), - A first color conversion function (140a) applied to the digital white light color image (114), - A second color conversion function (140b) applied to the digital fluorescent color image (112), From the group including, at least one color conversion function (140, 140a, 140b) is applied. Computer-based image processing method.

14. A computer program product or computer-readable medium comprising, when the program is executed by a computer, an instruction causing the computer to perform the method described in claim 13.

15. A method for operating a medical fluorescence observation device (100), such as a fluorescence microscope or fluorescence endoscope, wherein the method comprises the following steps: - A step of recording a digital fluorescence color image (112) in a second imaging spectrum (222) using a fluorescence color camera (111), - A step of recording a digital white light color image (114) in a first imaging spectrum (202) using a white light color camera (110), A method that includes this.

Images