Method using two color images and color cameras for fluorescence and white light, processor and medical fluorescence observation device

Abstract

The present application relates to an image processor (170) and a computer-implemented image processing method for generating a digitally output color image (160) of an object (106) using a medical fluorescent viewing device (100), such as a microscope or endoscope. The present application also relates to a medical fluoroscopy device (100) comprising such an image processor (170) and to a method of operating such a device comprising the above computer-implemented method. Two selectable operating modes (A, B) are provided. In a first mode of operation (A), a digital output color image (160) is generated by simply combining a digital white light color image (114) and a digital fluorescent image (112), such that the digital output color image (160) represents a view seen through an eyepiece. In a second mode of operation (B), a color conversion function (140, 140a, 140b) is applied to at least one of the digital white color image (112) and the digital fluorescent color image (114) to highlight display characteristics and / or compensate for color distortions introduced by the optical filter.

Description

Technical Field

[0001] The claimed subject matter relates to a computer-implemented image processing method and to an image processor for generating a digital output color image of an object using a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope. The claimed subject matter also relates to a method for operating such a medical fluorescence observation device and to the medical fluorescence observation device itself. Background Art

[0002] In existing medical fluorescence observation equipment, fluorescence and anatomical images are recorded by a single camera. This setup can be used in fluorescence imaging of pPIX after administration of 5-ALA to reveal tumors. The choice of optical filters is such that the camera captures a small portion of the fluorescence in the red wavelength and the excitation light in the blue wavelength. The excitation light reveals some of the tissue anatomy, while the fluorescence indicates the tumor. The optical filters used today have been standardized for pPIX imaging.

[0003] Although this type of fluorescence imaging provides important assistance in neurosurgical oncology, it still has serious drawbacks: the blue excitation light used to visualize the tissue anatomy provides only poor visualization, as its intensity is very low and the anatomy is only shown in a monochromatic blue shade. In particular, bleeding is very difficult or even impossible to see under blue light. Therefore, this setup is suboptimal for real-time surgical guidance.

[0004] Other medical fluorescence observation equipment uses a grayscale camera to record fluorescence and a color camera to record white light images. In this setup, the fluorescence image and the white light image are mixed digitally, and the fluorescence is labeled with pseudo color. Here, the disadvantage is that pseudo color cannot accurately represent the color of fluorescence perceived by the human eye.

[0005] In view of the above, there is a need to provide an apparatus and method that provides an improved and more natural rendering of anatomical images and fluorescence. At the same time, the user should be able to compare the improved rendering with the actual scene. Summary of the invention

[0006] This need is addressed by providing an image processor for a medical fluorescence observation device (such as a fluorescence microscope or a fluorescence endoscope), wherein the processor is configured to retrieve a digital white light color image of an object recorded in a first imaged spectrum, and to retrieve a digital fluorescence color image of an object recorded in a second imaged spectrum, the second imaged spectrum overlapping with a fluorescence emission spectrum of at least one fluorophore, the second imaged spectrum being different from the first imaged spectrum, and both the first and second imaged spectra overlapping with the visible spectrum; wherein the image processor is configured to be selectively operated in one of a first operating mode and a second operating mode, the first operating mode comprising an image combination step and the second operating mode comprising a color conversion step and an image combination step; wherein the image processor is configured to generate a digital output color image by combining the digital white light color image and the digital fluorescence color image in the image combination step; wherein the image processor is configured to apply in the color conversion step at least one color conversion function from the group consisting of: a first color conversion function applied to the digital white light color image, and a second color conversion function applied to the digital fluorescence color image.

[0007] The need is also addressed by a computer-implemented image processing method for a medical fluorescence observation device (such as a fluorescence microscope or a fluorescence endoscope), the computer-implemented image processing method comprising the following steps: retrieving a digital white light color image of an object recorded in a first imaged spectrum and retrieving a digital fluorescence color image of an object recorded in a second imaged spectrum, the second imaged spectrum overlapping with the fluorescence emission spectrum of at least one fluorophore, the second imaged spectrum being different from the first imaged spectrum, and both the first and second imaged spectra overlapping with the visible spectrum; wherein the method is configured to be selectively performed in one of a first operating mode and a second operating mode; wherein in the first operating mode, an image combination step is performed, and in the second operating mode, a color conversion step and an image combination step are performed; wherein, in the image combination step, a digital output color image is generated by combining the digital white light color image and the digital fluorescence color image, and, in the color conversion step, at least one color conversion function from the group consisting of: a first color conversion function applied to the digital white light color image, and a second color conversion function applied to the digital fluorescence color image is applied.

[0008] By providing two operating modes that can be selected, it is possible to provide in a first operating mode a digital output color image corresponding to the image seen through the eyepiece of the medical viewing device.

[0009] Digital white light color images typically record the anatomy of the surgical scene. Fluorescence color images may record the fluorescence of one or more fluorophores, but can also be used to provide additional information about the anatomy if separately excited fluorescence is not used.

[0010] By providing two color conversion functions that operate separately on the digital white light color image and the digital fluorescent color image, it is possible to decouple the color conversion of the digital white light image from the color conversion applied to the digital fluorescent color image. This allows each of the two images to be independently optimized for viewing before combining them into a digital output color image.

[0011] In addition, by using a color camera for both the fluorescence and white light images, a more color accurate, true color output image can be provided.

[0012] The terms "image processor", "processor" and "data processing device" are used synonymously.

[0013] The above solution can be further improved by the following features, which can be added and combined independently of each other, and each feature has its own beneficial technical effect. Each of the following features can be used to improve one of the above methods, and / or can be used to improve one of the above devices, regardless of whether a specific feature is mentioned only for the method or only for the device.

[0014] The computer-implemented image processing method may include a step of demosaicing at least one of the digital white light color image and the digital fluorescent color image. By demosaicing, color artifacts may be avoided.

[0015] The computer-implemented image processing method may further include the step of normalizing at least one of the digital white light color image and the digital fluorescent color image. Normalization facilitates joint processing of the digital white light color image and the digital fluorescent color image.

[0016] The computer-implemented image processing method may also include the step of registering at least one of the digital white light color image and the digital fluorescent color image. By registering, image features present in both the digital white light color image and the digital fluorescent color image are represented by the same size and orientation in each image. After registering, the digital white light color image and the digital fluorescent color image match each other.

[0017] The image processor may be configured to perform any of the above processing steps.

[0018] In one embodiment, the fluorescence color camera can record another digital white light color image that can supplement the digital white light color image recorded by the white light color camera. Thus, the fluorescence color camera is not limited to recording only fluorescence images. However, if the hardware is not changed (e.g., by exchanging one or more filters), then the fluorescence camera can be limited to recording only images in the second imaged spectrum.

[0019] According to another advantageous embodiment of the computer-implemented image processing method and / or the image processor, the first imaged spectrum and the second imaged spectrum are complementary to each other. In this configuration, crosstalk between fluorescence and white light is avoided and an exact reproduction of the fluorescence of at least one fluorophore on the one hand and the reflectivity of the object on the other hand is maintained. In addition, by having complementary spectra in the digital fluorescence color image and the digital white light color image, respectively, joint processing of the two images to form a multispectral image is facilitated.

[0020] The digital white light color image may be represented in a first color space using at least three first color bands.

[0021] The digital fluorescent color image may be represented in a second color space comprising at least three second color bands. The first and second color spaces are preferably identical, so that the first and second color bands are also identical. However, this need not be the case. The first and second color spaces may also be different.

[0022] The digital output color image may be generated in a third color space comprising at least three third color bands.The third color space may be the same color space as the first and / or second color space, or may be a different color space.

[0023] Any of the first, second and / or third color spaces may be an RGB color space, including three color bands R, G, B; or any other color space, such as a multispectral color space or a hyperspectral color space, i.e., using more than three color bands. Each color band is represented by a color space coordinate. The same color has different color space coordinates in different color spaces. Each pixel of the digital white light color image, the digital fluorescent color image, and the digital output color image includes a color space coordinate representing the color of the corresponding pixel.

[0024] According to one embodiment of the computer-implemented image processing method and / or image processor, the first color band and the second color band are mapped or converted to a third color band of the digital output color image using a color conversion function. The color conversion function may be a non-linear function, but is preferably a linear transformation function.

[0025] The linear transformation may include a color conversion matrix or other type of linear function. To map the first and second color bands to the third color band, the color conversion matrix may be applied to the first color band and the second color band. Preferably, each color band in the first color band and each color band in the second color band are input into the linear transformation as separate color bands. In particular, each color band in the first color band and each color band in the second color band may be input into the linear transformation simultaneously.

[0026] The color conversion matrix or linear function may reside in the memory of an image processor or medical fluorescence viewing device.

[0027] The color conversion matrix may have a dimension of a number X in one direction multiplied by a number Y in another direction. The number X may be the sum of the number of first color bands (i.e., the number of color bands in the first color band) and the number of second color bands (i.e., the number of color bands in the second color band). The number Y may be the number of color bands in the third color space. For example, if the first, second, and third color spaces are each RGB color spaces, then the color conversion matrix has a dimension of 6×3 or 3×6 or 3×3. In particular, one dimension of the color conversion matrix may correspond to the sum of the dimensions of the color spaces of the digital white light color image and the digital fluorescent color image. Another dimension of the color conversion matrix may correspond to the dimension of the color space of the digital output color image.

[0028] Thus, the third color band (ie the color band of the digital output color image) may be produced by a linear combination (in particular addition) of the color band of the digital white light color image and the color band of the digital fluorescent color image.

[0029] According to one aspect, the digital fluorescent color image and the digital white light color image are jointly processed into a multispectral image, the color bands of the multispectral image being formed or composed of the complementary color bands of the digital fluorescent color image and the digital white light color image. The color space coordinates of this multispectral image may correspond to the union of the set of color space coordinates of the digital white light color image and the set of color space coordinates of the digital fluorescent color image. By keeping the color space coordinates of the digital white light color image and the digital fluorescent color image separate but jointly processing them, the multispectral image may be physically created in the image processor, or only logically created.

[0030] Processing as a multispectral image results in improved color reproduction, especially if the fluorescence image represents reflected white light that has been recorded in the second imaged spectrum, and can therefore supplement the white light information in the white light color image. Using this approach, the white light information is contained in the color bands of the digital fluorescence color image, so the color granularity is finer than that provided by the digital white light color image alone. If the digital fluorescence color image represents fluorescence emission, then processing the digital white light color image and the digital fluorescence color image as a multispectral image represents a true representation of the object under reflected white light and fluorescence.

[0031] According to another advantageous embodiment, the first imaged spectrum may comprise a first sub-band in a color band of a color space and the second imaged spectrum may comprise a second sub-band in this color band, wherein the first and second sub-bands are preferably complementary to each other. The color space may be the color space of a digital white light color image, the color space of a digital fluorescent color image or preferably the color space of a digital output color image. The color bands with such subdivision facilitate a linear transformation of the color bands of the fluorescent and white light color images to the color bands of the digital output color image. It is preferred that the sub-bands of a color band complete the corresponding color band together. This ensures that useful spectral information is not lost.

[0032] Of course, there may be first and second sub-bands in more than one color band of the corresponding color space. The more sub-bands used in different color bands, the more accurate the spectral information recorded in the corresponding white light or fluorescent color image. The sub-bands in a color band may themselves be divided into a plurality of different wavelength bands separated from each other.

[0033] Preferably, the second imaged spectrum includes IR wavelengths. This allows for capturing fluorescence emissions in the near infrared, or, if digital fluorescence color images are not used to capture fluorescence, can add additional information about the anatomy.

[0034] According to one aspect, the first and / or second color conversion function is configured to shift the (recorded) color to a (different) color in a predetermined color space. The predetermined color space may be the color space of the digital white light color image and the digital fluorescent image, or a different color space. The shift is predetermined by the color conversion function with respect to at least one of an amount and a direction of the shift in the color space, wherein the color is represented by a color space coordinate.

[0035] For example, in the first and second imaged spectra, the natural color of oxygenated or deoxygenated blood may only be captured incompletely. This is because the reflection spectrum of blood may contain wavelengths that are not recorded in the first or second imaged spectrum, respectively. Therefore, if blood is recorded in a digital white light color image or a digital fluorescent image, its color may look unnatural, making it more difficult to identify blood vessels. By using the first and / or second color conversion function, the (recorded) colors may be shifted to more natural colors in the corresponding digital white light or fluorescent color image.

[0036] In another example, the recorded colors of oxygenated blood and deoxygenated blood may each be shifted differently in a predetermined color space, such as the color space of a digital white light color image or a digital fluorescent color image. Again, both colors are incompletely represented in the first and / or second imaged spectrum. The recorded color of deoxygenated blood may be shifted further toward blue, while the color of oxygenated blood may be shifted further toward red. This may require different amounts and / or directions of shift for the two recorded colors; a color conversion function may shift one color differently from another color in color space with respect to at least one of the amount and direction of the shift.

[0037] The above examples are of course applicable to any other type of tissue, such as liquid tissue (eg, lymph), or solid tissue (eg, bone, muscle and / or nerve tissue).

[0038] The converted color can correspond to a natural color, a pseudo color (such as a neon color), or a "hyper-real" color, wherein at least one of the color appearance parameters (i.e., hue, chroma, saturation, value, and brightness) is modified compared to the natural color to increase visibility and contrast.

[0039] Natural or true colors in this context correspond to colors that a CIE standard observer would perceive under a standard illuminant (eg, CIE illuminants A, B, or C, or any other standardized white light illumination).

[0040] The first and / or second color conversion functions can be configured to convert colors in the corresponding digital white light or fluorescent color image to colors that are not in the corresponding first and / or second imaged spectrum. Thus, the color conversion function can be used to correct for defects in imaging due to limited first and / or second imaged spectrum.

[0041] In another example, the first and / or second color conversion function can be configured to shift the white point of the digital white light and / or fluorescent color image to a predetermined position, in particular to another white point. The white point to which the white point is shifted can in particular correspond to a standardized white point defined in a standard illuminant, such as a CIE illuminant. This allows the color space to be centered.

[0042] In addition, the first and / or second color conversion functions can be configured to white balance or color balance the corresponding digital white light and / or fluorescent color image. In color balancing, the intensity of the color is adjusted to correctly render at least some colors, especially neutral colors. In white balancing, the color is adjusted so that white objects appear white rather than colorful.

[0043] According to another aspect, the first and / or second color conversion functions can be configured to expand a continuous or segmented region in the color space into a larger region in the corresponding digital white light and / or fluorescent color image. Such a color conversion function allows for better visualization of subtle differences between adjacent colors as they are further separated. For example, if the region of fluorescent colors of a particular fluorophore is expanded, then subtle differences in fluorescence become more visible. Of course, the expansion of the region can be accompanied by an appropriate color shift so that the expansion of the region occurs near a natural color or a non-natural color.

[0044] According to yet another aspect, the first and / or second color conversion function may be configured to shift all colors in the color space of the corresponding digital white light and / or fluorescent color image.

[0045] Any of the above color conversion functions may be combined into a single color conversion function, and / or any of the above color conversion functions may be applied sequentially to a corresponding one of the digital white light and / or fluorescent color images.

[0046] It may be further advantageous if the first and / or second color conversion function is configured to be applied depending on the color of the corresponding pixel in the digital white light color image and / or the digital fluorescent color image. This generally allows different colors to be processed in different ways.

[0047] According to another aspect, a plurality of different first and / or second color conversion functions may be provided, and the processor may be configured to determine or select a first and / or second color conversion function from a plurality of different first and / or second color conversion functions applied to a pixel according to the color of the pixel. The plurality of first and / or second color conversion functions may include any one of the above-mentioned color conversion functions. According to this aspect, targeted color-related modifications may be applied to the corresponding digital white light and / or fluorescent color images. For example, the area around some colors in the digital white light color image may be expanded to make the color differences more visible. Another color, such as the color of bones and / or nerves, may be simply shifted to make it more natural. Another color and therefore the tissue associated with this color may be highlighted by converting it into a neon color.

[0048] The above-described computer-implemented image processing method may be performed when executing a method for operating a medical fluorescence viewing device. The image processor as described above may be part of a medical fluorescence viewing device, such as a fluorescence microscope or a fluorescence endoscope, in particular a fluorescence microscope or a fluorescence endoscope configured for surgery (such as neurosurgery, orthopedic surgery and / or ophthalmic surgery). The fluorescence microscope may also be a laboratory microscope used, for example, for biopsy.

[0049] A medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope may include an image processor configured to perform any of the above steps. In addition, the medical fluorescence observation device may include a fluorescence color camera configured to record digital fluorescence color images and a white light color camera for recording digital white light color images.

[0050] Since the fluorescence can have a lower intensity, the fluorescence color camera can be operated with a higher integration time.

[0051] The method for operating a medical fluorescence viewing device may include the steps of recording a digital white light color image using a white light color camera and recording a digital fluorescence color image using a digital fluorescence color camera. The medical fluorescence viewing device may include a digital fluorescence color camera and a white light color camera.

[0052] To reduce post-processing, the fluorescence color camera and the white light color camera may have overlapping coaxial fields of view and / or preferably identical fields of view.

[0053] The medical fluorescence observation device can be a stereoscopic observation device or a monoscopic observation device. In one embodiment, a digital fluorescence color camera and a white light color camera can be provided in each stereoscopic channel. Alternatively, in a monoscopic medical fluorescence observation device, only a single white light color camera and a single digital fluorescence color camera can be provided.

[0054] In an alternative arrangement of a stereoscopic medical fluorescence viewing device, a fluorescence color camera may be provided in one stereo channel and a white light color camera may be provided in another stereo channel. In this arrangement, the fluorescence color camera may be configured to record a (white light) reflectance image in the second imaged spectrum. For the (white light) reflectance image, this arrangement provides a stereoscopic view, although strictly speaking the stereoscopic view is limited to the overlapping portion of the first and second imaged spectra, but this is rarely noticed by a human observer. In the case of recording fluorescence emissions, the fluorescence color camera provides a monoscopic view of the fluorescence emissions of the object, while the white light color camera provides a monoscopic view of the reflectance of the object.

[0055] The medical fluorescence observation device may include an optical color separation component configured to split the light entering the color separation component into a first imaged spectrum and a second imaged spectrum. The color separation component may include an optical element (such as a beam splitter, in particular a dichroic beam splitter), and / or an optical filter (such as a fluorescence filter that blocks light other than fluorescence emission and a white light filter that blocks fluorescence emission).

[0056] The medical fluorescence viewing device may also include an illumination assembly, which is preferably adjustable, for example, by including a plurality of differently colored LEDs or OLEDs or other light sources that emit light in a plurality of different spectral bands and which can be individually turned on and off.

[0057] In order to facilitate post-processing and ensure that the intensity values ​​of the recorded digital white light image and the digital fluorescent color image are comparable to each other, it is preferred that the digital fluorescent color image and the digital white light color image are recorded simultaneously and / or using the same exposure time and / or using the same gain and / or the same white balance and / or color compensation. Moreover, the gain of the digital fluorescent color camera and the gain of the digital white light color camera can be kept at a constant ratio, but in other cases automatic adjustment is allowed. Among these possibilities, it is most preferred that the white light color camera and the fluorescent color camera are synchronized in at least one aspect of exposure time and gain. However, the fluorescent color camera can be operated at a higher integration time. This allows compensation for low fluorescence intensities. The white light color camera and / or the fluorescent color camera are preferably CCD or CMOS cameras. The fluorescent camera and the white light color camera are preferably exactly the same.

[0058] According to another aspect, the medical fluorescence observation device may include a memory in which a plurality of different color conversion matrices are stored. Each color conversion matrix represents a different combination of the first and second imaged spectra, i.e., a different set of filters used in the color separation component, which is necessary if different fluorophores or a combination of different fluorophores are used. Examples of fluorophores used in this context are 5-ALA / pPIX, fluorescein, and ICG. Each of these fluorophores requires excitation at different wavelengths and also emits fluorescence at different wavelengths. Therefore, in a surgical environment using ICG, the first imaged spectrum and the second imaged spectrum will be different from a surgical environment using 5-ALA / pPIX, thereby requiring different color conversion matrices. Therefore, for each case of the fluorophore, the sub-bands of the first and second imaged spectra in the color band of the corresponding color space are different.

[0059] Depending on the fluorophore used, the color conversion matrix can be automatically or manually selected from a plurality of color conversion matrices. For example, the medical fluorescence observation device can be configured to automatically detect the configuration or setting of the color separation component and select the color conversion matrix according to this setting. For example, if the optical filter of the color separation component is replaced with 5-ALA / pPIX, the medical fluorescence observation device can automatically select the color conversion matrix for this 5-ALA / pPIX. Of course, this selection can also be made manually, or the automatic selection can be manually overwritten.

[0060] The claimed subject matter also relates to a computer-readable medium and a computer program comprising instructions for causing a computer to perform the computer-implemented image processing of any one of the above-described embodiments.

[0061] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0062] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0063] Below, the present invention will be described exemplarily with reference to the embodiments and with reference to the accompanying drawings. The combination of features shown in the embodiments should not be considered restrictive. For example, features of the embodiments that have technical effects such as those explained above but are not required in a specific application can be omitted. On the contrary, if the technical effect associated with this specific feature is required in a specific application, the above-mentioned features that are not part of the embodiments described below can be added. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Throughout the description and drawings, the same reference numerals are used to indicate elements that correspond to each other in function and / or structure.

[0065] In the accompanying drawings,

[0066] Figure 1 A schematic representation of a medical fluorescence viewing device is shown;

[0067] Figure 2 A schematic representation of the combination of a digital white light color image and a digital fluorescence spectrum color image to obtain a digital output color image is shown;

[0068] Figure 3 A schematic representation of jointly processing a digital white light color image and a fluorescent color image to obtain a digital output color image is shown;

[0069] Figure 4 A schematic representation of tissue reflectance spectra and spectral sensor sensitivity is shown;

[0070] Figure 5 shows a schematic representation of color conversion achieved by various color conversion functions;

[0071] Figure 6 shows a schematic representation of a method for obtaining a digital output color image from a combination of a digital white light color image and a digital fluorescence spectrum color image;

[0072] Figure 7 A schematic illustration of a system configured to perform a method for obtaining a digital output color image from a combination of a digital white light color image and a digital fluorescence spectrum color image is shown. DETAILED DESCRIPTION

[0073] First, refer to Figure 1 Exemplary embodiments of the present invention are described.

[0074] Figure 1 A medical fluorescence observation device 100 is schematically shown. The medical fluorescence observation device 100 may be a fluorescence microscope or a fluorescence endoscope, the difference between a microscope and an endoscope being that in an endoscope (not shown) the object 106 is viewed via an optical fiber which is brought close to the object 106 to be investigated, for example by being inserted into the body, whereas in a microscope the objective lens 174 is directed towards the object. Figure 1 The medical fluorescence observation device 100 is a microscope, but the following description is also applicable to an endoscope. The medical fluorescence observation device 100 may be a medical fluorescence observation device used in surgery. The medical fluorescence observation device 100 may also be a medical fluorescence observation device used in a laboratory, such as a laboratory microscope. The object 106 to be studied may consist of or include a biological tissue 107.

[0075] The object 106 may include one or more fluorophores 116, 118. At least one fluorophore may be a fluorophore naturally contained in the object. For example, bones and blood contain fluorophores. At least one fluorophore may also be added to the object 106, for example, by injecting it into biological tissue 107. Examples of fluorophores that may be added to the object 106 are ICG, fluorescein, and / or 5-ALA. 5-ALA is synthesized in cells as pPIX.

[0076] The medical viewing device 100 is a fluorescence device. This means that the medical fluorescence viewing device is configured to view, record and preferably also stimulate the fluorescence of one or more fluorophores 116 , 118 .

[0077] The medical fluorescence observation device 100 may be a stereoscopic device, such as Figure 1 As shown in the example. Therefore, it can include two identical subassemblies 101L and 101R, for each of the two stereo channels. Since the two subassemblies 101L, 101R are identical in function and structure, the following description focuses on the right subassembly 101R, but is completely applicable to the left stereo channel 101L.

[0078] The medical fluorescence observation device 100 may alternatively be a monoscopic device. In this case, only one of the two subassemblies 101L, 101R may exist. Therefore, for the monoscopic medical fluorescence observation device 100, the following description is also applicable.

[0079] The medical fluorescence viewing device 100 can be used to generate at least one digital white light color image 114 that represents a reflectance image of the object 106 in the visible light range. The visible light range or visible spectrum includes wavelengths from about 310 nm to about 1100 nm, or about 380 nm to 750 nm, or about 450 nm to about 700 nm. The fluorescence spectrum or, if more than one fluorophore is used, the fluorescence spectrum of the fluorophores 116, 118 is preferably omitted from the spectrum recorded in the digital white light color image 114. This ensures that if fluorescence is present, only reflected light is included in the digital white light color image 114. For example, if 5-ALA / pPIX is used as a fluorophore, the fluorescence spectrum of about 625 nm to about 650 nm may not be recorded in the digital white light color image 114.

[0080] If specific wavelengths of light are used to excite fluorescence, spectra including or consisting of these wavelengths may also not be recorded or represented in the digital white light color image 114. For example, if 5-ALA / pPIX is used as the fluorophore, fluorescence may be excited by illuminating the object 106 with a wavelength between about 380 nm and about 450 nm. For fluorescein and ICG, as well as other fluorophores, different but known ranges for the excitation and emission spectra apply compared to 5-ALA / pPIX.

[0081] By not recording the fluorescence excitation spectrum and the fluorescence emission spectrum, the digital white light color image 114 represents the reflectance of the object 106, that is, represents the white light image of the object 106 as a human observer would see it. Therefore, the digital white light color image 114 can be regarded as a true color image.

[0082] The digital imaging system 102 may also be used to generate a digital fluorescent color image 112 of the object 106. The digital fluorescent color image 112 represents the fluorescent emission of one or more fluorophores 116, 118. Thus, the digital fluorescent color image 112 preferably does not record any wavelengths outside of the emission spectrum or spectra of the one or more fluorophores.

[0083] Both the digital white light color image 114 and the digital fluorescent color image 112 are color images. They are recorded using at least three color bands or, equivalently, the primary colors of a color space. For example, the digital white light color image 114 and the digital fluorescent color image 112 can be recorded in an RGB color space using three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescent color image 112 can be recorded in different color spaces, respectively, and / or represent multi-spectral or hyperspectral color images. The digital white light color image 114 and the digital fluorescent color image 112 do not need to be recorded in the same color space, but this is preferred.

[0084] The digital white light color image 114 and the digital fluorescent image 112 include pixels 150. Each pixel includes color space coordinates, which represent the color of the pixel in the color space. The same color has different color space coordinates in different color spaces.

[0085] In a color space such as the RGB color space, each color of the pixel 150 is represented by a triplet of three integer numbers, where each integer number indicates the intensity of one of the primary colors R, G, B. The triplets correspond to color space coordinates.

[0086] For example, the most intense red can be indicated by the triplet [255,0,0]. The most intense green can be indicated by [0,255,0], and the most intense blue can be indicated by [0,0,255]. Therefore, the RGB color space is a three-dimensional space with three color space coordinates. By comparison, the CMYK color space would be a four-dimensional space. Colors can be viewed as points in the color space pointed to by vectors such as [0,0,255]. A multispectral or hyperspectral color space with n color bands would correspondingly produce an n-dimensional color space, and each color would be represented by an n-tuple of color space coordinates.

[0087] The spectrum recorded in the digital white light color image 114 and the spectrum recorded in the digital fluorescent color image 112 are preferably complementary to each other, ie, do not overlap.

[0088] More specifically, the digital imaging system 102 may include a digital imaging system 102 for generating a digital fluorescent color image 112 and a digital white light color image 114. The digital imaging system 102 may include a white light color camera 110 and a fluorescent color camera 111.

[0089] The white light color camera 110 is configured to record digital white light color images 114. In particular, the white light color camera 110 may be configured to generate a stream of digital white light color images 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images across the entire visible spectrum within the wavelength ranges indicated above. The white light color camera 110 may be a CCD, CMOS, or multispectral or hyperspectral camera.

[0090] The fluorescence color camera 111 is configured to record a digital fluorescence image 112. In particular, the fluorescence camera 111 may be configured to generate a stream of digital fluorescence color images 112 in the form of a digital video stream. The fluorescence color camera 111 may be configured to record the digital fluorescence color image 114 only in one or more fluorescence spectra of at least one fluorophore 116, 118. The fluorescence camera 111 may be configured to record the digital fluorescence image only in one or more narrow light bands. These narrow bands should overlap with one or more fluorescence spectra of the one or more fluorophores 116, 118 whose fluorescence is to be recorded. Preferably, the fluorescence spectra of the fluorophore 116 and the second fluorophore 118 are at least partially separated, preferably completely separated, so that the fluorescence camera 111 can record light in two separate fluorescence bands that are spaced apart from each other.

[0091] Fluorescence color camera 111 may be a CCD, CMOS or multispectral or hyperspectral camera. Preferably, white light color camera 110 and fluorescence color camera 111 are of the same type, but this is not required. Since fluorescence usually has very low intensity, fluorescence color camera 111 may have a higher integration time.

[0092] The respective fields of view 184 of the cameras 110, 111 are preferably aligned, or even coincident and coaxial. Therefore, it is preferred that the cameras 110, 111 provide exactly the same field of view 184 with exactly the same viewing angle and focal length. This results in exactly the same representation of the object 106 in the images 112, 114 generated by the different cameras 110, 111. The two cameras 110, 111 can use the same objective lens 174.

[0093] If a match of perspective and field of view cannot be generated optically, then a match may be generated through image processing by applying a matching or registration routine to the digital images 112, 114, as explained further below.

[0094] It is preferred that the two cameras 110 , 111 are operated synchronously. Specifically, the exposure times may be synchronized. Therefore, the medical fluorescence observation apparatus 100 may be configured to generate a digital white light color image 114 and a digital fluorescence image 112 simultaneously.

[0095] Preferably, the gains of the two cameras 110, 111 are synchronized, i.e., adjusted simultaneously in both cameras 110, 111. Furthermore, even if the gain is changed, the ratio of the gain applied in camera 110 to the gain applied in camera 111 may be constant. Gamma correction and color adjustment or white balance may be turned off or kept constant.

[0096] Any of the above measures facilitate comparing, jointly processing, and / or combining the two images 112 , 114 .

[0097] In order to separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescent color image 112, that is, to separate the reflection spectrum from the fluorescent spectrum, an optical color separation component 176 may be provided. The color separation component 176 may include optical elements such as a beam splitter 192, which may be a dichroic. The color separation component 176 may also or alternatively include an optical white light filter 188 and / or an optical fluorescence filter 190.

[0098] The fluorescence filter 190 is preferably configured to transmit light within the fluorescence spectrum or spectrums of the one or more fluorophores 116 , 118 and to block light outside of the one or more fluorescence spectrums.

[0099] The fluorescence filter 190 can be configured as a bandpass filter including one or more passbands. Each passband should overlap with the fluorescence emission spectrum of the corresponding fluorophore 116, 118 whose fluorescence is to be recorded. Since the fluorescence filter 190 is located in the optical path between the beam splitter 192 and the fluorescence camera 111, only wavelengths in the passband of the fluorescence filter 190 are transmitted to the fluorescence camera 111.

[0100] The white light filter 188 is preferably configured to block light in the fluorescence spectrum of one or more fluorophores 116, 118. The white light filter 188 may also be configured to block light in the fluorescence excitation spectrum.

[0101] The white light filter 188 is preferably configured as a band stop filter, whose stop band corresponds to or at least includes the pass band of the fluorescence filter 190. The white light filter 188 is located in the light path between the beam splitter 192 and the white light camera 110. Therefore, the white light camera 110 only records wavelengths outside the stop band of the white light filter 188, and therefore also outside the pass band of the fluorescence filter 190.

[0102] Either the white light filter 188 or the fluorescence filter 190 may be a tunable filter.

[0103] If the beam splitter 192 is a dichroic beam splitter, at least one of the filters 188, 190 can be omitted, since in this case the optical spectral filtering is already integrated in the dichroic beam splitter. The above description of the passband and stopband should then apply mutatis mutandis to the dichroic beam splitter 192.

[0104] Thus, white light color camera 110 records digital white light color image 114 in a first imaged spectrum (reflectance spectrum) that is different from a second imaged spectrum that fluorescence camera records as a fluorescence spectrum. The wavelengths contained in the first and second imaged spectrum are determined by the filter settings of color separation component 176.

[0105] The medical fluorescence observation device 100 may also include an illumination assembly 178 configured to illuminate the object 106, preferably through the objective lens 174, through which the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 may be configured to selectively generate white light (i.e., light uniformly distributed across the entire visible spectrum) and fluorescence excitation light (which contains only light in wavelengths that stimulate fluorescence of at least one fluorophore 116, 118). The illumination light generated by the illumination assembly 178 may be fed into the objective lens 174 using an illumination beam splitter 180.

[0106] An illumination filter 179 may be provided based on the fluorophore and its fluorescence-specific excitation spectrum. For example, if 5-ALA / pPIX is used as the fluorophore, the illumination filter may have a transmittance of 90% to 98% up to a wavelength of 425 nm, a transmittance of 0.5% to 0.7% between 450 nm and 460 nm, a transmittance of no more than 0.1% between 460 nm and 535 nm, and a transmittance of almost zero at wavelengths above 535 nm.

[0107] Instead of or in addition to the illumination filter 179 , the illumination assembly 178 may also include an adjustable light source, including, for example, a plurality of LEDs or OLEDs of different colors.

[0108] The medical fluorescence observation device 100 may further include an image processor 170. The image processor 170 may be a hardware module (such as a microprocessor), or may be a software module. The image processor 170 may also be a combination of a hardware module and a software module, for example, by using a software module configured to run on a specific processor (such as a vector processor, a floating-point graphics processor, a parallel processor, and / or a plurality of processors). The image processor 170 may be part of a general-purpose computer 186 (such as a PC).

[0109] The image processor 170 is configured to retrieve the digital white light color image 114 and the digital fluorescence image 112. For example, the image processor 170 may be configured to retrieve the digital white light color image 114 and the digital fluorescence image 112 from the memory 194 and / or directly from the cameras 110, 111. The memory 194 may be part of the image processor 170, or may reside elsewhere in the medical fluorescence viewing device 100.

[0110] Image processor 170 is also configured to calculate digital output color image 160 from digital white light color image 114 and digital fluorescent image 112. Digital output color image 160 is a color image represented in a color space. The color space of the digital output color image may be different from the color space of any one of digital white light color image 114 and digital fluorescent image 112. However, preferably, the color space of digital output color image 160 is the same as the color space of digital white light color image 114 and digital fluorescent image 112.

[0111] The image processor 170 includes at least two operating modes, and the image processor 170 is configured to selectively operate in the at least two operating modes. For example, a user (not shown) can manually select which of the two operating modes to execute. In order to select the operating mode from the at least two available operating modes, a selector device 165 can be provided. The selector device 165 can be a mechanical device, such as a dial or switch; or a button or other element on a graphical user interface displayed on a display (such as display 132, 182), which will be described in more detail below.

[0112] like Figure 2 As shown in , the image processor 170 may include a first operation mode A and a second operation mode B.

[0113] In the first mode of operation A, the image processor 170 is preferably configured to generate a digital output color image 160 that represents what is seen through the eyepiece 104. To achieve this, in the mode of operation A, the image processor 170 may be configured to combine the digital fluorescent color image 112 and the digital white light color image 114 to generate the digital output color image 160. In particular, the image processor 170 may be configured to additively combine or add, at each corresponding pixel 150 in the digital fluorescent color image 112 and the digital white light color image 114, the color space coordinates of the digital fluorescent color image 112 and the digital white light color image 114 in each color band of their color spaces.

[0114] In the second operating mode B, the image processor 170 is configured to first apply the color conversion function 140 to at least one of the digital white light color image 114 and the digital fluorescent color image 112, and then generate the digital output color image 160 from the digital white light color image 114 and the digital fluorescent color image 112. Thus, in the second operating mode B, at least one of the digital white light color image 114 and the digital fluorescent color image 112 is color converted prior to combination. By switching between the first operating mode A and the second operating mode B, the user is able to switch between a view that faithfully renders what is seen through the eyepiece and a view that can highlight different features of the object 106.

[0115] Color conversion function 140 may be any type of function, such as a one-dimensional or n-dimensional interpolation function. However, preferably, color conversion function 140 is a linear transformation function. Specifically, color conversion function 140 may be a color conversion matrix 142. One dimension of color conversion matrix 142 may be the sum of the number of color bands in digital fluorescent color image 112 and the number of color bands in digital white light color image 114. Another dimension of color conversion matrix 142 may correspond to the number of color bands in digital output color image 160 in another direction of the matrix.

[0116] In another embodiment, color conversion matrix 142 may have a first dimension and a second dimension, wherein the first dimension corresponds to the dimension of the color space of digital white light color image 114 or digital fluorescent color image 112 (depending on which of the two the color conversion matrix is ​​applied to), and the second dimension corresponds to the dimension of the color space of digital output color image 160.

[0117] The dimensions of the color conversion matrix 142 are preferably a 3×3 matrix if all images 112 , 114 are in the RGB color space, or a 6×3 or 3×6 matrix if the color conversion matrix is ​​configured to operate on both digital white light color images and digital fluorescent color images.

[0118] The second operating mode B itself may include different sub-modes. Each sub-mode may apply a different color conversion function 140 to at least one of the digital white light color image 114 and the digital fluorescent color image 112 before combining the digital fluorescent color image 112 with the digital white light color image 114. Figure 2 In FIG. 1 , three sub-modes B-1, B-II, B-III are shown as examples only. Of course, more or less than three sub-modes may be provided. Different color conversion functions 140 may be stored in the memory 194.

[0119] For example, the first color conversion function 140a may be applied only to the digital white light color image 114, while the second color conversion function 140b may be applied only to the digital fluorescent color image 112. The first color conversion function 140a and the second color conversion function 140b may be applied independently of each other, thereby decoupling the color conversion of the digital white light color image 114 from the color conversion of the digital fluorescent color image 112. For example, only one of the first color conversion function 140a and the second color conversion function 140b may be applied to the respective images 112, 114, or both may be applied to their respective images.

[0120] In one embodiment, a user may select between different first color conversion functions 140a and / or different second color conversion functions 140b.

[0121] For example, in operation mode Bl, a first color conversion function 140a configured to convert colors in the digital white light color image 112 into colors that are not in the second imaged spectrum may be used. In this case, the natural colors of the object 106 or parts thereof that are not correctly recorded in the digital white light color image 114 because they are partially filtered and thus distorted by the color separation component 176 may be converted by the first color conversion function 140a into colors that more closely correspond to the natural colors.

[0122] In another additional or alternative example, a first color conversion function 140a may be used in operating mode B1 or an additional operating mode that converts the colors in the digital white light color image 114 into pseudo colors or different hues. This allows colors associated with certain types of tissue to be highlighted. For example, nerves, the color of arterial blood, and / or the color of venous blood may all be converted into different pseudo colors.

[0123] By using the selector device 165, the user can quickly browse the assignments of different pseudo colors to different types of tissue. In one operating mode, the color of arterial blood can be converted to neon red, in another operating mode, the color of venous blood can be converted to neon blue, and in yet another operating mode, the color of neural tissue can be converted to neon yellow. Other operating modes can use combinations of these modes. If the first color conversion function 140a assigns different hues instead of pseudo colors, the contrast can be enhanced. For example, the first color conversion function 140a can extend the red color in the digital white light color image 114 to a wider range, making smaller changes in blood oxygenation more visible to the user.

[0124] In yet another additional or alternative example, the first color conversion function can be used, for example, in operating mode BI or an additional operating mode to color balance and / or white balance the digital white light color image 114. In white balancing, the overall mix of colors is changed, but neutral colors (i.e., gray, black, and white) are maintained as neutral colors. In color balancing, colors including neutral colors are adjusted to correspond to neutral colors.

[0125] In yet another additional or alternative example, the first color conversion function 140a can be used, for example, in operating mode BI or an additional operating mode to shift the white point of the digital white light color image 114 to a predetermined position in the color space. Adjusting the white point allows adjustment for different lighting and adjusting filter settings of the color separation component 176.

[0126] According to one example, at least one of the above-mentioned first color conversion functions 140a may be applied to the digital white light color image 114. According to another example, at least two of the above-mentioned first color conversion functions 140a may be sequentially applied to the digital white light color image 114. Alternatively or additionally, two or more of the above-mentioned color conversion functions 140a may be combined into a single color conversion function 140a.

[0127] The above description of the various first color conversion functions 140a also applies mutatis mutandis to the various second color conversion functions 140b, the only difference between the two functions being that the second color conversion functions 140b are configured to operate on the digital fluorescent image 114 and to operate on different colors.

[0128] In operation mode B-II, for example, only one or more color conversion functions 140b may be applied to the digital fluorescent image without applying any color conversion function 140a.

[0129] In operating mode B-III, for example, one or more color conversion functions 140 a may be applied to the digital white light color image, and one or more color conversion functions 140 b may be applied to the digital fluorescent color image 112 .

[0130] An optional third operating mode C may be provided, wherein a third color conversion function 140 c is applied to both the digital white light color image 114 and the digital fluorescent color image 112 , wherein application of the color conversion function 140 c directly produces a digital output color image 160 .

[0131] If another type of fluorophore with a different excitation and / or fluorescence wavelength is used, then a different color conversion function 140a, 140b, 140c needs to be provided, because the combination of filter settings in the color separation component 176 and thus the first and second imaged spectra need to be adapted to the excitation and / or fluorescence wavelength.

[0132] Therefore, in one embodiment, the medical viewing device 100 or the processor 170 may be configured to store a plurality of different sets 143 of color conversion functions 140a, 140b, and / or 140c. Each different set 143 includes one or more of the above-mentioned color conversion functions 140a, 140b, and / or 140c. Each different set 143 corresponds to the use of a different fluorophore, and thus represents a different filter setting of the color separation component 176. For example, the first set 143a may be used for ICG as a fluorophore, while the second set 143b may be used for 5-ALA / pPiX as a fluorophore, and of course, additional sets may be used to accommodate additional fluorophores or combinations of fluorophores.

[0133] The medical fluorescence viewing device 100 can be adjusted to accommodate different fluorophores by reconfiguring the color separation component 176 (e.g., by exchanging its optical elements, such as filters 190 and / or 192, or dichroic beam splitter 180). For example, if the medical viewing device 100 is configured to automatically detect the settings of the color separation component 176, then the selection of the appropriate set 143 of color conversion functions 140 can be performed automatically.

[0134] Alternatively or additionally, the medical viewing device 100 may include a filter setting selector device 168 that allows a user to manually select a set 143 of color conversion functions 140 to be applied to the digital white light color image 114 and / or the digital fluorescent color image. The filter setting selector device 168 may be mechanical and / or part of a graphical user interface. For example, if the filter setting selector device 169 is operated to be in position a, then set 143a is selected. In position b, set 143b is selected, and so on.

[0135] The digital output color image 160 may be displayed on a display 132 that is integral with the medical fluorescence viewing device 100. For example, the display 132 may be integrated into an eyepiece or eyepiece 104 of the medical fluorescence viewing device 100. The display 132 may also display a graphical user interface for operating the medical viewing device 100.

[0136] The medical fluorescence viewing device 100 may include a direct optical path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In this case, the display may be a semi-transparent display 132 located in the direct optical path 134, or the display may be projected into the direct optical path 134. A beam splitter 136 may be provided to split the light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed to the eyepiece 104.

[0137] Alternatively, the medical fluorescence viewing device 100 may not have a direct optical path 134, but only display images from the integral display 132. As a further alternative, the medical fluorescence viewing device may not have any display.

[0138] The medical fluorescence observation device 100 may include an output interface 172 to which one or more (external) displays 182 may be connected. To this end, the output interface 172 may include standardized connectors and data transmission protocols, such as USB, HDMI, DVI, DisplayPort, Bluetooth, and / or other items. The external display may be a monitor, 3D goggles, eyepieces, etc. Any combination of external displays may be connected to the output interface 172. Any of the displays 182 may display a graphical user interface for operating the medical observation device 100.

[0139] The computer 186 or the image processor 170 is connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission line may be wired or wireless, or partially wired and partially wireless. The computer 186 and / or the image processor 170 may not be integrated as a whole in the medical fluorescence observation device 100, but may be physically located away from the digital imaging system 102. To this end, the digital imaging system 102 and the computer 186 and / or the image processor 170 may be connected to a network, such as a LAN, WLAN or WAN, to which at least one display 182 is also connected.

[0140] According to the modification, the medical fluorescence observation device 100 can be stereoscopic, but only includes two cameras, one for each stereo channel. In one stereo channel, a fluorescence color camera 111 is used and is configured to selectively also record white light reflectance, while in the other stereo channel, a white light color camera 110 is used. This arrangement provides a stereoscopic white light color image without using fluorescence, and provides a monoscopic white light color image and a monoscopic fluorescence color image when using fluorescence. The description above and below also applies to this configuration.

[0141] Figure 2 An example of a first operating mode A is shown.

[0142] Reference numeral 200 is a quantitative illustration of a first imaged spectrum 202 as it is recorded by digital white light camera 110 and / or represented in digital white light color image 114, respectively. Intensity I across wavelength / color λ is shown normalized. First imaged spectrum 202 preferably extends across at least visible spectrum 212.

[0143] By way of example only, the color space in which the first imaged spectrum 202 is recorded is an RGB color space having three primary colors or color bands 204, 206, and 208. One primary color 204 is blue, another primary color 206 is green, and the third primary color 208 is red. The sensitivity of the sensor of the white light color camera 110 in the different primary colors 204, 206, 208 is tuned to result in a sensitivity across the visible spectrum 212 that is as constant as possible.

[0144] If a color space other than RGB is used, the number, position and / or width of the color bands may be different. Apart from this, there are no major differences from the RGB color space.

[0145] The first imaged spectrum 202 does not include fluorescence excitation light and the fluorescence emission spectrum of at least one fluorophore 116, 118. Therefore, the first imaged spectrum 202 may include at least one stop band 210 that is consistent with the fluorescence emission of at least one fluorophore whose fluorescence is to be recorded by the fluorescence color camera 111. The stop band 210 is created, for example, by the white light filter 188. The number, width, and / or position of the stop bands 210 depends on the number and type of fluorophores to be observed in the object 106.

[0146] At reference numeral 220, a second imaged spectrum 222 is shown as it is recorded by the fluorescence color camera 111 and / or represented in the digital fluorescence color image 112, respectively. Merely by way of example, the color space in which the second imaged spectrum 222 is recorded is also an RGB color space. The sensitivity of the sensor of the fluorescence color camera 111 in the different primary colors 204, 206, 208 is tuned to result in a sensitivity across the visible spectrum 212 that is as constant as possible.

[0147] The spectra 202, 222 need not be recorded in the same color space, but this is preferred.

[0148] The second imaged spectrum 222 may include one or more passbands 224. The number, position and / or width of the passbands depends on the number and type of fluorophores used. At least one passband 224 preferably corresponds to at least one stopband 210. At least one passband is generated, for example, by the fluorescence filter 190.

[0149] The first imaged spectrum 202 and the second imaged spectrum 222 complement each other. They preferably complement each other to cover the entire or most of the visible spectrum 212.

[0150] Each passband 224 of the second imaged spectrum 222 preferably overlaps with a fluorescence emission spectrum 226, 228 of a fluorophore 116, 118 whose fluorescence is to be recorded, and may overlap with one or more primary colors 204, 206, 208 of the color spectrum. For example, the fluorescence emission spectrum 226 of one fluorophore 116 may overlap with all three primary colors 204, 206, 208. The sensor of the fluorescence camera 111 recording the color band 204 records only a small part 230 of the fluorescence emission spectrum 226. The sensor of the fluorescence camera 111 recording the color band 206 records a large part 232 of the fluorescence emission spectrum 226, while the sensor of the fluorescence camera 111 recording the color band 208 again records only a small part 236 of the fluorescence emission spectrum 226.

[0151] If present under certain circumstances, fluorescence spectrum 228 is, by contrast, recorded only by the sensor of fluorescence camera 111 which records color strip 208 .

[0152] In its simplest form, the combination 240 of the digital white light color image 114 and the digital fluorescent color image 112 may be an additive combination. For example, the color space coordinates in each color band of the digital white light color image 114 and the fluorescent color image 112 are added at each corresponding pixel 150. If the images 112, 114 are registered and have the same size, then the corresponding pixels 150 may be pixels 150 located at the same position in both images 112, 114. If both images 112, 114 are RGB images and the color space coordinates in the RGB color band of the pixel 150 in the digital white light color image are {R1, G1, B1}, and the color space coordinates in the RGB color band of the corresponding pixel 150 in the digital fluorescent color image are {R2, G2, B2}, then the color space coordinates in the RGB color band of the corresponding pixel 150 in the digital output color image may be calculated as {R1+R2, G1+G2, B1+B2}.

[0153] Reference numeral 250 points to a qualitative reproduction of a spectrum 252 of the digital output image 160. In essence, the spectrum 252 of the digital output image 160 is the addition of the first imaged spectrum 202 and the second imaged spectrum 222.

[0154] Instead, the individual spectra 202 and 222 are processed as a single combined multispectral spectrum whose number of color bands corresponds to the total number of color bands in the first imaged spectrum 202 and the second imaged spectrum 222. In other words, the digital fluorescent color image 112 and the digital white light color image 114 are jointly processed as a single (virtual) multispectral image.

[0155] This is consistent with operation mode C ( Figure 1 ), wherein the color conversion functions 140, 140c are applied to the digital white light color image 114 and the digital fluorescent color image 112 to obtain their spectra in Figure 2 The digital output color image 160 indicated at reference numeral 250 in FIG.

[0156] Reference below Figure 3 To keep it simple, Figure 3 It is shown in RGB color space Figure 2 spectral spectrum, and only a single passband 224 and stopband 210 are shown.

[0157] The digital white light color image 114 recorded in the first imaged spectrum 202 is composed of signals R1, G1, and B1, each representing the intensity I in a corresponding color band. Signal G1 is produced by two separate wavelength bands due to the stop band 210. However, the sensor recording G1 cannot distinguish between these two wavelength bands.

[0158] The digital fluorescent color image 112 recorded in the second imaged spectrum 222 is composed of signals R2, G2, B2 in each color band.

[0159] Since the first imaged spectrum 202 and the second imaged spectrum 222 are complementary to each other, each color band is split into two signals, each signal coming from a different camera 110, 111 and representing or equivalent to a sub-band of the color band. The color band 204 is split into R1 and R2, respectively. The color band 206 is split into G1 and G2, respectively, and the color band 208 is split into B1 and B2, respectively. Preferably, there is no overlap between the individual signals in the color band.

[0160] Thus, even if the same type of color camera is used to record the first imaged spectrum 202 and the second imaged spectrum 222 in the same color space, a multispectral image of the object 106 is obtained, which in this case consists of six color bands R1, R2, G1, G2, B1, B2. This is schematically shown at reference numeral 300. It is noted that this is independent of whether the fluorescence camera actually records fluorescence emission or reflectance.

[0161] In other words, at least one color band 204, 206, 208 of the color space of at least the digital output color image 160 is subdivided into two sub-bands R1, R2 and / or G1, G2 and / or B1, B2, wherein one of the two sub-bands is included in the digital fluorescent color image 112 and the other of the two sub-bands is included in the digital white light color image 114. Preferably, the two sub-bands in the color band do not overlap. They are preferably complementary. Most preferably, together they at least nearly complete the corresponding color band. The sub-bands may include or consist of two separate spectral bands. Each sub-band may be considered to be a color band in itself.

[0162] The color conversion function 140 represents the subdivision of the color band into sub-bands, which is determined by the stop band 210 and the pass band 224. Since the width and / or position of a sub-band determines how much light will be collected in that sub-band by the respective camera 110, 111, the color conversion function 140c needs to be adjusted for each different filter setting of the color separation component 176.

[0163] Thus, the digital white light image 112 and the digital fluorescent image 114 can be processed together as an image consisting of the union of the color bands in the two images 112 and 114. This provides improved color resolution. The color space coordinates (set of) {R1, R2, G1, G2, B1, B2} of this multispectral image correspond to the union (set) of the set of color space coordinates {R1, G1, B1} of the digital white light color image 114 and the set of color space coordinates {R2, G2, B2} of the digital fluorescent color image 112.

[0164] A color conversion function 140 c may be applied to this union set in the form of a linear transform using a color conversion matrix 142 to generate a digital output color image 160 .

[0165] If the digital output color image 160 is represented in the RGB color space, then a 6×3 or 3×6 color conversion matrix can be applied to the combined digital fluorescent and white light images 112, 114 or their component signals R2, G2, B2, R1, G1, B1, respectively, to obtain the RGB signals R*, G*, B*. The matrix coefficients C11, C12, ..., C63 can be determined experimentally.

[0166] Next, the operation mode B is explained.

[0167] exist Figure 4 , a schematic reflection spectrum 400 of biological tissue of the object 106 is indicated, wherein the term "tissue" includes both fluid tissue and solid tissue. For example, the reflection spectrum 400 may correspond to the reflection spectrum of oxygenated blood.

[0168] Figure 4Also shown in FIG. 5 are the spectral sensitivities R, G, B of the R, G, B sensors of the sample RGB color camera.

[0169] In human perception, tissue (such as oxygenated blood) is perceived as having a natural color, i.e., red, under white light illumination (such as CIE illuminants). Therefore, the reflection spectrum 400 corresponds to the natural color, but it is widely distributed across the spectrum of visible light. In the first imaged spectrum 202 in which the digital white light color image 114 is recorded, the wavelength λ in the stop band 210 is not recorded. Therefore, the reflection spectrum 400 represented in the digital white light color image 114 cannot faithfully represent the natural color of the tissue.

[0170] The same applies to the second imaged spectrum 222 and the digital fluorescence color image 114, wherein the reflection spectrum 400 will also be distorted.

[0171] Figure 5 Schematic representation of the CIE 1932 color space or any other tristimulus or uniform color space is shown, wherein color 500 is shown. Color 500 may, for example, represent the color of a color having a reflectance spectrum 400 ( Figure 4 ) when illuminated by CIE illuminants A, B, or C or another standard illuminant.

[0172] Since wavelengths in stopband 210 are not recorded, color 500 is represented in digital white light color image 114 as (recorded) color 502. To more faithfully represent color 500, recorded color 502 is converted to color 500 by a color conversion function 140, which may be a first color conversion function 140a or a second color conversion function 140b. For example, the color conversion may be a shift or translation in an LMS color space, a uniform color space, a tristimulus color space, or even an RGB color space.

[0173] The color conversion function 140 may be determined by color calibration using, for example, known colors on a color chart, known filter settings, and known lighting conditions. The wavelength of color 500 may be within the stop band 210 and therefore not within the first imaged spectrum 202. Together with the shift from 502 to 500, the entire color space may be shifted.

[0174] In some applications, it may be beneficial if color 500 (to which color 502 is converted) is not a natural color, but rather a pseudo-color or a color that is close to but clearly different from a natural color, to better offset this color and the associated tissue type relative to other tissue or fluorescent colors that are spatially or perceptually close to color 500, and thus highlight this type of tissue to the experienced eye. For example, in the case of oxygenated blood, color 500 may be a "super-realistic" red that is brighter than the natural color of oxygenated blood, or a red pseudo-color (such as neon red). Of course, this may be done for any other color and / or tissue.

[0175] Due to the stop band 210, the location of the natural white point 504 may be shifted to the recorded white point 506. Therefore, another color conversion function 140 (e.g., color conversion function 140d) may shift the recorded white point 506 to or closer to the natural white point 504. In this regard, the white point is treated like any color.

[0176] A shift of a single color may correspond to a shift of the entire color space, ie, all colors are shifted by the same amount.

[0177] The digital white light color image 112 may be white balanced or color balanced using the same amount and direction of shift across the entire color space.

[0178] Alternatively, different colors may be transformed or shifted differently in at least one of the amount and direction of the shift. For example, the recorded color 508, e.g., deoxygenated blood, may be further shifted into the blue range, to color 510. The shift of color 508 to 510 may be different from the shift of color 502 to color 500. Thus, the color conversion function 140 may be configured to generate a color-dependent transformation, rather than shifting the entire color space. For different (recorded) colors 502, 504, 508, different color conversion functions 140 may be applied.

[0179] Another example of a color conversion function 140 may extend region 512 in color space to a larger region 514. In region 514, the distance between colors is greater than the distance between colors in region 512, so the differences between colors become more visible. The application of such a color conversion function may be color dependent. In this case, only regions 512 having predetermined colors are extended. This color conversion function 140 may include a color shift so that region 514 is not only extended, but also moved to a different region of the color space.

[0180] Any of the above color conversion functions 140 may convert only a selected subset of the color appearance parameters of a color. The color appearance parameters are hue, chromaticity, saturation, lightness, and brightness. For example, when converting color 502 to color 500, only the hue may be changed, or the color conversion function 140 may only change the saturation and lightness.

[0181] Any combination of the above color conversion functions 140a may be applied. It should be understood that although the above description uses the expression "image", the color conversion function 140 is applied to the image at the pixel level, that is, to each pixel 150 of the digital white light color image 112. The color of the pixel 150 may determine which color conversion function 140 or which combination of color conversion functions 140 is applied. This may be done automatically by the processor 170.

[0182] Figure 6 A schematic representation of an imaging method is shown, which method may be implemented as a computer-implemented method running on an image processor 170, for example.

[0183] At optional step 600, a digital white light color image 114 is recorded using, for example, a white light color camera 110. At optional step 602, a digital fluorescent color image 112 is recorded using, for example, a fluorescent camera 111. Steps 600 and 602 are optional, as the images 112, 114 may also be retrieved from memory. As explained above, the cameras 110, 111 should be synchronized in terms of exposure time and locked to each other in terms of gain, i.e., maintaining the same gain ratio. Gamma may be set to a fixed value and all automatic color adjustments are preferably turned off.

[0184] At optional step 604, the respective images 112, 114 may be demosaiced.

[0185] At optional step 606 , one or both images 112 or 114 may be matched and registered such that identical image features are represented in each image 112 , 114 in a geometrically identical manner with respect to size and orientation.

[0186] In one mode of operation (e.g., mode of operation B), but optionally also in mode of operation C, a color conversion is performed in a color conversion step 608. Step 608 is omitted in mode of operation A. In step 608, a color conversion function 140 or a combination of color conversion functions 140 as described above is applied to at least one of the digital white light color image 112 and the digital fluorescent image 114.

[0187] At image combination step 610, the digital white light color image 114 and the digital fluorescent color image 112 are combined in all operating modes A, B, C to generate a digital output color image 160. As described above, the images 112, 114 may be combined, for example, by simply adding them together. Figure 2 In operation mode C, the color conversion function 140 may be applied to the digital white light color image 114 and the digital fluorescent image 112 in step 419 to generate a digital output color 160, such as Figure 3 as shown in .

[0188] In step 612, post-processing may be performed. For example, the digital output color image 160 may be homogenized, its contrast may be enhanced, and / or a color space conversion, such as a color space conversion from RGB to sRGB and / or gamma correction may be performed. It is important to note that the color conversion in steps 608 and / or 610 is not gamma correction.

[0189] At step 612, the digital output image 160 is displayed.

[0190] Some embodiments relate to microscopes, particularly including as combined Figures 1 to 6 Alternatively, the microscope may be a fluorescence microscope such as one or more of the systems described herein. Figures 1 to 6 Part of or connected to one or more of the described systems.

[0191] Figure 7 A schematic diagram of a system 700 configured to perform the methods described herein is shown. System 700 includes a microscope 710 and a computer system 720. Microscope 710 is configured to capture images and is connected to computer system 720. Computer system 720 is configured to perform at least a portion of the methods described herein. Computer system 720 can be configured to execute a machine learning algorithm. Computer system 720 and microscope 710 can be separate entities, but can also be integrated together in a common housing. Computer system 720 can be part of a central processing system of microscope 710 and / or computer system 720 can be part of a subcomponent of microscope 710, such as a sensor, an actor, a camera, or a lighting unit of microscope 710.

[0192] The computer system 720 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in different locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 720 may include any circuit or combination of circuits. In one embodiment, the computer system 720 may include one or more processors that may be of any type. As used herein, a processor may mean any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA) (e.g., of a microscope or microscope component (e.g., a camera)), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 720 may be custom circuits, application specific integrated circuits (ASICs), and the like, such as, for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 720 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as a main memory in the form of a random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media such as compact disks (CDs), flash memory cards, digital video disks (DVDs), and the like. The computer system 720 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 720.

[0193] Some or all of the method steps may be performed by (or using) a hardware device (e.g., a processor, a microprocessor, a programmable computer or an electronic circuit). In some embodiments, such a device may perform one or some of the most important method steps.

[0194] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transitory storage medium (such as a digital storage medium, e.g., a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM and EPROM, EEPROM or flash memory) having an electronically readable control signal stored thereon, which cooperates (or is capable of cooperating with) a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0195] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0196] Generally speaking, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

[0197] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0198] In other words, therefore, an embodiment of the inventive method is a computer program with a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0199] Therefore, another embodiment of the invention is a storage medium (or a data carrier, or a computer-readable medium) comprising a computer program stored thereon, the computer program being for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0200] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection, for example via the Internet.

[0201] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0202] A further embodiment comprises a computer having installed on it the computer program for performing one of the methods described herein.

[0203] Another embodiment according to the invention comprises an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transmitting the computer program to the receiver.

[0204] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the method described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. In general, the method is preferably performed by any hardware device.

[0205] Reference numerals

[0206] 100 Medical fluorescence observation equipment

[0207] 101L Stereo subassembly for left channel

[0208] 101R Stereo subassembly for right channel

[0209] 102 Digital Imaging System

[0210] 104 Eyepiece

[0211] 106 Objects to be studied

[0212] 107 Biological tissue

[0213] 110 White light color camera

[0214] 111 Fluorescent Color Camera

[0215] 112 Digital fluorescent color images

[0216] 114 Digital white light color image

[0217] 116 Fluorophore

[0218] 118 Second fluorophore

[0219] 132 Integrated / Internal Display

[0220] 134 Direct optical path

[0221] 136 Beam Splitter

[0222] 140 color conversion functions

[0223] 140a-140d Different color conversion functions

[0224] 142 Color conversion matrix

[0225] A collection of 143 color conversion functions

[0226] 143a-143c Different sets of color conversion functions

[0227] 150 pixels

[0228] 160 digital output color image

[0229] 165 Selector Device

[0230] 169 Filter setting selector

[0231] 170 Image Processor

[0232] 172 Output Interface

[0233] 174 Objective

[0234] 176 Color Separation Component

[0235] 178 Lighting components

[0236] 179 Lighting Filter

[0237] 180 Illumination Beam Splitter

[0238] 182 Display

[0239] 184 Field of View

[0240] 186 Computer

[0241] 188 White light filter

[0242] 190 Fluorescence Filter

[0243] 192 Dichroic Beam Splitter

[0244] 194 Memory

[0245] 196 Data transmission line

[0246] 200 Spectrum

[0247] 202 First Imaging Spectrum

[0248] 204, 206, 208 Original color or color band

[0249] 210 Stopband

[0250] 212 Visible spectrum or range of light

[0251] 220 Spectrum

[0252] 222 Second Imaging Spectrum

[0253] 224 Passband

[0254] 226 Fluorescence emission spectra of fluorophores

[0255] 228 Fluorescence emission spectrum of another fluorophore

[0256] 230 Portion of the fluorescence emission spectrum in the color band 204

[0257] 232 Portion of the fluorescence emission spectrum in color band 206

[0258] 236 Portion of the fluorescence emission spectrum in color band 208

[0259] Combination of 240 images

[0260] 250 Spectra

[0261] 300 Multispectral images from digital white light and fluorescence images

[0262] 400 Reflectance Spectra of Tissue

[0263] 500 Natural

[0264] 502 Record Color

[0265] 504 Natural White Spot

[0266] 506 recorded white spots

[0267] 508 Recorded Color

[0268] 510 converted colors

[0269] 512 Recorded color area

[0270] 514 Target color area

[0271] 600 Recording of digital white light color images

[0272] 602 Recording digital fluorescent color images

[0273] 604 De-mosaic

[0274] 606 Registration

[0275] 608 Color Conversion Steps

[0276] 610 Image Combination Steps

[0277] 612 Post-processing

[0278] 614 Display

[0279] 700 System

[0280] 710 Microscope

[0281] 720 Computer Systems

[0282] λ Wavelength

[0283] I. Intensity

[0284] A The first operating mode of the processor

[0285] B The second operating mode of the processor

[0286] C11,C12,…,C63 Coefficients of the color conversion matrix

[0287] Example of RGB color space

[0288] R, G, B, R1, G1, B1, R2, G2, B2, R*, G*, B* color space coordinates

Claims

1. An image processor (170) for a medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the processor (170) being configured to: - retrieving a digital white light color image (114) of the object (106) recorded in the first imaged spectrum (202); and - retrieving a digital fluorescent color image (112) of the object (106) recorded in a second imaged spectrum (222), the second imaged spectrum (222) overlapping with a fluorescence emission spectrum (226) of at least one fluorophore (116), the second imaged spectrum (222) being different from the first imaged spectrum (202), the first imaged spectrum (202) and the second imaged spectrum (222) both overlapping with the visible spectrum (212); wherein the image processor (170) is configured to selectively operate in one of a first operating mode (A) and a second operating mode (B), the first operating mode comprising an image combining step (610) and the second operating mode comprising a color conversion step (608) and the image combining step (610); wherein the image processor (170) is configured to generate a digital output color image (160) by combining the digital white light color image (114) and the digital fluorescent color image (112) in the image combining step (610); and wherein the image processor (170) is configured to apply in the color conversion step (608) at least one color conversion function (140, 140a, 140b) from the group consisting of: - a first color conversion function (140a) applied to the digital white light color image (114); and - a second color conversion function (140b) applied to said digital fluorescent color image (114).

2. The image processor (170) of claim 1, wherein the first imaged spectrum (202) and the second imaged spectrum (222) are complementary to each other.

3. The image processor (170) of claim 1 or 2, wherein the colors of the digital white light color image (114), the digital fluorescent color image (112), and the digital output image (160) are represented in the same color space.

4. An image processor (170) according to any one of claims 1 to 3, wherein the image processor (170) is configured to additively combine the color space coordinates (R, G, B) of corresponding pixels (150) of the digital white light color image (112) and the digital fluorescent color image (114) in the image combination step (610).

5. An image processor (170) according to any one of claims 1 to 4, wherein the first color conversion function (140a) is configured to convert recorded colors (502, 504, 508) in the digital white light color image (112) into colors (500, 506, 510) that are not in the first imaged spectrum (202) and / or the first color conversion function (140b) is configured to convert colors (502, 504, 508) in the digital fluorescent color image (112) into colors (500, 506, 510) that are not in the second imaged spectrum (222).

6. The image processor (170) of any one of claims 1 to 5, wherein the first color conversion function (140a) is configured to color balance the digital white light color image (114).

7. An image processor (170) according to any one of claims 1 to 6, wherein the first color conversion function (140a) is configured to shift the recorded colors (502, 504, 508) in the digital white light color image (114) to predetermined colors (500, 506, 510) in a color space and / or the second color conversion function (140b) is configured to shift the colors (502, 504, 508) of the digital fluorescent color image (114) to predetermined colors (500, 506, 510) in a color space.

8. The image processor (170) of claim 7, wherein the processor (170) is configured to shift different recorded colors (502, 504, 508) by different amounts and / or directions in a predetermined color space.

9. An image processor (170) according to any one of claims 1 to 8, wherein the first color conversion function (140a) and / or the second color conversion function (140b) is configured to convert a region (512) of recorded colors (502, 504, 508) into a larger region (514) in a predetermined color space.

10. A medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the medical fluorescence observation device (100) comprising: An image processor (170) according to any one of claims 1 to 9; a fluorescent color camera (111) configured to record the digital fluorescent color image (112); and A white light color camera (110) is configured to record the digital white light color image (114).

11. The medical fluorescence observation device (100) according to claim 10, wherein the fluorescence camera (111) has a higher integration time than the white light color camera (110).

12. The medical fluorescence observation device (100) according to claim 10 or 11, wherein the medical fluorescence observation device (100) is a surgical fluorescence microscope.

13. A computer-implemented image processing method for a fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the computer-implemented image processing method comprising the following steps: - retrieving a digital white light color image (114) of the object (106) recorded in the first imaged spectrum (202); and - retrieving a digital fluorescent color image (112) of the object (106) recorded in a second imaged spectrum (222), the second imaged spectrum (222) overlapping with a fluorescence emission spectrum (226) of at least one fluorophore (116), the second imaged spectrum (222) being different from the first imaged spectrum (202), the first imaged spectrum (202) and the second imaged spectrum (222) both overlapping with the visible spectrum (212); wherein the method is configured to be selectively performed in one of a first operating mode and a second operating mode; wherein, in the first operation mode (A), an image combining step (610) is performed, and in the second operation mode (B), a color conversion step (608) and the image combining step (610) are performed; wherein, in the image combining step (610), a digital output color image (160) is generated by combining the digital white light color image (114) and the digital fluorescent color image (112); and, applying in said color conversion step (610) at least one color conversion function (140, 140a, 140b) from the group consisting of: - a first color conversion function (140a) applied to the digital white light color image (114); and - a second color conversion function (140b) applied to said digital fluorescent color image (114).

14. A computer program product or a computer readable medium comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to claim 13.

15. A method for operating a medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the method comprising the following steps: - recording a digital fluorescence color image (112) in the second imaged spectrum (222) using a fluorescence color camera (111); as well as - recording a digital white light color image (114) in the first imaged spectrum (202) using a white light color camera (110).

Images