Method for switching images, processor and medical fluorescence observation device

Abstract

The invention relates to an image processor (170) and / or a computer-implemented image processing method for a medical fluorescent viewing device (100), such as a fluorescent microscope or a fluorescent endoscope. A digital white light color image (114) of the object (106) recorded in the first imaged spectrum (202) and a digital fluorescent color image (112) of the object (106) recorded in the second imaged spectrum (222) are retrieved. The second imaged spectrum (222) overlaps a fluorescence emission spectrum (226) of the at least one fluorophore (116) and is different from the first imaged spectrum (202). Both the first and second imaged spectra (202, 222) overlap the visible spectrum (212). In order to facilitate a user's manipulation of a medical viewing device (100) and migration to a new fluorescence imaging technique, at least one digital output color image (160, 160-I, 160-II, 160-III) of a set (1000) of output digital output color images (160) is provided, the set comprising: digital output color images (160, 160-I) generated by an image processor (170) only from digital fluorescence color images (112); a digital output color image (160, 160-II) generated by the image processor (170) only from the digital white light color image (114); and a digital output color image (160, 160-III) generated by the image processor (170) from a combination of the digital fluorescent color image (114) and the digital white color image (112). For switching between the respective output images, receiving a display selection signal (900) from a selector device (165); and selecting at least one digital output color image (160, 160-I, 160-II, 160-III) from the group for output according to the display selection signal (900).

Description

Technical Field

[0001] The claimed subject matter relates to a computer-implemented image processing method and to an image processor for generating one or more digital output color images of an object for display in a medical fluorescence viewing device such as a fluorescence microscope or a fluorescence endoscope. The claimed subject matter also relates to a method for operating such a medical fluorescence viewing device and to the medical fluorescence viewing device itself. Background Art

[0002] In existing medical fluorescence observation equipment, fluorescence and anatomical images are recorded by a single camera. This setup can be used in fluorescence imaging of pPIX after administration of 5-ALA to reveal tumors. The choice of optical filters allows the camera to capture fluorescence in red wavelengths and a small portion of the excitation light in blue wavelengths. The excitation light reveals some of the tissue anatomy, while the fluorescence indicates the tumor. The optical filters used today have been standardized for pPIX imaging. This setup can also be used for other fluorophores such as ICG.

[0003] Although this type of fluorescence imaging has been of great help, especially in neurosurgical oncology, it still has serious drawbacks: the blue fluorescence excitation light used to visualize the tissue anatomy provides only poor visualization, as its intensity is very low and the anatomy is only shown in a monochromatic blue shade. In particular, bleeding is very difficult or even impossible to see under blue light. Therefore, this setup is suboptimal for real-time surgical guidance.

[0004] Other medical fluorescence observation equipment uses a grayscale camera to record fluorescence and a color camera to record white light images. In this setup, the fluorescence image and the white light image are mixed digitally, and the fluorescence is labeled with pseudo color. Here, the disadvantage is that pseudo color cannot accurately represent the color of fluorescence perceived by the human eye.

[0005] In view of the above, there is a need to provide an apparatus and method with improved representation of images to facilitate handling of medical viewing methods and help users migrate from different visualization techniques. Summary of the invention

[0006] This need is addressed by an image processor for a medical fluorescence observation device (such as a fluorescence microscope or a fluorescence endoscope) configured to:

[0007] - retrieving a digital white light color image of the object recorded in the first imaged spectrum;

[0008] - retrieving a digital fluorescent color image of the object recorded in a second imaged spectrum, the second imaged spectrum overlapping the fluorescence emission spectrum of at least one fluorophore and being different from the first imaged spectrum, both of which overlap the visible spectrum; and

[0009] - outputting at least one digital output color image from a set of digital output color images, the set comprising:

[0010] - a digital output color image generated by an image processor solely from the digital fluorescent color image;

[0011] - a digital output color image generated by an image processor solely from the digital white light color image; and

[0012] - a digital output color image generated by an image processor from a combination of the digital fluorescent color image and the digital white light color image;

[0013] The image processor is also configured as

[0014] - receiving a display selection signal from a selector device; and

[0015] - selecting at least one digital output color image from the set for output according to a display selection signal.

[0016] Additionally, this need is addressed by a computer-implemented image processing method for a fluorescence observation device (such as a fluorescence microscope or a fluorescence endoscope), the computer-implemented image processing method comprising the following steps:

[0017] - retrieving a digital white light color image of the object recorded in the first imaged spectrum;

[0018] - retrieving a digital fluorescent color image of the object recorded in a second imaged spectrum, the second imaged spectrum overlapping the fluorescence emission spectrum of at least one fluorophore and being different from the first imaged spectrum, both of which overlap the visible spectrum; and

[0019] - outputting at least one digital output color image from a set of digital output color images, the set comprising:

[0020] - a digital output color image generated by an image processor solely from the digital fluorescent color image;

[0021] - a digital output color image generated by an image processor solely from the digital white light color image; and

[0022] - a digital output color image generated by an image processor from a combination of the digital fluorescent color image and the digital white light color image;

[0023] The computer-implemented image processing method further comprises the following steps:

[0024] - receiving a display selection signal from a selector device; and

[0025] - selecting at least one digital output color image from the set for output according to a display selection signal.

[0026] The image processor and computer-implemented image processing method described above facilitate handling of a medical viewing device, as the device can be customized to display certain digital output color images in color schemes that may be more familiar to a user or that correspond to different image recording settings. A user-dependent display selection signal controls which digital output color image of the set is displayed.

[0027] The above solution can be further improved by the following features, which can be added and combined independently of each other and each feature has its own advantageous technical effect. Each of the following features can be used to improve one of the above methods and / or can be used to improve one of the above devices, regardless of whether a specific feature is mentioned only for the method or only for the device.

[0028] For example, the image processor may be configured to output at least two digital output color images in the output set simultaneously. This facilitates comparison of different image modalities and / or different color schemes or color conversions, thereby facilitating detection of certain features of an object.

[0029] At least two output color images may be displayed in a non-overlapping manner, for example, side by side or in a grid. The image processor may also be configured to determine and / or change the position of the output color images on the display based on a user selection (e.g., based on a signal from a pointer device such as a mouse, joystick, microphone, foot pedal, touchpad, or digital pen). For example, this may allow a user to manually overlay two or more images to, for example, directly compare the spatial extent of image features in different images.

[0030] Since the first and second spectra each represent only a portion of the visible spectrum, the colors of the materials and tissue types recorded in the digital white light color image and the digital fluorescent color image may appear unnatural and / or not clearly distinguishable given the perceptual characteristics of the human eye. To adjust the colors, a color conversion function may be used. Different color conversion functions may be used to highlight different materials or tissue types, or to affect the colors of the color space in a specific way. This will be explained in more detail below.

[0031] According to one advantageous aspect, the image processor may include a set of color conversion functions, each color conversion function being configured to map the color of an input pixel to a different color of an output pixel; and may be configured to calculate a digital output color image generated only from a digital fluorescent color image by applying at least one color conversion function from the set of color conversion functions to a fluorescent color image; to calculate a digital output color image generated only from a digital white light color image by applying at least one color conversion function from the set of color conversion functions to a white light color image; to calculate a digital output color image generated from a combination of a digital fluorescent color image and a digital white light color image by applying at least one color conversion function from the set of color conversion functions to a white light color image, a fluorescent color image, and / or a combination of a digital fluorescent color image and a digital white light color image. The selectability of such digital output color images facilitates interpretation of the image data.

[0032] Although a digital white light color image typically records the anatomical structure of a surgical scene by recording reflectivity, it can also be used to record fluorescence or to record both fluorescence and reflectivity of an object illuminated with a fluorescence excitation spectrum. A fluorescence color image can record the fluorescence of one or more fluorophores, but if fluorescence is not used, it can also be used to record reflectivity images of anatomical structures. By using a separate color camera for digital fluorescence on the one hand and for white light images on the other hand, a natural color output image with much more accurate color can be provided, regardless of what is recorded in the digital fluorescence color image and the digital white light color image. Since the digital output color image in this configuration mainly contains fluorescence, for example, fluorescence excitation light as illumination or fluorescence emission, the expression "digital fluorescence output color image" is used. Therefore, a digital fluorescence output color image is just a special case of a digital output color image that can be generated from a digital white light color image and a digital fluorescence color image.

[0033] Thus, according to one embodiment, a digital white light color image may represent a reflectance image of an object under illumination with the fluorescence excitation spectrum of at least one fluorophore, and / or a digital fluorescence color image may represent the fluorescence emission of at least one fluorophore. Typically, the fluorescence excitation spectrum is limited to wavelengths where the absorptivity of light is high, resulting in colored illumination. For example, illumination with the fluorescence excitation spectrum of 5-ALA / pPiX appears blue to the human eye. This combination corresponds to an image recorded by a single color camera that captures both fluorescence excitation and fluorescence emission.

[0034] According to another embodiment, the digital white light color image can represent a reflectance image of the object under white light illumination, and / or the digital fluorescence color image can represent the fluorescence emission of at least one fluorophore. As explained in more detail below, this combination allows for a faithful representation of natural-looking reflectance images (e.g., in anatomical procedures) and clearly visible fluorescence images of certain regions of interest.

[0035] According to yet another embodiment, the digital white light color image and the digital fluorescent color image may represent a reflectance image of an object under white light illumination. This combination, which will also be discussed in more detail below, allows for a more accurate analysis of image content to improve color representation and resolution.

[0036] In one embodiment, the image processor can be part of a medical fluorescence observation device. The medical fluorescence observation device can have a white light color camera for recording one or more digital white light color images and a fluorescence color camera for recording one or more digital fluorescence color images. The digital white light color image can include a plurality of first pixels and each first pixel can include a first set of color space coordinates. In addition, the first imaged spectrum can overlap with the fluorescence emission spectrum of at least one of the one or more fluorescent fluorophores. In the digital fluorescence color image, the one or more fluorescent fluorophores can be represented in the second imaged spectrum. The digital fluorescence color image can include a plurality of second pixels and each second pixel can include a second set of color space coordinates. The digital fluorescence output color image can include a plurality of output pixels. The image processor can be configured to calculate the color of the output pixel by applying a color conversion function to the input union of the first set of color space coordinates of the first pixel and the second set of color space coordinates of the second pixel.

[0037] A computer-implemented image processing method may include the steps of generating a digital fluorescent output color image from a digital white light color image and a digital fluorescent color image; and calculating the color of an output pixel by applying a color conversion function to an input union of a first set of color space coordinates of a first pixel and a second set of color space coordinates of a second pixel.

[0038] The image processor and computer-implemented image processing method described above allow for a clearer display of the fluorescence of the fluorophore. Both the digital white light color image and the digital fluorescence color image contain information about the fluorescence in different spectra (i.e., the first and second imaged spectra). The input union includes more color bands and therefore more spectral information than each of the digital white light color image and the digital fluorescence image individually. The additional spectral information arises from the difference between the first and second spectra.

[0039] The input union of the first and second sets corresponds to a multispectral image of one or more fluorescing fluorophores. By jointly processing the digital white light color image and the digital fluorescence image as a multispectral image, the use of increased spectral resolution allows for separate color conversions to be applied for different tissue types and fluorescence speeds. It also allows for more reliable detection of fluorescence and differentiation of different types of fluorescence, such as differentiation between fluorescence of different fluorophores, including differentiation between excited fluorescence of fluorophores added to the object and autofluorescence of materials inherent in the object.

[0040] The input union includes more color bands, and therefore more spectral information, than each of the digital white light color image and the digital fluorescent image individually. The additional spectral information results from the first and second spectra being different.

[0041] The input union of the first and second sets corresponds to a multispectral image of one or more fluorescing fluorophores. By jointly processing the digital white light color image and the digital fluorescence image as a multispectral image, the use of increased spectral resolution allows for separate color conversions to be applied for different tissue types and fluorescence speeds. It also allows for more reliable detection of fluorescence and differentiation of different types of fluorescence, such as differentiation between fluorescence of different fluorophores, including differentiation between excited fluorescence of fluorophores added to the object and autofluorescence of materials inherent in the object.

[0042] The image processor may be configured to perform any of the processing steps described above.The terms "image processor" and "data processing device" may be used synonymously.

[0043] For example, one embodiment may relate to an image processor for a medical fluorescence observation device (such as a fluorescence microscope or a fluorescence endoscope), the processor being configured to retrieve a digital white light color image of an object recorded in a first imaged spectrum, the digital white light color image comprising a plurality of first pixels, each first pixel comprising a first set of color space coordinates in a first set of color bands; retrieve a digital fluorescence color image of an object recorded in a second imaged spectrum, the digital fluorescence color image comprising a plurality of second pixels, each second pixel comprising a second set of color space coordinates in a second set of color bands; wherein the second imaged spectrum overlaps with a fluorescence emission spectrum of at least one fluorophore, wherein the second imaged spectrum is different from the first imaged spectrum, and wherein both the first and second imaged spectra overlap with a visible spectrum; generate a digital output color image from the digital white light color image and the digital fluorescence color image, the digital output color image comprising a plurality of output pixels; wherein the image processor is configured to calculate the color of the output pixel by applying a color conversion function to an input union of a first set of color space coordinates of the first pixel and a second set of color space coordinates of the second pixel, wherein the application of the color conversion function depends on the color space coordinates in the input union.

[0044] Another embodiment may relate to a computer-implemented image processing method for a fluorescence observation device (such as a fluorescence microscope or a fluorescence endoscope), the computer-implemented image processing method comprising the following steps: retrieving a digital white light color image of an object recorded in a first imaged spectrum, the digital white light color image comprising a plurality of first pixels, each first pixel comprising a first set of color space coordinates in a first set of color bands; retrieving a digital fluorescence color image of an object recorded in a second imaged spectrum, the digital fluorescence color image comprising a plurality of second pixels, each second pixel comprising a second set of color space coordinates in a second set of color bands; wherein the second imaged spectrum overlaps with a fluorescence emission spectrum of at least one fluorophore, wherein the second imaged spectrum is different from the first imaged spectrum, and wherein both the first and second imaged spectra overlap with a visible spectrum; generating a digital output color image from the digital white light color image and the digital fluorescence color image, the digital output color image comprising a plurality of output pixels; wherein the color of the output pixel is calculated by applying a color conversion function to an input union of a first set of color space coordinates of the first pixel and a second set of color space coordinates of the second pixel, wherein the color conversion function is applied according to the color space coordinates in the input union.

[0045] At least one fluorescent fluorophore may be contained in the object imaged by the medical viewing device and in particular the white light color camera and the fluorescence color camera.The object may contain or consist of biological tissue.

[0046] At least one of the digital white light color image and the digital fluorescence color image may contain or equivalently represent a reflectance image of the object illuminated by light consisting of or including the fluorescence excitation spectrum. Preferably, the reflectance image is contained in the digital white light color image in addition to the fluorescence of the at least one fluorophore. Thus, the first imaged spectrum may overlap with the fluorescence excitation spectrum.

[0047] The image processor and computer-implemented method described above allow different color conversions to be performed for different colors in an object. By applying a color conversion function based on the input union of two sets of color space coordinates, more accurate color conversion schemes are possible. For example, color-dependent conversions allow different color schemes to be assigned to different fluorescent colors and reflected colors under fluorescent excitation illumination.

[0048] According to another aspect, the computer-implemented image processing method may include a step of demosaicing at least one of the digital white light color image and the digital fluorescent color image. By demosaicing, color artifacts may be avoided. Preferably, demosaicing is performed before applying any color conversion function.

[0049] The computer-implemented image processing method may further include the step of normalizing at least one of the digital white light color image and the digital fluorescent color image. Normalization facilitates joint processing of the digital white light color image and the digital fluorescent color image.

[0050] According to one aspect, a computer-implemented image processing method may include the step of registering a digital white light color image with a digital fluorescent color image. By registering, image features present in both the digital white light color image and the digital fluorescent color image are represented by the same size and orientation in each image. After registration, the digital white light color image and the digital fluorescent color image match each other. Therefore, pixels at the same position in the two registered images are corresponding pixels.

[0051] In particular, when the image processor retrieves the digital white light color image and the digital fluorescent color image, they may already be registered relative to each other.Alternatively, the image processor may be configured to register the digital white light color image and the digital fluorescent color image relative to each other.

[0052] The registration should be performed before applying the color conversion function, and preferably after demosaicing. In the registered digital white light and fluorescent images, a first pixel comprising a first set of color space coordinates and a second pixel comprising a second set of color space coordinates are corresponding pixels, and / or an output pixel and the first and second pixels are corresponding pixels.

[0053] In one embodiment, the fluorescence medical viewing device may be operated in various operating modes in which different image modalities are assigned to digital fluorescence color images and digital white light color images in addition to or instead of two images representing fluorescing fluorophores.

[0054] For example, in one such operating mode, a fluorescence color camera may record a reflectance color image under the same illumination as the digital white light color image, particularly a standard illuminant. Such a fluorescence color image may complement the reflectance image recorded by the white light color camera. Thus, the fluorescence color camera is not limited to recording only fluorescence images. However, if the hardware is not changed (e.g., by exchanging one or more filters), then the fluorescence camera may be limited to recording only images in the second imaged spectrum. In another operating mode, the digital white light color image may represent a reflectance image of an object illuminated by a standard illuminant, while the digital fluorescence color image may represent one or more fluorescing fluorophores. In all of these operating modes, the generation of the output color image may be the same.

[0055] According to another advantageous embodiment of the computer-implemented image processing method and / or the image processor, the first imaged spectrum and the second imaged spectrum are complementary to each other, i.e., do not overlap. In this configuration, crosstalk between fluorescence and white light is avoided and an exact reproduction of the fluorescence of at least one fluorophore on the one hand and the reflectivity of the object on the other hand is maintained. In addition, by having complementary spectra in the digital fluorescence color image and the digital white light color image, respectively, a joint processing of the input union of the two images to form a multispectral image is facilitated, since there is no redundant special information.

[0056] The digital white light color image may be represented in a first color space using at least three first color bands. The digital fluorescent color image may be represented in a second color space including at least three second color bands. The first and second color spaces are preferably identical, so that the first and second color bands are also identical. However, this need not be the case. The first and second color spaces may also be different.

[0057] The digital output color image may be generated in a third color space comprising at least three third color bands.The third color space may be the same color space as the first and / or second color space, or may be a different color space.

[0058] Any of the first, second and / or third color spaces may be an RGB color space, comprising three color bands R, G, B, or any other color space, such as HSV, LAB, CIELAB, XYZ, CMYK, a multispectral color space or a hyperspectral color space, i.e., employing more than three color bands. The term "color" is used to denote a coordinate in a color space, i.e., a tuple of color space coordinates in a color band of a color space, wherein the color band does not necessarily denote chromaticity, but may also denote other aspects of color, such as saturation, luminance or lightness, as in HSV, LUV or LAB color spaces. In this regard, the term "color band" is used synonymously with a color space dimension in this document.

[0059] According to one aspect of a computer-implemented image processing method and / or image processor, a first color band and a second color band are mapped to a third color band of a digital output color image using a color conversion function. The color conversion function may be non-linear, but is preferably a linear transformation. The color conversion function may include a color conversion matrix or other type of linear function. In order to map the first and second color bands to the third color band, a color conversion matrix may be applied to the first color band and the second color band. Preferably, each color band in the first (set) color band and each color band in the second (set) color band are input as separate color bands into the linear transformation. In particular, each color band in the first color band and each color band in the second color band may be input into the linear transformation simultaneously.

[0060] The color conversion function may be stored in a memory of the image processor or the medical fluorescence viewing device.As explained further below, more than one color conversion function may be provided.

[0061] The color conversion matrix may have a dimension of number X in one direction and a dimension of number Y in another direction. Number X may be the sum of the number of first color bands (i.e., the number of color bands in the first set of color bands) and the number of second color bands (i.e., the number of color bands in the second set of color bands). Number Y may be the number of color bands in the third color space. In other words, one dimension of the color conversion matrix may correspond to the sum of the dimensions of the color spaces of the digital white light color image and the digital fluorescent image. Another dimension of the color conversion matrix may correspond to the dimension of the color space of the digital output color image. For example, if the first, second, and third color spaces are each RGB color spaces, then the color conversion matrix may have a dimension of 6×3 or 3×6 or 3×3.

[0062] A color conversion matrix may be produced by a combination of various matrices applied successively to a set of unions.

[0063] For example, the first 6×1 intermediate matrix can be applied to the digital white light color image and the digital fluorescence color image. This can be seen as reducing the color information in the union set to intensity values. Such intensity values ​​can, for example, correspond to the fluorescence intensity of a specific fluorophore. The second 1×3 intermediate matrix can then be used to map the intensity values ​​to colors in the digital output color image. The application of these two matrices corresponds to the application of the 6×3 matrix. Of course, these two intermediates.

[0064] Therefore, the third color band (i.e., the color band of the digital output color image) can be generated by a linear combination (in particular, addition) of the color band of the digital white light color image and the color band of the digital fluorescent color image. This linear combination or addition can be regarded as a color conversion function, because the two colors of the pixels in the digital white light color image and the digital fluorescent color image are mapped to the color of the pixel in the output image.

[0065] According to one aspect, a digital fluorescent color image and a digital white light color image are jointly processed into a multispectral image, the color bands of the multispectral image being formed or composed of the complementary color bands of the digital fluorescent color image and the digital white light color image. This results in improved color accuracy, especially if the fluorescent image represents reflected white light that has been recorded in a second imaged spectrum, and can therefore supplement the white light information in the white light color image. Using this method, the white light information is contained in the color bands of the digital fluorescent color image, so the color granularity is finer than that provided by the digital white light color image alone. If the digital fluorescent color image represents fluorescent emission, then processing the digital white light color image and the digital fluorescent color image into a multispectral image represents a true representation of the object under reflected white light and fluorescence. In particular, if the digital fluorescent color image represents a reflectance image of the object under the same illumination as the digital white light color image, then additional spectral information is available.

[0066] According to another advantageous embodiment, the first imaged spectrum may comprise a first sub-band in a color band of a color space and the second imaged spectrum may comprise a second sub-band in this color band, wherein the first and second sub-bands are preferably complementary to each other. The color space may be the color space of a digital white light color image, the color space of a digital fluorescent color image or preferably the color space of a digital output color image. It is preferred that the sub-bands of a color band complete the respective color band together. This ensures that no useful spectral information is lost.

[0067] There may be first and second sub-bands in more than one color band of the corresponding color space.The sub-bands in a color band may be further divided into a plurality of different wavelength bands separated from each other.

[0068] Preferably, the second imaged spectrum includes IR wavelengths. This allows for the capture of fluorescence emissions in the near infrared, or, if digital fluorescence color images are not used to capture fluorescence, additional information about the anatomical structure can be added. Additionally, IR wavelengths can contain spectral information that allows for better discrimination of specific tissue types and the specific color conversion functions assigned to them.

[0069] The image processor may be configured to generate the output pixel by simultaneously applying the color conversion function to both the first pixel and the second pixel.

[0070] According to an embodiment, the image processor may include at least two different color conversion functions. The image processor may be configured to select one of the at least two different color conversion functions according to the input union, or in the method, one of the at least two different color conversion functions may be selected according to the input union.

[0071] Different color conversion functions can be used for different tissue types, where the tissue type includes a predetermined color range. The predetermined color range can correspond to one or more regions in a color space, where each region is characterized by a set of color space coordinates. The range of colors can be determined for each tissue type during a calibration process.

[0072] In order to assign different color conversion functions to different tissue types, the image processor may include at least two sets of different target unions. Each of the at least two different color conversion functions may be assigned to a different set of the at least two sets of different target unions. Different target unions of the set correspond to colors to be converted using the color conversion function assigned to the set. Each set of target unions may, for example, represent different tissue types and / or fluorescence emissions of different fluorophores, and each target union in the set may correspond to a color assigned to this tissue type or fluorescence during calibration.

[0073] In one embodiment, to determine whether the input union is included in the set of predetermined target unions, the image processor may be configured to compare the input union with a target union in the set of different target unions.

[0074] In one embodiment, the image processor may be configured to select a color conversion function from a set of at least two different color conversion functions, which is assigned according to a union of a first set of color space coordinates of the first pixel and a second set of color space coordinates of the second pixel.

[0075] The processor may be configured to select a color transform assigned to a set of target unions, the set containing the input unions. This color function may then be applied to the input unions.

[0076] In one embodiment, the image processor includes at least one of the following:

[0077] - a predetermined set of target unions, wherein at least one target union corresponds to a color representing oxygenated blood; such a set may represent tissue containing oxygenated blood, for example, arterial tissue;

[0078] - a predetermined set of target unions, wherein at least one target union corresponds to a color representing deoxygenated blood; such a set may represent tissue containing deoxygenated blood, for example, venous tissue;

[0079] - a predetermined set of target unions, wherein at least one target union corresponds to a color representative of living grey brain matter;

[0080] - a predetermined set of target sets, wherein at least one target set corresponds to a color representative of living white brain matter;

[0081] - a predetermined set of target sets, wherein at least one target set corresponds to a color representative of the fluorescence of a fluorophore;

[0082] - a predetermined set of target sets, wherein at least one target set corresponds to a color representative of autofluorescence of biological tissue;

[0083] - a predetermined set of target unions, wherein at least one target union corresponds to a natural color representative of the color of biological tissue illuminated by a standard illuminant.

[0084] The target union maps the colors represented in the input union set to different output colors. The target union in the set can be determined by, for example, spectral decomposition, principal component analysis, and / or color calibration.

[0085] For example, a set of target unions can be used to map the colors in a reflectance image of an object under illumination of a fluorescence excitation spectrum to (more) natural colors. The fluorescence excitation spectrum contains or preferably consists of a wavelength shorter than the fluorescence emission wavelength of at least one fluorophore. In addition, the fluorescence excitation spectrum preferably has a peak at a wavelength or in a region of wavelengths where the light absorption of one or more fluorophores is maximum. This results in a strong coloration of the illumination of the object by the fluorescence excitation spectrum. For example, if 5-ALA / pPiX is used as a fluorophore, the fluorescence excitation spectrum will have a strong blue hue, and the reflectance image of the object under such illumination will appear blue. Using color conversion functions, in particular a set of target unions, the blue hue can be mapped to a color that appears more natural.

[0086] According to one embodiment, the image processor can be configured to be selectively operated in one of a first operating mode and a second operating mode, the first operating mode including an image combination step, and the second operating mode including a color conversion step and an image combination step; wherein, in the image combination step, the image processor is configured to generate a digital output color image by combining a digital white light color image and a digital fluorescent color image; and wherein, in the color conversion step, the image processor is configured to apply at least one color conversion function from a group comprising: a first color conversion function applied to the digital white light color image, and a second color conversion function applied to the digital fluorescent color image.

[0087] A computer-implemented image processing method may be configured to be selectively executed in one of a first operating mode and a second operating mode; wherein, in the first operating mode, an image combination step is performed, and in the second operating mode, a color conversion step and an image combination step are performed; wherein, in the image combination step, a digital output color image is generated by combining a digital white light color image and a digital fluorescent color image, and in the color conversion step, at least one color conversion function from a group comprising: a first color conversion function applied to the digital white light color image and a second color conversion function applied to the digital fluorescent color image is applied.

[0088] By providing two operating modes which may be selected, it is possible to provide in a first operating mode a digital output color image corresponding to the image seen through the eyepiece of the medical viewing device.

[0089] By providing two color conversion functions that operate separately on the digital white light color image and the digital fluorescent color image, it is possible to separate the color conversion of the digital white light image from the color conversion applied to the digital fluorescent color image. This allows each of the two images to be optimized independently for viewing before combining them to form a digital output color image. According to one aspect, the first and / or second color conversion function is configured to shift a (recorded) color to a (different) color in a color space in a corresponding one of the digital white light color image and the digital fluorescent image. The shift is predetermined by the color conversion function with respect to at least one of an amount and a direction of the shift in the color space.

[0090] For example, in the first and second imaged spectra, the natural color of oxygenated or deoxygenated blood may only be captured incompletely. This is particularly true if the reflection spectrum of blood extends to wavelengths that are not recorded in the first or second imaged spectrum, respectively. Therefore, if blood is recorded in a digital white light color image or a digital fluorescence image, its color will look unnatural, making it more difficult for a user or even an automatic image processing device to identify the blood vessels. By using the first and / or second color conversion function, the (recorded) color (here the blood color) can be shifted to a more natural color in the corresponding digital white light or fluorescence color image.

[0091] In another example, by using different color conversion functions, the recorded colors of oxygenated blood and deoxygenated blood can each be shifted differently. Again, both colors are incompletely represented in the first and / or second imaged spectra. The recorded color of deoxygenated blood can be shifted further toward blue, while the color of oxygenated blood is shifted further toward red. This can require different amounts and / or directions of shifts of the two recorded colors in color space; a color conversion function can shift one color differently from another color in color space with respect to at least one of the amount and direction of the shift.

[0092] The above examples are of course applicable to any other type of tissue, such as liquid tissue (eg, lymph), or solid tissue (eg, bone, muscle and / or nerve tissue).

[0093] The converted color can correspond to a natural color, a pseudo color (such as a neon color), or a "hyper-real" color, wherein at least one of the color appearance parameters (i.e., hue, chroma, saturation, value, and brightness) is modified compared to the natural color to increase visibility and contrast.

[0094] Natural or true colors in this context correspond to colors that a CIE standard observer would perceive under a standard illuminant (eg, CIE illuminants A, B, or C, or any other standardized white light illumination).

[0095] The first and / or second color conversion functions can be configured to convert colors in the corresponding digital white light or fluorescent color image to colors that are not in the corresponding first and / or second imaged spectrum. Thus, the color conversion function can be used to correct for defects in imaging due to limited first and / or second imaged spectrum.

[0096] In another example, the first and / or second color conversion function can be configured to shift the white point of the digital white light and / or fluorescent color image to a predetermined position, in particular to another white point. The white point to which the white point is shifted can in particular correspond to a standardized white point defined in a standard illuminant, such as a CIE illuminant. This allows the color space to be centered.

[0097] In addition, the first and / or second color conversion functions can be configured to white balance or color balance the corresponding digital white light and / or fluorescent color image. In color balancing, the intensity of the color is adjusted to correctly render at least some colors, especially neutral colors. In white balancing, the color is adjusted so that white objects appear white rather than colorful.

[0098] According to another aspect, the first and / or second color conversion functions can be configured to expand a continuous or segmented region in the color space into a larger region in the corresponding digital white light and / or fluorescent color image. Such a color conversion function allows for better visualization of subtle differences between adjacent colors as they are further separated. For example, if the region of fluorescent colors of a particular fluorophore is expanded, then subtle differences in fluorescence become more visible. Of course, the expansion of the region can be accompanied by an appropriate color shift so that the expansion of the region occurs near a natural color or a non-natural color.

[0099] According to yet another aspect, the first and / or second color conversion function may be configured to shift all colors in the color space of the corresponding digital white light and / or fluorescent color image.

[0100] Any of the above color conversion functions may be combined into a single color conversion function, and / or any of the above color conversion functions may be applied sequentially to a corresponding one of the digital white light and / or fluorescent color images.

[0101] It may be further advantageous if the first and / or second color conversion function is configured to be applied depending on the color of the corresponding pixel in the digital white light color image and / or the digital fluorescent color image. This generally allows different colors to be processed in different ways.

[0102] According to another aspect, a plurality of different first and / or second color conversion functions may be provided, and the processor may be configured to determine the first and / or second color conversion function of the plurality of different first and / or second color conversion functions applied to the pixel according to the color of the pixel. The plurality of first and / or second color conversion functions may include any one of the above-mentioned color conversion functions. According to this aspect, targeted color-related modifications may be applied to the corresponding digital white light and / or fluorescent color images. For example, the area around some colors in the digital white light color image may be expanded to make the color differences more visible. Another color, such as the color of bones and / or nerves, may be simply shifted to make it more natural. Another color and therefore the tissue associated with this color may be highlighted by converting it into a neon color.

[0103] The above-described computer-implemented image processing method may be performed when executing a method for operating a medical fluorescence viewing device. The image processor as described above may be part of a medical fluorescence viewing device, such as a fluorescence microscope or a fluorescence endoscope, in particular a fluorescence microscope or a fluorescence endoscope configured for surgery (such as neurosurgery, orthopedic surgery and / or ophthalmic surgery). The fluorescence microscope may also be a laboratory microscope used, for example, for biopsy.

[0104] A medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope may include an image processor configured to perform any of the above steps. In addition, the medical fluorescence observation device may include a fluorescence color camera configured to record digital fluorescence color images and a white light color camera for recording digital white light color images.

[0105] Since the fluorescence can have a lower intensity, the fluorescence color camera can be operated with a higher integration time.

[0106] The method for operating a medical fluorescence observation device may include the steps of recording a digital white light color image using a white light color camera and recording a digital fluorescence color image using a digital fluorescence color camera. The medical fluorescence observation device may include a digital fluorescence color camera and a white light color camera. In particular, both the fluorescence color camera and the white light color camera can record images of one or more fluorescent fluorophores. The white light color camera can also record a reflectance image of an object illuminated by a fluorescence excitation spectrum.

[0107] To reduce post-processing, the fluorescence color camera and the white light color camera may have overlapping coaxial fields of view and / or preferably identical fields of view.

[0108] The medical fluorescence observation device can be a stereoscopic observation device or a monoscopic observation device. In one embodiment, a digital fluorescence color camera and a white light color camera can be provided in each stereoscopic channel. Alternatively, in a monoscopic medical fluorescence observation device, only a single white light color camera and a single digital fluorescence color camera can be provided.

[0109] According to one embodiment, the medical fluorescence observation device may include a selector device configured to be operated by a user of the medical fluorescence observation device and to generate a display selection signal.

[0110] The selector device may be a hardware device (such as a dial, knob, or switch), a software device (such as an interactive graphical element that may be similar to a dial, knob, or switch), or a combination of software and hardware devices. The selector device may be used to generate a display selection signal. The selector device may be configured to generate a plurality of display selection signals, each of which may be uniquely assigned to a particular color conversion function or a particular combination of color conversion functions and a particular combination of a digital fluorescent color image and a digital white light color image.

[0111] The medical fluorescence observation device may include at least one display connected to at least one image processor, the display being configured to receive and display at least one digital output color image output by the image processor.

[0112] In an alternative arrangement of a stereoscopic medical fluorescence viewing device, a fluorescence color camera may be provided in one stereo channel and a white light color camera may be provided in another stereo channel. In this arrangement, the fluorescence color camera may be configured to record a (white light) reflectance image in the second imaged spectrum. For the (white light) reflectance image, this arrangement provides a stereoscopic view, although strictly speaking the stereoscopic view is limited to the overlapping portion of the first and second imaged spectra, but this is rarely noticed by a human observer. In the case of recording fluorescence emissions, the fluorescence color camera provides a monoscopic view of the fluorescence emissions of the object, while the white light color camera provides a monoscopic view of the reflectance of the object.

[0113] The medical fluorescence observation device may include an optical color separation component configured to split the light entering the color separation component into a first imaged spectrum and a second imaged spectrum. The color separation component may include an optical element (such as a beam splitter, in particular a dichroic beam splitter), and / or an optical filter (such as a fluorescence filter that blocks light other than fluorescence emission and a white light filter that blocks fluorescence emission).

[0114] The medical fluorescence viewing device may also include an illumination assembly, which is preferably adjustable, for example, by including a plurality of differently colored LEDs or OLEDs or other light sources that emit light in a plurality of different spectral bands and which can be individually turned on and off.

[0115] In order to facilitate post-processing and ensure that the color space coordinates of the recorded digital white light image and the digital fluorescent color image are comparable to each other, it is preferred to record the digital fluorescent color image and the digital white light color image simultaneously and / or using the same exposure time and / or using the same gain and / or the same white balance and / or color compensation. Moreover, the gain of the digital fluorescent color camera and the gain of the digital white light color camera can be kept at a constant ratio, but in other cases automatic adjustment is allowed. Among these possibilities, it is most preferred that the white light color camera and the fluorescent color camera are synchronized in at least one aspect of exposure time and gain. However, the fluorescent color camera can be operated at a higher integration time. This allows compensation for low fluorescence intensities. The white light color camera and / or the fluorescent color camera are preferably CCD or CMOS cameras. The fluorescent camera and the white light color camera are preferably exactly the same.

[0116] According to another aspect, fluorophore-related sets of different color conversion functions can be provided, for example, stored in an image processor or a medical fluorescence device. Each fluorophore-related set represents a different combination of the first and second imaged spectra, i.e., a different set of filters used in the color separation component, which is necessary if different fluorophores or combinations of different fluorophores are used. Examples of fluorophores used in this context are 5-ALA / pPIX, fluorescein and ICG. Each of these fluorophores requires excitation at a different wavelength and also emits fluorescence at a different wavelength. Therefore, in a surgical environment using ICG, the first imaged spectrum and the second imaged spectrum will be different from a surgical environment using 5-ALA / pPIX, requiring a different color conversion matrix. Therefore, for each case of the fluorophore, the sub-bands of the first and second imaged spectra in the color band of the corresponding color space are different.

[0117] The fluorophore-related set can be automatically or manually selected from a plurality of such sets according to the fluorophore used or the setting of the color separation component. For example, the medical fluorescence observation device can be configured to automatically detect the configuration or setting of the color separation component and select the color conversion matrix according to this setting. For example, if the optical filter of the color separation component used for using ICG is replaced with that used for using 5-ALA / pPIX, the medical fluorescence observation device can automatically select the color conversion matrix for this 5-ALA / pPIX.

[0118] The claimed subject matter also relates to a computer-readable medium and a computer program comprising instructions for causing a computer to perform the computer-implemented image processing of any one of the above-described embodiments.

[0119] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0120] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0121] Below, the present invention will be described exemplarily with reference to the embodiments and with reference to the accompanying drawings. The combination of features shown in the embodiments should not be considered restrictive. For example, features of the embodiments that have technical effects such as those explained above but are not required in a specific application can be omitted. On the contrary, if the technical effect associated with this specific feature is required in a specific application, the above-mentioned features that are not part of the embodiments described below can be added. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Throughout the description and drawings, the same reference numerals are used to indicate elements that correspond to each other in function and / or structure.

[0123] In the accompanying drawings,

[0124] Figure 1 A schematic representation of a medical fluorescence viewing device is shown;

[0125] Figure 2 A schematic representation of the combination of a digital white light color image and a digital fluorescence spectrum color image to obtain a digital output color image is shown;

[0126] Figure 3 A schematic representation of jointly processing a digital white light color image and a fluorescent color image to obtain a digital output color image is shown;

[0127] Figure 4 A schematic representation of tissue reflectance spectra and spectral sensor sensitivity is shown;

[0128] Figure 5 shows a schematic representation of color conversion achieved by various color conversion functions;

[0129] Figure 6 shows a schematic representation of a method for obtaining a digital output color image from a combination of a digital white light color image and a digital fluorescence spectrum color image;

[0130] Figure 7 shows a schematic illustration of a system configured to perform a method for obtaining a digital output color image from a combination of a digital white light color image and a digital fluorescence spectrum color image;

[0131] Figure 8shows a schematic illustration of applying one or more color conversion functions based on an input union of color bands of pixels of a digital white light color image and pixels of a fluorescent color image;

[0132] Fig. 9 shows schematic illustrations of different digital output color images obtained by respectively applying different color conversion functions to the same digital white light color image and the same digital fluorescent color image;

[0133] Fig.10 Shows the display Fig. 9 Schematic illustration of a display of different combinations of digital output colors. DETAILED DESCRIPTION

[0134] First, refer to Figure 1 Exemplary embodiments of the present invention are described.

[0135] Figure 1 A medical fluorescence observation device 100 is schematically shown. The medical fluorescence observation device 100 may be a fluorescence microscope or a fluorescence endoscope, the difference between a microscope and an endoscope being that in an endoscope (not shown) the object 106 is viewed via an optical fiber which is brought close to the object 106 to be investigated, for example by being inserted into the body, whereas in a microscope the objective lens 174 is directed towards the object. Figure 1 The medical fluorescence observation device 100 is a microscope, but the following description is also applicable to an endoscope. The medical fluorescence observation device 100 may be a medical fluorescence observation device used in surgery. The medical fluorescence observation device 100 may also be a medical fluorescence observation device used in a laboratory, such as a laboratory microscope. The object 106 to be studied may consist of or include a biological tissue 107.

[0136] The object 106 may include one or more fluorophores 116, 118. At least one fluorophore may be a fluorophore naturally contained in the object. For example, bones and blood contain fluorophores. At least one fluorophore may also be added to the object 106, for example, by injecting it into biological tissue 107. Examples of fluorophores that may be added to the object 106 are ICG, fluorescein and / or 5-ALA. 5-ALA is synthesized in cells as pPIX / pPIX.

[0137] The medical viewing device 100 is a fluorescence device. This means that the medical fluorescence viewing device is configured to view, record and preferably also stimulate the fluorescence of one or more fluorophores 116 , 118 .

[0138] The medical fluorescence observation device 100 may be a stereoscopic device, such as Figure 1As shown in the example. Therefore, it can include two identical subassemblies 101L and 101R, for each of the two stereo channels. Since the two subassemblies 101L, 101R are identical in function and structure, the following description focuses on the right subassembly 101R, but is completely applicable to the left stereo channel 101L.

[0139] The medical fluorescence observation device 100 may alternatively be a monoscopic device. In this case, only one of the two subassemblies 101L, 101R may exist. Therefore, for the monoscopic medical fluorescence observation device 100, the following description is also applicable.

[0140] The medical fluorescence viewing device 100 can be used to generate at least one digital white light color image 114 that represents a reflectance image of the object 106 in the visible light range. The visible light range or visible spectrum includes wavelengths from about 310 nm to about 1100 nm, or about 380 nm to 750 nm, or about 450 nm to about 700 nm. The fluorescence spectrum or, if more than one fluorophore is used, the fluorescence spectrum of the fluorophores 116, 118 is preferably omitted from the spectrum recorded in the digital white light color image 114. This ensures that if fluorescence is present, only reflected light is included in the digital white light color image 114. For example, if 5-ALA / pPIX is used as a fluorophore, the fluorescence spectrum of about 625 nm to about 650 nm may not be recorded in the digital white light color image 114.

[0141] If specific wavelengths of light are used to excite fluorescence, spectra including or consisting of these wavelengths may also not be recorded or represented in the digital white light color image 114. For example, if 5-ALA / pPIX is used as the fluorophore, fluorescence may be excited by illuminating the object 106 with a wavelength between about 380 nm and about 450 nm. For fluorescein and ICG, as well as other fluorophores, different but known ranges for the excitation and emission spectra apply compared to 5-ALA / pPIX.

[0142] By not recording the fluorescence excitation spectrum and the fluorescence emission spectrum, the digital white light color image 114 preferably represents the reflectance of the object 106, that is, represents a white light image of the object 106, just as a human observer would see it. Therefore, the digital white light color image 114 can be regarded as a true color image. However, in some embodiments, the emission spectra of some fluorophores or at least part of the emission spectra can also be recorded in the digital white light color image 114.

[0143] The digital imaging system 102 may also be used to generate a digital fluorescent color image 112 of the object 106. The digital fluorescent color image 112 represents the fluorescent emission of one or more fluorophores 116, 118. Thus, the digital fluorescent color image 112 preferably does not record any wavelengths outside of the emission spectrum or spectra of the one or more fluorophores.

[0144] Both the digital white light color image 114 and the digital fluorescent color image 112 are color images. They are recorded using at least three color bands or, equivalently, the primary colors of a color space. For example, the digital white light color image 114 and the digital fluorescent color image 112 can be recorded in an RGB color space using three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescent color image 112 can be recorded in different color spaces, respectively, and / or represent multi-spectral or hyperspectral color images. The digital white light color image 114 and the digital fluorescent color image 112 do not need to be recorded in the same color space, but this is preferred.

[0145] The digital white light color image 114 and the digital fluorescent image 112 include pixels 150. In a color space such as the RGB color space, each color of the pixel 150 is represented by a triple of three integer numbers, each integer number indicating the intensity of one of the primary colors R, G, B herein. For example, the most intense red may be indicated by the triple {255,0,0}. The most intense green may be indicated by {0,255,0}, and the most intense blue may be indicated by {0,0,255}. Thus, the RGB color space is a three-dimensional space, while the CMYK color space would be a four-dimensional space. Colors may be viewed as points in the color space with color space coordinates such as {0,0,255}. A multispectral or hyperspectral color space with n color bands would accordingly produce an n-dimensional color space, and each color would be represented by an n-tuple of values.

[0146] The spectrum recorded and represented in the digital white light color image 114 (the first imaged spectrum) and the spectrum recorded in the digital fluorescent color image 112 (the second imaged spectrum) are preferably complementary to each other, i.e., do not overlap except for unavoidable filter leakage. Preferably, together they represent the complete visible light spectrum.

[0147] More specifically, the medical observation device 100 may include a digital imaging system 102 for generating a digital fluorescent color image 112 and a digital white light color image 114. The digital imaging system 102 may include a white light color camera 110 and a fluorescent color camera 111.

[0148] The white light color camera 110 is configured to record digital white light color images 114. In particular, the white light color camera 110 may be configured to generate a stream of digital white light color images 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images across the entire visible spectrum within the wavelength ranges indicated above. The white light color camera 110 may be a CCD, CMOS, or multispectral or hyperspectral camera.

[0149] The fluorescence color camera 111 is configured to record a digital fluorescence image 112. In particular, the fluorescence camera 111 may be configured to generate a stream of digital fluorescence color images 112 in the form of a digital video stream. The fluorescence color camera 111 may be configured to record the digital fluorescence color image 114 only in one or more fluorescence spectra of at least one fluorophore 116, 118. The fluorescence camera 111 may be configured to record the digital fluorescence image only in one or more narrow light bands. These narrow bands should overlap with one or more fluorescence spectra of the one or more fluorophores 116, 118 whose fluorescence is to be recorded. Preferably, the fluorescence spectra of the fluorophore 116 and the second fluorophore 118 are at least partially separated, preferably completely separated, so that the fluorescence camera 111 can record light in two separate fluorescence bands that are spaced apart from each other.

[0150] Fluorescence color camera 111 may be a CCD, CMOS or multispectral or hyperspectral camera. Preferably, white light color camera 110 and fluorescence color camera 111 are of the same type, but this is not required. Since fluorescence usually has very low intensity, fluorescence color camera 111 may have a higher integration time.

[0151] The respective fields of view 184 of the cameras 110, 111 are preferably aligned, or even coincident and coaxial. Therefore, it is preferred that the cameras 110, 111 provide exactly the same field of view 184 with exactly the same viewing angle and focal length. This results in exactly the same representation of the object 106 in the images 112, 114 generated by the different cameras 110, 111. The two cameras 110, 111 can use the same objective lens 174.

[0152] If a match of perspective and field of view cannot be generated optically, a match can be generated through image processing by applying a matching or registration routine to the digital images 112, 114, as explained further below. Registration can also be performed if the cameras 110, 111 have exactly the same perspective and field of view.

[0153] It is preferred that the two cameras 110 , 111 are operated synchronously. Specifically, the exposure times may be synchronized. Therefore, the medical fluorescence observation apparatus 100 may be configured to generate a digital white light color image 114 and a digital fluorescence image 112 simultaneously.

[0154] Preferably, the gains of the two cameras 110, 111 are synchronized, i.e., adjusted simultaneously in both cameras 110, 111. Furthermore, even if the gain is changed, the ratio of the gain applied in camera 110 to the gain applied in camera 111 may be constant. Gamma correction and color adjustment or white balance may be turned off or kept constant.

[0155] Any of the above measures facilitate comparing, jointly processing, and / or combining the two images 112 , 114 .

[0156] In order to separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescent color image 112, that is, to separate the reflection spectrum from the fluorescent spectrum, the medical observation device 100 may include an optical color separation component 176. The color separation component 176 may include an optical element such as a beam splitter 192, which may be a dichroic. The color separation component 176 may also or alternatively include an optical white light filter 188 and / or an optical fluorescence filter 190.

[0157] The fluorescence filter 190 is preferably configured to transmit light within the fluorescence spectrum or spectrums of the one or more fluorophores 116 , 118 and to block light outside of the one or more fluorescence spectrums.

[0158] The fluorescence filter 190 can be configured as a bandpass filter including one or more passbands. Each passband should overlap with the fluorescence emission spectrum of the corresponding fluorophore 116, 118 whose fluorescence is to be recorded. Since the fluorescence filter 190 is located in the optical path between the beam splitter 192 and the fluorescence camera 111, only wavelengths in the passband of the fluorescence filter 190 are transmitted to the fluorescence camera 111.

[0159] The white light filter 188 is preferably configured to block light in the fluorescence spectrum of one or more fluorophores 116, 118. The white light filter 188 may also be configured to block light in the fluorescence excitation spectrum.

[0160] The white light filter 188 is preferably configured as a band stop filter, whose stop band corresponds to or at least includes the pass band of the fluorescence filter 190. The white light filter 188 is located in the light path between the beam splitter 192 and the white light camera 110. Therefore, the white light camera 110 only records wavelengths outside the stop band of the white light filter 188, and therefore also outside the pass band of the fluorescence filter 190.

[0161] Either the white light filter 188 or the fluorescence filter 190 may be a tunable filter.

[0162] If the beam splitter 192 is a dichroic beam splitter, at least one of the filters 188, 190 can be omitted, since in this case the optical spectral filtering is already integrated in the dichroic beam splitter. The above description of the passband and stopband should then apply mutatis mutandis to the dichroic beam splitter 192.

[0163] Thus, white light color camera 110 records digital white light color image 114 in a first imaged spectrum (reflectance spectrum) that is different from a second imaged spectrum that fluorescence camera records as a fluorescence spectrum. The wavelengths contained in the first and second imaged spectrum are determined by the filter settings of color separation component 176.

[0164] The medical fluorescence observation device 100 may also include an illumination assembly 178 configured to illuminate the object 106, preferably through the objective lens 174, through which the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 may be configured to selectively generate white light (i.e., light uniformly distributed across the entire visible spectrum) and fluorescence excitation light (which contains only light in wavelengths that stimulate fluorescence of at least one fluorophore 116, 118). The illumination light generated by the illumination assembly 178 may be fed into the objective lens 174 using an illumination beam splitter 180.

[0165] An illumination filter 179 may be provided based on the fluorophore and its fluorescence-specific excitation spectrum. For example, if 5-ALA / pPIX is used as the fluorophore, the illumination filter may have a transmittance of 90% to 98% up to a wavelength of 425 nm, a transmittance of 0.5% to 0.7% between 450 nm and 460 nm, a transmittance of no more than 0.1% between 460 nm and 535 nm, and a transmittance of almost zero at wavelengths above 535 nm.

[0166] Instead of or in addition to the illumination filter 179 , the illumination assembly 178 may also include an adjustable light source, including, for example, a plurality of LEDs or OLEDs of different colors.

[0167] The medical fluorescence observation device 100 may further include an image processor 170. The image processor 170 may be a hardware module (such as a microprocessor), or may be a software module. The image processor 170 may also be a combination of a hardware module and a software module, for example, by using a software module configured to run on a specific processor (such as a vector processor, a floating-point graphics processor, a parallel processor, and / or a plurality of processors). The image processor 170 may be part of a general-purpose computer 186 (such as a PC).

[0168] The image processor 170 is configured to retrieve the digital white light color image 114 and the digital fluorescence image 112. For example, the image processor 170 may be configured to retrieve the digital white light color image 114 and the digital fluorescence image 112 from the memory 194 and / or directly from the cameras 110, 111. The memory 194 may be part of the image processor 170, or may reside elsewhere in the medical fluorescence viewing device 100.

[0169] Image processor 170 is also configured to calculate digital output color image 160 from digital white light color image 114 and digital fluorescent image 112. Digital output color image 160 is a color image represented in a color space. The color space of the digital output color image may be different from the color space of any one of digital white light color image 114 and digital fluorescent image 112. However, preferably, the color space of digital output color image 160 is the same as the color space of digital white light color image 114 and digital fluorescent image 112.

[0170] The image processor 170 includes at least two operating modes, and the image processor 170 is configured to selectively operate in the at least two operating modes. For example, a user (not shown) can manually select which of the two operating modes to execute. In order to select the operating mode from the at least two available operating modes, a selector device 165 can be provided. The selector device 165 can be a mechanical device, such as a dial or switch; or a button or other element on a graphical user interface displayed on a display (such as display 132, 182), which will be described in more detail below.

[0171] like Figure 1 As shown in , the image processor 170 may include a first operation mode A and a second operation mode B, and an optional third operation mode C. Alternatively, the image processor 170 may be configured to operate in only a single operation mode, for example, in one of modes A, B, and C.

[0172] In the first mode of operation A, the image processor 170 is preferably configured to generate a digital output color image 160 that represents what is seen through the eyepiece 104. To achieve this, in the mode of operation A, the image processor 170 may be configured to combine the digital fluorescent color image 112 and the digital white light color image 114 to generate the digital output color image 160. In particular, the image processor 170 may be configured to additively combine or add the values ​​of the digital fluorescent color image 112 and the digital white light color image 114 in each color band of their color space at each corresponding pixel 150 in the digital fluorescent color image 112 and the digital white light color image 114. From this combination, the color in the output pixel in the digital output color image 160 is generated.

[0173] In the second operating mode B, the image processor 170 is configured to first apply the color conversion function 140 to at least one of the digital white light color image 114 and the digital fluorescent color image 112, and then generate the digital output color image 160 from the digital white light color image 114 and the digital fluorescent color image 112. Thus, in the second operating mode B, at least one of the digital white light color image 114 and the digital fluorescent color image 112 is color converted prior to combination. By switching between the first operating mode A and the second operating mode B, the user is able to switch between a view that faithfully renders what is seen through the eyepiece and a view that can highlight different features of the object 106.

[0174] The color conversion function 140 is configured to map the color of an input pixel or more than one input pixel to different colors in an output pixel. Therefore, the color conversion function changes the (one or more) input colors in a predictable manner when applied. The color conversion function 140 can be any type of function, such as a one-dimensional or n-dimensional interpolation function. However, preferably, the color conversion function 140 is a linear transformation function. Specifically, the color conversion function 140 can be a color conversion matrix 142. One dimension of the color conversion matrix 142 can be the sum of the number of color bands in the digital fluorescent color image 112 and the number of color bands in the digital white light color image 114. Another dimension of the color conversion matrix 142 can correspond to the number of color bands in the digital output color image 160 in another direction of the matrix.

[0175] In another embodiment, color conversion matrix 142 may have a first dimension and a second dimension, wherein the first dimension corresponds to the number of color bands in digital white light color image 114 or digital fluorescent color image 112 (depending on which of the two the color conversion matrix is ​​applied to), and the second dimension corresponds to the number of color bands in digital output color image 160.

[0176] If all images 112, 114 are in RGB color space, then the dimension of color conversion matrix 142 is preferably a 3×3 matrix, or if the color conversion matrix is ​​configured to operate on both digital white light color images and digital fluorescent color images, then the dimension of color conversion matrix 142 is a 6×3 or 3×6 matrix. A color conversion matrix can also be generated by the subsequent application of two or more matrices. For example, color conversion matrix 142 can be generated by first applying a 6×1 color conversion matrix and then applying a 1×3 color conversion matrix, which has the same effect as applying a 6×3 color conversion matrix.

[0177] The second operating mode B itself may include different sub-modes. Each sub-mode may apply a different color conversion function 140 to at least one of the digital white light color image 114 and the digital fluorescent color image 112 before combining the digital fluorescent color image 112 with the digital white light color image 114. Figure 1 In FIG. 1 , three sub-modes B-1, B-II, B-III are shown as examples only. Of course, more or less than three sub-modes may be provided. Different color conversion functions 140 may be stored in the memory 194.

[0178] For example, the first color conversion function 140a may be applied only to the digital white light color image 114, while the second color conversion function 140b may be applied only to the digital fluorescent color image 112. The first color conversion function 140a and the second color conversion function 140b may be applied independently of each other, thereby decoupling the color conversion of the digital white light color image 114 from the color conversion of the digital fluorescent color image 112. For example, only one of the first color conversion function 140a and the second color conversion function 140b may be applied to the respective images 112, 114, or both may be applied to their respective images.

[0179] In one embodiment, a user may select between different first color conversion functions 140a and / or different second color conversion functions 140b.

[0180] For example, in operation mode Bl, a first color conversion function 140a configured to convert colors in the digital white light color image 112 into colors that are not in the second imaged spectrum may be used. In this case, the natural colors of the object 106 or parts thereof that are not correctly recorded in the digital white light color image 114 because they are partially filtered and thus distorted by the color separation component 176 may be converted by the first color conversion function 140a into colors that more closely correspond to the natural colors.

[0181] In another additional or alternative example, a first color conversion function 140a may be used in operating mode B1 or an additional operating mode that converts the colors in the digital white light color image 114 into pseudo colors or different hues. This allows colors associated with certain types of tissue to be highlighted. For example, nerves, the color of arterial blood, and / or the color of venous blood may all be converted into different pseudo colors.

[0182] By using the selector device 165, the user can quickly browse the assignments of different pseudo colors to different types of tissue. In one operating mode, the color of arterial blood can be converted to neon red, in another operating mode, the color of venous blood can be converted to neon blue, and in yet another operating mode, the color of neural tissue can be converted to neon yellow. Other operating modes can use combinations of these modes. If the first color conversion function 140a assigns different hues instead of pseudo colors, then the contrast can be enhanced. For example, the first color conversion function 140a can extend the red color in the digital white light color image 114 to a wider range, thereby making smaller changes in blood oxygenation more visible to the user.

[0183] In yet another additional or alternative example, the first color conversion function can be used, for example, in operating mode BI or an additional operating mode to color balance and / or white balance the digital white light color image 114. In white balancing, the overall mix of colors is changed, but neutral colors (i.e., gray, black, and white) are maintained as neutral colors. In color balancing, colors including neutral colors are adjusted to correspond to neutral colors.

[0184] In yet another additional or alternative example, the first color conversion function 140a can be used, for example, in operating mode BI or an additional operating mode to shift the white point of the digital white light color image 114 to a predetermined position in the color space. Adjusting the white point allows adjustment for different lighting and adjusting filter settings of the color separation component 176.

[0185] According to one example, at least one of the above-mentioned first color conversion functions 140a may be applied to the digital white light color image 114. According to another example, at least two of the above-mentioned first color conversion functions 140a may be sequentially applied to the digital white light color image 114. Alternatively or additionally, two or more of the above-mentioned color conversion functions 140a may be combined into a single color conversion function 140a.

[0186] The above description of the various first color conversion functions 140a also applies mutatis mutandis to the various second color conversion functions 140b, the only difference between the two functions being that the second color conversion functions 140b are configured to operate on the digital fluorescent image 114 and to operate on different colors.

[0187] In operation mode B-II, for example, only one or more color conversion functions 140b may be applied to the digital fluorescent image without applying any color conversion function 140a.

[0188] In operating mode B-III, for example, one or more color conversion functions 140 a may be applied to the digital white light color image, and one or more color conversion functions 140 b may be applied to the digital fluorescent color image 112 .

[0189] An optional third operating mode C may be provided, wherein a third color conversion function 140 c is applied to both the digital white light color image 114 and the digital fluorescent color image 112 , wherein application of the color conversion function 140 c directly produces a digital output color image 160 .

[0190] If another type of fluorophore with a different excitation and / or fluorescence wavelength is used, then a different color conversion function 140a, 140b, 140c needs to be provided, because the combination of filter settings in the color separation component 176 and thus the first and second imaged spectra need to be adapted to the excitation and / or fluorescence wavelength.

[0191] Therefore, in one embodiment, the medical viewing device 100 or the processor 170 may be configured to store a plurality of different sets 143 of color conversion functions 140a, 140b, and / or 140c. Each different set 143 includes one or more of the above-mentioned color conversion functions 140a, 140b, and / or 140c. Each different set 143 corresponds to the use of a different fluorophore, and thus represents a different filter setting of the color separation component 176. For example, the first set 143a may be used for ICG as a fluorophore, while the second set 143b may be used for 5-ALA / pPiX as a fluorophore, and of course, additional sets may be used to accommodate additional fluorophores or combinations of fluorophores.

[0192] The medical fluorescence viewing device 100 can be adjusted to accommodate different fluorophores by reconfiguring the color separation component 176 (e.g., by exchanging its optical elements, such as filters 190 and / or 192, or dichroic beam splitter 180). For example, if the medical viewing device 100 is configured to automatically detect the settings of the color separation component 176, then the selection of the appropriate set 143 of color conversion functions 140 can be performed automatically.

[0193] Alternatively or additionally, the medical viewing device 100 may include a filter setting selector device 168 that allows a user to manually select a set 143 of color conversion functions 140 to be applied to the digital white light color image 114 and / or the digital fluorescent color image. The filter setting selector device 168 may be mechanical and / or part of a graphical user interface. For example, if the filter setting selector device 169 is operated to be in position a, then set 143a is selected. In position b, set 143b is selected, and so on.

[0194] In another embodiment, the filter setting selector device 168 may allow the user to select different image components of the digital white light color image 114 and the digital fluorescence color image 112. For example, in one position of the filter setting selector device 168, only the excited fluorescence of one or more fluorophores is shown. In another position, only the autofluorescence may be shown. And in yet another position of the filter setting selector device, a combination of a reflectance image of an object illuminated with only the fluorescence excitation spectrum and the excited fluorescence of at least one fluorophore 116, 188 may be shown.

[0195] The digital output color image 160 may be displayed on a display 132 that is integral with the medical fluorescence viewing device 100. For example, the display 132 may be integrated into an eyepiece or eyepiece 104 of the medical fluorescence viewing device 100. The display 132 may also display a graphical user interface for operating the medical viewing device 100.

[0196] The medical fluorescence viewing device 100 may include a direct optical path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In this case, the display may be a semi-transparent display 132 located in the direct optical path 134, or the display may be projected into the direct optical path 134. A beam splitter 136 may be provided to split the light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed to the eyepiece 104.

[0197] Alternatively, the medical fluorescence viewing device 100 may not have a direct optical path 134, but only display images from the integral display 132. As a further alternative, the medical fluorescence viewing device may not have any display.

[0198] The medical fluorescence observation device 100 may include an output interface 172 to which one or more (external) displays 182 may be connected. To this end, the output interface 172 may include standardized connectors and data transmission protocols, such as USB, HDMI, DVI, DisplayPort, Bluetooth, and / or other items. The external display may be a monitor, 3D goggles, eyepieces, etc. Any combination of external displays may be connected to the output interface 172. Any of the displays 182 may display a graphical user interface for operating the medical observation device 100.

[0199] The computer 186 or the image processor 170 is connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission line may be wired or wireless, or partially wired and partially wireless. The computer 186 and / or the image processor 170 may not be integrated as a whole in the medical fluorescence observation device 100, but may be physically located away from the digital imaging system 102. To this end, the digital imaging system 102 and the computer 186 and / or the image processor 170 may be connected to a network, such as a LAN, WLAN or WAN, to which at least one display 182 is also connected.

[0200] According to the modification, the medical fluorescence observation device 100 can be stereoscopic, but only includes two cameras, one for each stereo channel. In one stereo channel, a fluorescence color camera 111 is used and is configured to selectively also record white light reflectance, while in the other stereo channel, a white light color camera 110 is used. This arrangement provides a stereoscopic white light color image without using fluorescence, and provides a monoscopic white light color image and a monoscopic fluorescence color image when using fluorescence. The description above and below also applies to this configuration.

[0201] Figure 2 An example of a first operating mode A is shown.

[0202] Reference numeral 200 is a quantitative illustration of a first imaged spectrum 202 as it is recorded by digital white light camera 110 and / or represented in digital white light color image 114, respectively. Intensity I across wavelength / color λ is shown normalized. First imaged spectrum 202 preferably extends across at least visible spectrum 212.

[0203] By way of example only, the color space in which the first imaged spectrum 202 is recorded is an RGB color space having three primary colors or color bands 204, 206, and 208. One primary color 204 is blue, another primary color 206 is green, and the third primary color 208 is red. The sensitivity of the sensor of the white light color camera 110 in the different primary colors 204, 206, 208 is tuned to result in a sensitivity across the visible spectrum 212 that is as constant as possible.

[0204] If a color space other than RGB is used, the number, position and / or width of the color bands may be different. Apart from this, there are no major differences from the RGB color space.

[0205] The first imaged spectrum 202 does not include fluorescence excitation light and the fluorescence emission spectrum of at least one fluorophore 116, 118. Therefore, the first imaged spectrum 202 may include at least one stop band 210 that is consistent with the fluorescence emission of at least one fluorophore whose fluorescence is to be recorded by the fluorescence color camera 111. The stop band 210 is created, for example, by the white light filter 188. The number, width, and / or position of the stop bands 210 depends on the number and type of fluorophores to be observed in the object 106.

[0206] At reference numeral 220, a second imaged spectrum 222 is shown as it is recorded by the fluorescence color camera 111 and / or represented in the digital fluorescence color image 112, respectively. Merely by way of example, the color space in which the second imaged spectrum 222 is recorded is also an RGB color space. The sensitivity of the sensor of the fluorescence color camera 111 in the different primary colors 204, 206, 208 is tuned to result in a sensitivity across the visible spectrum 212 that is as constant as possible.

[0207] The spectra 202, 222 need not be recorded in the same color space, but this is preferred.

[0208] The second imaged spectrum 222 may include one or more passbands 224. The number, position and / or width of the passbands depends on the number and type of fluorophores used. At least one passband 224 preferably corresponds to at least one stopband 210. At least one passband is generated, for example, by the fluorescence filter 190.

[0209] The first imaged spectrum 202 and the second imaged spectrum 222 complement each other. They preferably complement each other to cover the entire or most of the visible spectrum 212.

[0210] Each passband 224 of the second imaged spectrum 222 preferably overlaps with a fluorescence emission spectrum 226, 228 of a fluorophore 116, 118, the fluorescence of which is to be recorded, and with one or more primary colors 204, 206, 208 of the color spectrum. For example, the fluorescence emission spectrum 226 of one fluorophore 116 may overlap with all three primary colors 204, 206, 208. The sensor of the fluorescence camera 111 recording the color band 204 records only a small part 230 of the fluorescence emission spectrum 226. The sensor of the fluorescence camera 111 recording the color band 206 records a large part 232 of the fluorescence emission spectrum 226, while the sensor of the fluorescence camera 111 recording the color band 208 again records only a small part 236 of the fluorescence emission spectrum 226.

[0211] If present under certain circumstances, fluorescence spectrum 228 is, in contrast, recorded only by fluorescence camera 111 which records color strip 208 .

[0212] Some fluorophores, such as 5-ALA / pPiX, may have very broad fluorescence emission spectra 226. In this case, it may not be advisable to extend the passband 224 to cover the entire fluorescence emission spectrum, because otherwise too many wavelengths will be missing from the white light reflectance image. Therefore, the passband 224 may not cover the entire fluorescence emission spectrum 226. Therefore, some fluorescence may be included in the first imaged spectrum 202 and recorded in the digital white light color image 114. If recorded as a white light reflectance image under a standard illuminant, this will not detract from the digital white light color image 114, because the intensity of the fluorescence will be much weaker than the intensity of the reflected white light. In addition, any fluorescence recorded in the digital white light color image can be used to supplement the fluorescence information in the digital fluorescence color image.

[0213] exist Figure 2 2, those portions of the fluorescence emission spectrum 226 that are included in the first imaged spectrum 202 are respectively indicated by reference numerals 260 and 262. Reference numeral 260 indicates wavelengths in the fluorescence emission spectrum 226 that are less than the lower cutoff wavelength of the passband 224. Reference numeral 260 indicates wavelengths in the fluorescence emission spectrum 226 that are less than the upper cutoff wavelength of the passband 224.

[0214] Fluorescence emission in first imaged spectrum 260 may be captured or recorded in all color bands 204, 206, 208. A first portion 260 of fluorescence emission spectrum 226 in first imaged spectrum may, for example, overlap color bands 204 and 206. A second portion 262 may overlap color bands 206 and 208.

[0215] If the digital white light color image 112 is not used to record a white light reflectance image but only to record fluorescence, for example, if the object is not illuminated with a white light standard illuminant but with a fluorescence excitation spectrum that only contains wavelengths shorter than those in the fluorescence emission spectrum, then the digital white light color image may also represent a fluorescence image of the object 106. In this case, the fluorescence information is contained in non-overlapping spectral bands in the digital white light color image 114 and the digital fluorescence color image 112.

[0216] The fluorescence excitation light illuminating the object may be blocked in the color separation component 176, for example, by appropriate filters 188 and / or 190, and / or an appropriate dichroic beam splitter 176. However, in another embodiment, the fluorescence excitation light may be recorded in at least one of the digital white light color image 114 and the digital fluorescence color image 112. In the latter case, some reflectance information is retained, which may be used for post-processing.

[0217] In its simplest form, the combination 240 of the digital white light color image 114 and the digital fluorescent color image 112 is an additive combination. For example, the values ​​in each color band of the digital white light color image 114 and the fluorescent color image 112 are added at each corresponding pixel 150. If the images 112, 114 are registered and have the same size, then the corresponding pixel 150 can be the pixel 150 located at the same position in both images 112, 114. If both images 112, 114 are RGB images and the value in the RGB color band of the pixel 150 in the digital white light color image is {R1, G1, B1}, and the value in the RGB color band of the corresponding pixel 150 in the digital fluorescent color image is {R2, G2, B2}, then the value in the RGB color band of the corresponding pixel 150 in the digital output color image is {R1+R2, G1+G2, B1+B2}. The combination 240 results in a color conversion because the color of the pixel of the output image is different from the color of the pixel in the images 112, 114. Thus, combination 240 is an example of color conversion function 140. Combination 240 may be implemented using color conversion matrix 142.

[0218] Reference numeral 250 points to a qualitative reproduction of a spectrum 252 of the digital output image 160. In essence, the spectrum 252 of the digital output image 160 is the addition of the first imaged spectrum 202 and the second imaged spectrum 222.

[0219] Instead, the individual spectra 202 and 222 are processed as a single combined multispectral spectrum whose number of color bands corresponds to the total number of color bands in the first imaged spectrum 202 and the second imaged spectrum 222. In other words, the digital fluorescent color image 112 and the digital white light color image 114 are jointly processed as a single (virtual) multispectral image. Rather than adding color space coordinates in each color band of the images 112, 114, a union set is formed and processed so that the color bands of the images 112, 114 remain separated.

[0220] This is consistent with operation mode C ( Figure 1 ), wherein the color conversion functions 140, 140c are applied to the digital white light color image 114 and the digital fluorescent color image 112 to obtain their spectra in Figure 2 The digital output color image 160 indicated at reference numeral 250 in FIG.

[0221] Reference below Figure 3 To keep it simple, Figure 3 It is shown in RGB color space Figure 2 spectral spectrum, and only a single passband 224 and stopband 210 are shown.

[0222] The digital white light color image 114 recorded in the first imaged spectrum 202 is composed of signals R1, G1, and B1, each representing the intensity I in a corresponding color band. Signal G1 is produced by two separate wavelength bands due to the stop band 210. However, the sensor recording G1 cannot distinguish between these two wavelength bands.

[0223] The digital fluorescent color image 112 recorded in the second imaged spectrum 222 is composed of signals R2, G2, B2 in each color band.

[0224] Since the first imaged spectrum 202 and the second imaged spectrum 222 are complementary to each other, each color band is split into two signals, each signal coming from a different camera 110, 111 and representing or equivalent to a sub-band of the color band. The color band 204 is split into R1 and R2, respectively. The color band 206 is split into G1 and G2, respectively, and the color band 208 is split into B1 and B2, respectively. Preferably, there is no overlap between the individual signals in the color band.

[0225] Thus, even if the same type of color camera is used to record the first imaged spectrum 202 and the second imaged spectrum 222 in the same color space, a multispectral image of the object 106 is obtained, which in this case consists of six color bands R1, R2, G1, G2, B1, B2. This is schematically shown at reference numeral 300. It is noted that this is independent of whether the fluorescence camera actually records fluorescence emission or reflectance.

[0226] In other words, at least one color band 204, 206, 208 of the color space of at least the digital output color image 160 is subdivided into two sub-bands R1, R2, G1, G2, B1, B2, wherein one of the two sub-bands is included in the digital fluorescent color image 112 and the other of the two sub-bands is included in the digital white light color image 114. Preferably, the two sub-bands in the color band do not overlap. They are preferably complementary. Most preferably, together they at least nearly complete the corresponding color band. The sub-bands may include or consist of two separate spectral bands. Each sub-band may be considered to be a color band in itself.

[0227] The color conversion function 140 reflects the subdivision of the color band into sub-bands, which is determined by the stop band 210 and the pass band 224. Since the width and / or position of a sub-band determines how much light will be collected in that sub-band by the respective camera 110, 111, the color conversion function 140c needs to be adjusted for each different filter setting of the color separation component 176.

[0228] Thus, the digital white light image 112 and the digital fluorescent image 114 may be processed together into an image consisting of the sum of the color bands in the two images 112 and 114. This provides improved color resolution.

[0229] For example, the color conversion function 140 c may be applied in the form of a linear transformation using the color conversion matrix 142 to generate the digital output color image 160 .

[0230] If the digital output color image 160 is represented in the RGB color space, then a 6×3 or 3×6 color conversion matrix 142 can be applied to the combined digital fluorescent and white light images 112, 114 or their component signals R2, G2, B2, R1, G1, B1, respectively, to obtain the RGB signals R*, G*, B*. The matrix coefficients C11, C12, ..., C63 can be determined experimentally.

[0231] It is noted that the color conversion matrix 142 may actually be obtained by applying several matrices in sequence.

[0232] exist Figure 3 In the embodiment of the present invention, the input union or, synonymously, the input union set {R1, R2, G1, G2, B1, B2} is formed by the set of color space coordinates {R1, G1, B1} in the color bands R, G, B of the (first) pixel of the digital white light color image 114 and the set of color space coordinates {R2, G2, B2} in the color bands R, G, B of the preferably corresponding (second) pixel of the digital fluorescent color image 112. The input union set corresponds to the pixels of the multispectral image. The color conversion matrix is ​​then applied to the input union. In order to form the input union, no additional memory is required. The input union can be logically generated by, for example, combining pointers to the color space coordinates of the first and second pixels. Of course, the color space coordinates of the two pixels can also be copied to a memory, which then physically stores the input union.

[0233] Next, the operation mode B is explained.

[0234] exist Figure 4 , a schematic reflection spectrum 400 of biological tissue of the object 106 is indicated, wherein the term "tissue" includes both fluid tissue and solid tissue. For example, the reflection spectrum 400 may correspond to the reflection spectrum of oxygenated blood.

[0235] Figure 4 Also shown in FIG. 5 are the spectral sensitivities R, G, B of the R, G, B sensors of the sample RGB color camera.

[0236] In human perception, tissue (such as oxygenated blood) is perceived as having a natural color, i.e., red, under white light illumination (such as CIE illuminants). Therefore, the reflection spectrum 400 is perceived as a single natural color, but it is widely distributed across the spectrum of visible light. In the first imaged spectrum 202 in which the digital white light color image 114 is recorded, the wavelength λ in the stop band 210 is not recorded. Therefore, the reflection spectrum 400 represented in the digital white light color image 114 cannot faithfully represent the natural color of the tissue.

[0237] The same applies to the second imaged spectrum 222 and the digital fluorescence color image 114, wherein the reflection spectrum 400 will also be distorted.

[0238] The same is true if spectrum 400 is not a reflectance spectrum but a fluorescence emission spectrum 226, which is recorded simultaneously in digital white light color image 114 and digital fluorescence color image 112, as shown in FIG. Figure 2 The peak intensity of the fluorescence emission spectrum and / or a large portion of the fluorescence energy will be within the stop band 210 and will therefore be recorded by the fluorescence camera.

[0239] Therefore, the case where both the fluorescence color camera 111 and the white light color camera 110 are used to record fluorescence alone corresponds to the case where both cameras record white light reflectance. In the former case, the spectral fluorescence information is mainly recorded by the fluorescence color camera 111 and supplemented by the spectral information recorded by the white light color camera 110; in the latter case, the spectral white light reflectance information is mainly recorded by the white light color camera 110 and supplemented by the spectral information recorded by the fluorescence color camera 111.

[0240] Figure 5 Schematic reproduction of a CIE 1932 color space diagram is shown, wherein a color 500 is shown. Color 500 may, for example, represent the color of a color having a reflection spectrum 400 ( Figure 4 ) when illuminated by CIE illuminants A, B, or C or another standard illuminant.

[0241] Since the wavelengths in the stopband 210 are not recorded, the color 500 is represented in the digital white light color image 114 as a (recorded) color 502 in the digital white light color image 114. In order to more faithfully represent the color 500, the recorded color 502 is converted to the color 500 by the color conversion function 140, which may be a first color conversion function 140a or a second color conversion function 140b. For example, the color conversion may be a shift or translation in the LMS color space or tristimulus coordinates.

[0242] The color conversion function 140 may be determined by color calibration using, for example, known colors on a color chart, known filter settings, and known lighting conditions. The wavelength of color 500 may be within the stop band 210 and therefore not within the first imaged spectrum 202. Together with the shift from 502 to 500, the entire color space may be shifted.

[0243] In some applications, it may be beneficial if color 500 (to which color 502 is converted by a color conversion function) is not a natural color, but rather a pseudo-color or a color that is close to but clearly different from a natural color. This may help to offset this color and the associated tissue type relative to other tissue or fluorescent colors that are close to color 500 and highlight this type of tissue to an experienced eye. For example, in the case of oxygenated blood, color 500 may be a "hyper-realistic" color, such as a brighter red than the natural color of oxygenated blood, or a red pseudo-color (such as neon red). Of course, this conversion may also be performed for any other color or tissue type.

[0244] Due to the stop band 210, the location of the natural white point 504 is shifted to the recorded white point 506. Therefore, another color conversion function 140 (e.g., color conversion function 140c) can shift the recorded white point 506 to or closer to the natural white point 504. In this regard, the white point is treated like any color.

[0245] A shift of a single color may correspond to a shift of the entire color space, ie, all colors are shifted by the same amount.

[0246] The digital white light color image 112 may be white balanced or color balanced using the same amount and direction of shift across the entire color space.

[0247] Alternatively, different colors or different sets of colors may be transformed or shifted differently in at least one of the amount and direction of the shift. For example, the recorded color 508, e.g., deoxygenated blood, may be further shifted into the blue range, to color 510. The shift of color 508 to 510 may be different from the shift of color 502 to color 500. Thus, the color conversion function 140 may be configured to generate a color-dependent transformation, rather than shifting the entire color space. For different (recorded) colors 502, 504, 508, different color conversion functions 140 may be applied.

[0248] Another example of a color conversion function 140, e.g., 140d, may expand region 512 in color space to a larger region 514. In region 514, the distance between colors is greater than the distance between colors in region 512, so the differences between colors become more visible. The application of such a color conversion function may be color dependent. In this case, only regions 512 having predetermined colors are expanded. This color conversion function 140 may include a color shift so that region 514 is not only expanded, but also moved to a different region of the color space.

[0249] Any of the above color conversion functions 140 may convert only a selected subset of the color appearance parameters of a color. The color appearance parameters are hue, chromaticity, saturation, lightness, and brightness. For example, when converting color 502 to color 500, only the hue may be changed, or the color conversion function 140 may only change the saturation and lightness.

[0250] Any combination of the above color conversion functions 140a may be applied. It should be understood that although the above description uses the expression "image", the color conversion function 140 is applied to the image at the pixel level, that is, to each pixel 150 of the digital white light color image 112. The color of the pixel 150 may determine which color conversion function 140 or which combination of color conversion functions 140 is applied. This may be done automatically by the processor 170.

[0251] exist Figure 8 , an example of color-dependent color conversion is shown. In this example, the digital white light color image 114 and the digital fluorescence color image 112 may both be reflectance images of the object 106. However, this is not necessarily the case, and the digital fluorescence image 112 may instead be a fluorescence image or a reflectance image, which, for example, contains only the fluorescence of the fluorophores.

[0252] In the images 112, 114, three different tissue types 852a, 852b, 852c are shown. Of course, more or fewer tissue types may be present in the images 112, 114. Various tissue types may not be clearly visible in one or both of the images 112, 114. For example, tissue 852a may be an arterial vessel comprising oxygenated blood, tissue 852b may be a venous vessel comprising deoxygenated blood, or, if fluorescence is recorded in a digital fluorescence image, a tumor, and 852c may be the background, e.g., in brain surgery, neural tissue.

[0253] Pixel 150a1 of the digital white light color image 114 is located in the region of tissue 852a. If an RGB color space is assumed for explanation purposes only, the color space coordinates of pixel 150a1 are {R 1,1 , G1,1 , B 1,1}. Pixel 150a2 of the digital white light color image 114 is located in the region of tissue 852b. The color space coordinates of pixel 150a2 are {R 1,2 , G 1,2 , B 1,2}. Pixel 150a3 of the digital white light color image 114 is located in the region of tissue 852c. The color space coordinates of pixel 150a3 are {R 1,3 , G 1,3 , B 1,3 The color space coordinates are recorded in the first imaged spectrum 202 .

[0254] In the digital fluorescence image 112, pixel 150b1 is located in tissue 852a and is preferably a pixel corresponding to pixel 150a1, pixel 150b2 is located in tissue 852b and is preferably a pixel corresponding to pixel 150a2, and pixel 150b3 is located in tissue 852c and is preferably a pixel corresponding to pixel 150a3. The corresponding pixels are preferably located in corresponding positions of the images 112, 114, or in corresponding positions of a pattern previously identified in the images 112, 114 using a pattern recognition algorithm.

[0255] If we assume the RGB color space again, the color space coordinates of pixel 150b1 are {R 2,1 , G 2,1 , B 2,1}. The color space coordinates of pixel 150b2 are {R 2,2 , G 2,2 , B 2,2}. The color space coordinates of pixel 150b3 are {R 2,3 , G 2,3 , B 2,3 These color space coordinates are recorded in the second imaged spectrum 222 and therefore contain different spectral information than the color space coordinates of the corresponding pixels 150a-c in the digital white light color image. If the color spaces of the images 112, 114 are different, they can be converted to a common color space.

[0256] like Figure 3 As shown in FIG. 8 , an input union (set) 846 of color space coordinates may be formed at each of pixels 150a1-150a3 and 150b1-150b3. Thus, the color space coordinates of pixels 150a1 and 150b1, {R 1,1 , G 1,1 , B 1,1} and {R 2,1 , G 2,1 , B 2,1}, forming the input union {R1,1 , R 2,1 , G 1,1 , G 2,1 , B 1,1 , B 2,1}. The color space coordinates of pixels 150a2 and 150b2, {R 1,2 , G 1,2 , B 1,2} and {R 2,2 , G 2,2 , B 2,2}, forming the input union {R 1,2 , R 2,2 , G 1,2 , G 2,2 , B 1,2 , B 2,2}. The color space coordinates of pixels 150a3 and 150b3, {R 1,3 , G 1,3 , B 1,3} and {R 2,3 , G 2,3 , B 2,3}, forming the input union {R 1,3 , R 2,3 , G 1,3 , G 2,3 , B 1,3 , B 2,3}. Note that each union 846 is essentially the same as Figure 2 Each different input union 846 corresponds to a different color.

[0257] A set 143 comprising at least two (ie, a plurality of) color conversion functions 140 may be provided, for example, and stored in the image processor 170. The set 143 may, for example, include Figure 5 140-I, 140-II, 140-III. A single color conversion function 140 may be provided instead of a set. In one embodiment, the color conversion functions 140-I, 140-II, 140-III may correspond to the color conversion functions 140a, 140b, 140c.

[0258] If only a single color conversion function 140 is provided, its application may depend on Figure 8, i.e., the color space coordinates contained therein. For example, if the input union belongs to a predetermined set 856 of target unions 858, which is assigned to or has been assigned to a specific color conversion function 140, then this specific color conversion function 140 is applied to the input union. Otherwise, the color conversion function 140 is not applied. In this case, the color conversion function 140 can be applied only when the input union 846 represents the typical color of the tissue 852a but not the typical color of the tissue 852b or 852c, because the color conversion function 140 is assigned to or has been assigned to the color representing the tissue 852a.

[0259] If two or more color conversion functions 140 are provided, for example, color conversion functions 140-I, 140-II, 140-III, which one of the color conversion functions 140-I, 140-II, 140-III to apply or whether any color conversion function to apply can be determined based on the input union 846 (i.e., the set of color space coordinates of the input union). Each set 856 is assigned to a different color conversion function.

[0260] For example, if the input union 846 or, more precisely, the target union 858 corresponding to the input union 846, is included in the predetermined set 856 of the target union 858, then the color conversion function 140-I assigned to this specific set 856 can be applied. If the input union 846 is included in a second predetermined set of target unions 858 (the second predetermined set preferably does not overlap with the first set), then the color conversion function 140-II assigned to this set can be applied. If the input union 846 is included in a third predetermined set 856 of the target union, then a third color conversion function 140-III can be applied or no color conversion function can be applied. Alternatively, if the input union 846 is not included in any set 856, then no color conversion function can be applied or a specific color conversion function can be applied. Therefore, tissue 852a may undergo a different color conversion than tissue 852b and / or tissue 852c, such as Figure 5 As explained with reference to colors 500, 508, 504, each color may be represented by a different input union.

[0261] Each set 856 represents a different group of colors. Each set 856 can be specifically a specific type of tissue that exhibits the colors contained in this set. For example, one set 856 can include a series of brighter reds to represent oxygenated blood, another set 856 can include a series of blue-reds to represent deoxygenated blood, another set 856 can include a series of gray-pink colors to represent living gray brain matter, another set 856 can include a series of white-pink colors to represent living white brain matter, and so on. Any number and combination of sets can be used. However, the target union 858 should only be included in one set 856 in order to have a one-to-one assignment of color conversion functions and input unions 846.

[0262] If the digital white light color image 114 and the digital fluorescence color image 112 record only fluorescence, then the various sets 856 and their target unions 858 may represent different types of fluorescence. For example, one set 856 may include a target union that maps the fluorescence of a fluorophore (such as 5-ALA / pPiX) to an output color. Another set 856 may include a target union 858 that represents the autofluorescence of biological tissue and maps its color (as represented in the input union) to the reflectance color of the biological tissue or a natural-looking autofluorescence color or pseudocolor. Another set 856 may include a target union 858 that represents the reflectance color of an object under illumination with a fluorescence emission spectrum and maps it to a more natural color.

[0263] The target union 858 can be determined empirically during the calibration process.

[0264] At least one set 856 of target unions 856 may be stored in or included within the image processor 170 , for example as a lookup table.

[0265] is assigned and applied to the input union {R 1,1 , R 2,1 , G 1,1 , G 2,1 , B 1,1 , B 2,1} results in the input union being converted into color space coordinates {R*1, G*1, B*1}, which are assigned to pixel 150c1 in the digital output color image 150. Output pixel 150c1 is preferably a pixel corresponding to at least one of pixels 150a1 and 150b1. 1,2 , R 2,2 , G 1,2 , G 2,2 , B 1,2 , B 2,2} results in the input union being converted into color space coordinates {R*2, G*2, B*2}, which are assigned to pixel 150c2 in the digital output color image 150. Output pixel 150c2 is preferably a pixel corresponding to at least one of pixels 150a2 and 150b2. 1,3 , R 2,3 , G 1,3 , G 2,3 , B 1,3 , B 2,3} results in the input union being converted into color space coordinates {R*3, G*3, B*3}, which are assigned to pixel 150c3 in digital output color image 150. Output pixel 150c1 is a pixel corresponding to at least one of pixels 150a3 and 150b3. If no color conversion function is assigned to the input union and therefore no color conversion is applied to this input union, the color space coordinates in the digital output color image 160 can be calculated by adding the color space coordinates in the union that are in the same color band, as shown in reference Figure 2 Explained.

[0266] Since the input union includes many more color bands than each of the images 112, 114, the accuracy of the union-dependent color conversion matches the accuracy of the tissue type detection. In fact, by accurately calibrating the different predetermined sets 856 of target unions 858 that determine which color conversion function to use, the tissue type-dependent color conversion can be integrated into the color conversion process.

[0267] Figure 6 A schematic representation of an imaging method is shown, which method may be implemented as a computer-implemented method running on an image processor 170, for example.

[0268] At optional step 600, a digital white light color image 114 is recorded using, for example, a white light color camera 110. At optional step 602, a digital fluorescent color image 112 is recorded using, for example, a fluorescent camera 111. Steps 600 and 602 are optional, as the images 112, 114 may also be retrieved from memory. As explained above, the cameras 110, 111 should be synchronized in terms of exposure time and locked to each other in terms of gain, i.e., maintaining the same gain ratio. Gamma may be set to a fixed value and all automatic color adjustments are preferably turned off.

[0269] At optional step 604, the respective images 112, 114 may be demosaiced.

[0270] At optional step 606, one or both images 112 or 114 may be matched and registered so that identical image features are represented in geometrically identical ways with respect to location, size, and orientation in each image 112, 114. After registration, each pixel in the digital white light color image 114 has a corresponding pixel in the digital fluorescent color image 112. Preferably, the corresponding pixels are located at the same location in each of the images 112, 114.

[0271] In one mode of operation (e.g., mode of operation B), but optionally also in mode of operation C, a color conversion is performed in an optional color conversion step 608. For example, step 608 is omitted in mode of operation A. In step 608, a color conversion function 140 or a combination of color conversion functions 140 as described above is applied to at least one of the digital white light color image 112 and the digital fluorescent image 114.

[0272] At an image combination step 610, the digital white light color image 114 and the digital fluorescent color image 112 are combined to generate a digital output color image 160. This can occur in all operating modes A, B, C. As described above, the images 112, 114 can be combined, for example, by simply adding them together. Figure 2 . This addition can be performed by adding the color space coordinates of the two images that are in the same color band. If a pixel in the digital white light color image 114 has color space coordinates {R1, G1, B1} in color space and a corresponding pixel in the digital fluorescent color image 112 has color space coordinates {R2, G2, B2}, then the color space coordinates {R*, G*, B*} of the corresponding pixel in the digital output color image can be calculated as R*=R1+R2, G*=G1+G2, B*=B1+B2.

[0273] In operation mode C, color conversion function 140 may be applied to digital white light color image 114 and digital fluorescent image 112 to generate digital output color 160 in step 610, such as Figure 3 In step 610, a color conversion function may be selected from a predetermined set of color functions 140 based on the input union and applied to the input union, as shown in FIG. Figure 8 Explained.

[0274] In step 612, post-processing may be performed. For example, the digital output color image 160 may be homogenized, its contrast may be enhanced, and / or a color space conversion, such as a color space conversion from RGB to sRGB and / or gamma correction may be performed. It is important to note that the color conversion in steps 608 and / or 610 is not gamma correction.

[0275] At step 612, the digital output image 160 is displayed.

[0276] Some embodiments relate to microscopes, particularly including as combined Figures 1 to 6 Alternatively, the microscope may be a fluorescence microscope such as one or more of the systems described herein. Figures 1 to 6 Part of or connected to one or more of the described systems.

[0277] The medical fluorescence viewing apparatus 100 may be configured to select which one of a set 143 of different color conversion functions 140 (e.g., 140-I, 140-II, 140-III) is applied to at least one of the digital fluorescence color image 112 and the digital white light color image 114. More specifically, the input selector may be configured to perform this selection.

[0278] In addition, the medical fluorescence viewing device 100 can be configured to select which digital output color image 160 is displayed in the display 132, 182, such as an eyepiece, VR goggles and / or an external monitor. Again, the selector device can be configured to select which digital output color image 160 is displayed on the display 132, 182. Now referring to Fig. 9 This is explained where the image processor 170 is shown as being configured to apply different color conversion functions 140-I, 140-II, 140-III from one or more sets 143 of color conversion functions to the digital fluorescent color image 112 and / or the digital white light color image 114, as explained above. Any number of different color conversion functions 140 and / or sets 143 may be used.

[0279] For example, in color conversion step 608, each color conversion function 140-I, 140-II, 140-III, if applied to digital fluorescent color image 112 and / or digital white light color image 114, results in the generation of a different digital color output image 160-I, 160-II, 160-III. For example, if color conversion function 140-I is applied to digital fluorescent color image 112 and / or digital white light color image 114, digital output color image 160-I is generated; applying color conversion function 140-II results in the generation of digital output color image 160-II, and so on.

[0280] The selector device 165 is shown connected to the image processor 170. The display selection signal 900 may be sent from the selector device 165 to the image processor 170 via any kind of wireless and / or wired transmission path.

[0281] The selector device 165 is hardware, such as a knob or switch. However, the selector device 165 may also be a software device, such as a graphical interactive element that can be operated by a user similar to a hardware switch and resides at least partially in the image processor 170. The selector device may also be a combination of hardware and software, in which case the software portion may also reside at least partially in the image processor 170.

[0282] As described above, the selector device 165 may be used to select different operating modes of the medical fluorescence viewing device 100. The display selection signal 900 may be representative of at least some of these different operating modes.

[0283] exist Fig. 9 In the embodiment of the present invention, each different operation mode involves applying a different color conversion function 140 or a different set of color conversion functions 140. For example, the selector device 165 can be used to select six different operation modes, such as mode I, mode II, mode III, mode I+II, mode I+III, and mode II+III. In mode I, the color conversion function 140-I is applied, while in mode II, the color conversion function 140-II is applied, and in mode III, the color conversion function 140-III is applied. In mode I+II, the color conversion functions I and II are applied, in mode I+III, the color conversion functions 140-I and 140-III are applied, and in mode II+III, the color conversion functions 140-II and 140-III are applied. Of course, there may also be a mode I+II+III, in which the color conversion functions 140-I, 140-II, and 140-III are applied.

[0284] In Mode I+II and other modes, when two or more different color conversion functions are selected for operating on the digital white light color image 114 and / or the digital fluorescent color image 112, each color conversion function may be applied sequentially to another color conversion function. For example, in Mode I+II, the color conversion function 140-I is first applied to the digital white light color image 114 and / or the digital fluorescent color image 112, and then the color conversion function 140-II is applied to the result.

[0285] exist Fig.10In another variation shown in , a selector device 165 can be used to select which of the different digital output color images 160-I, 160-II, and 160-III is displayed on the displays 132, 182. If I is selected on the selector device, then the digital output color image 160-I is displayed, if II is selected on the selector device, then the digital output color image 160-II is displayed, and so on. In addition, if I+II is selected, then the digital output color images 160-I and 160-II are displayed simultaneously on separate portions of the displays 132, 182, for example, side by side. If II, III are selected, then the digital output color images 160-II and 160-III are displayed simultaneously on separate portions of the displays 132, 182. This can continue for any number and combination of color conversion functions 140 and digital color output images 160.

[0286] Thus, from the color transformed set 143, an output set 1000 of different digital output color images 160 may be generated. The image processor 170 is configured to select at least one digital output color image from the output set 1000 for display according to the display selection signal 900. The selector device 165 may be configured to generate a plurality of different display selection signals 900, wherein each selector signal may be uniquely assigned to a digital output color image 160 of the output set 1000 or a combination of digital output color images 160. The image processor is preferably configured to display the digital output color image 160 assigned to the last received display selection signal 900.

[0287] If more than one digital output color image 160 is selected for display, then the displayed digital output color images 160 are preferably synchronized, ie, generated from the same digital white light color image 114 and digital fluorescent color image 112 .

[0288] In one embodiment, the digital output color image 160-I may be generated only from the digital fluorescence color image 112 as described above, which may specifically represent the fluorescence emission of at least one fluorophore 116. In this case, the color conversion function 140-I may be applied only to the fluorescence color image 112, which enhances the contrast of the fluorescence.

[0289] In another embodiment, the digital output color image 160-II may be generated by a combination of the above-described digital white light color image 114 and the digital fluorescence color image 112, wherein the digital white light color image 114 represents a reflectance image of the object 106 under illumination of a fluorescence excitation spectrum, and the digital fluorescence color image 114 represents fluorescence emission of at least one fluorophore. In this case, the color conversion function 140-II may be applied only to the white light color image 114. Such a color conversion function may be configured to shift the colors in the digital white light color image to more natural colors.

[0290] In yet another embodiment, the digital output color image 160-III may be generated by a combination of the digital white light color image 114 and the digital fluorescence color image 112. Here, the digital white light color image 114 may represent a reflectance image of the object 106 under white light illumination, and the digital fluorescence color image 114 may represent the fluorescence emission of at least one fluorophore. As described above, separate color conversion functions 140 may be applied to the digital fluorescence color image 112 and the digital white light color image 114, or the color conversion function may be applied to only one of the digital fluorescence color image 112 and the digital white light color image 114, or to a combination of the digital fluorescence color image 112 and the digital white light color image 114.

[0291] In the combined view, any two of the three images can be combined according to the user's selection.

[0292] Figure 7 A schematic diagram of a system 700 configured to perform the methods described herein is shown. System 700 includes a microscope 710 and a computer system 720. Microscope 710 is configured to capture images and is connected to computer system 720. Computer system 720 is configured to perform at least a portion of the methods described herein. Computer system 720 can be configured to execute a machine learning algorithm. Computer system 720 and microscope 710 can be separate entities, but can also be integrated together in a common housing. Computer system 720 can be part of a central processing system of microscope 710 and / or computer system 720 can be part of a subcomponent of microscope 710, such as a sensor, an actor, a camera, or a lighting unit of microscope 710.

[0293] The computer system 720 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in different locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 720 may include any circuit or combination of circuits. In one embodiment, the computer system 720 may include one or more processors that may be of any type. As used herein, a processor may mean any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA) (e.g., of a microscope or microscope component (e.g., a camera)), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 720 may be custom circuits, application specific integrated circuits (ASICs), and the like, such as, for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 720 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as a main memory in the form of a random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media such as compact disks (CDs), flash memory cards, digital video disks (DVDs), and the like. The computer system 720 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 720.

[0294] Some or all of the method steps may be performed by (or using) a hardware device (e.g., a processor, a microprocessor, a programmable computer or an electronic circuit). In some embodiments, such a device may perform one or some of the most important method steps.

[0295] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transitory storage medium (such as a digital storage medium, e.g., a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM and EPROM, EEPROM or flash memory) having an electronically readable control signal stored thereon, which cooperates (or is capable of cooperating with) a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0296] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0297] Generally speaking, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

[0298] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0299] In other words, therefore, an embodiment of the inventive method is a computer program with a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0300] Therefore, another embodiment of the invention is a storage medium (or a data carrier, or a computer-readable medium) comprising a computer program stored thereon, the computer program being for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0301] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection, for example via the Internet.

[0302] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0303] A further embodiment comprises a computer having installed on it the computer program for performing one of the methods described herein.

[0304] Another embodiment according to the invention comprises an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transmitting the computer program to the receiver.

[0305] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the method described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. In general, the method is preferably performed by any hardware device.

[0306] Reference numerals

[0307] 100 Medical fluorescence observation equipment

[0308] 101L Stereo subassembly for left channel

[0309] 101R Stereo subassembly for right channel

[0310] 102 Digital Imaging System

[0311] 104 Eyepiece

[0312] 106 Objects to be studied

[0313] 107 Biological tissue

[0314] 110 White light color camera

[0315] 111 Fluorescent Color Camera

[0316] 112 Digital fluorescent color images

[0317] 114 Digital white light color image

[0318] 116 Fluorophore

[0319] 118 Second fluorophore

[0320] 132 Integrated / Internal Display

[0321] 134 Direct optical path

[0322] 136 Beam Splitter

[0323] 140 color conversion functions

[0324] 140a-140d Different color conversion functions

[0325] 140-I, 140-II, 140-III different color conversion functions

[0326] 142 Color conversion matrix

[0327] A collection of 143 color conversion functions

[0328] 143a-143c Different sets of color conversion functions

[0329] 150 pixels

[0330] 150a1-150a3 Pixels in digital white light color images

[0331] 150b1-150b3 Pixels in digital fluorescent color images

[0332] 150c1-150c3 digitally outputs pixels in a color image

[0333] 160 digital output color image

[0334] 165 Selector Device

[0335] 169 Filter setting selector

[0336] 170 Image Processor

[0337] 172 Output Interface

[0338] 174 Objective

[0339] 176 Color Separation Component

[0340] 178 Lighting components

[0341] 179 Lighting Filter

[0342] 180 Illumination Beam Splitter

[0343] 182 Display

[0344] 184 Field of View

[0345] 186 Computer

[0346] 188 White light filter

[0347] 190 Fluorescence Filter

[0348] 192 Dichroic Beam Splitter

[0349] 194 Memory

[0350] 196 Data transmission line

[0351] 200 Spectrum

[0352] 202 First Imaging Spectrum

[0353] 204, 206, 208 original color or color ribbon

[0354] 210 Stopband

[0355] 212 Visible spectrum or range of light

[0356] 220 Spectrum

[0357] 222 Second Imaging Spectrum

[0358] 224 Passband

[0359] 226 Fluorescence emission spectra of fluorophores

[0360] 228 Fluorescence emission spectrum of another fluorophore

[0361] 230 portion of the fluorescence emission spectrum in the color band 204

[0362] 232 Portion of the fluorescence emission spectrum in color band 206

[0363] 236 Portion of the fluorescence emission spectrum in color band 208

[0364] 240 combined images

[0365] 250 Spectra

[0366] 260 The portion of the fluorescence emission spectrum in the first imaged spectrum

[0367] 262 The portion of the fluorescence emission spectrum in the first imaged spectrum

[0368] 300 Multispectral images from digital white light and fluorescence color images

[0369] 400 Reflectance Spectrum

[0370] 500 converted colors

[0371] 502 Record Color

[0372] 504 Natural White Spot

[0373] 506 recorded white spots

[0374] 508 Recorded Color

[0375] 510 converted colors

[0376] 512 Recorded color area

[0377] 514 Target color area

[0378] 600 Recording of digital white light color images

[0379] 602 Recording digital fluorescent color images

[0380] 604 De-mosaic

[0381] 606 Registration

[0382] 608 Color Conversion Steps 610 Image Combination Steps 612 Post-processing

[0383] 614 Display

[0384] 700 System

[0385] 710 Microscope

[0386] 720 Computer Systems

[0387] 846 Input Union

[0388] 852a-852b Organization Type

[0389] 856 Collection of different target unions

[0390] 858 Target Union

[0391] 900 Display selection signal

[0392] Output collection of 1000 digitally output color images

[0393] λ Wavelength

[0394] I. Intensity

[0395] A processor's first operating mode

[0396] B The second operating mode of the processor

[0397] C11,C12,…,C63 Coefficients of the color conversion matrix

[0398] Example of RGB color space

[0399] (Intensity) signal in the corresponding color band or subband of the R, G, B, R1, G1, B1, R2, G2, B2, R*, G*, B* color space

Claims

1. An image processor (170) for a medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the image processor (170) being configured to: - retrieving a digital white light color image (114) of the object (106) recorded in the first imaged spectrum (202); - retrieving a digital fluorescent color image (112) of the object (106) recorded in a second imaged spectrum (222), the second imaged spectrum (222) overlapping with the fluorescence emission spectrum (226) of at least one fluorophore (116) and different from the first imaged spectrum (202), the first imaged spectrum (202) and the second imaged spectrum (222) both overlapping with the visible spectrum (212); and - outputting at least one digital output color image (160, 160-I, 160-II, 160-III) of a set (1000) of digital output color images (160), the set comprising: - a digital output color image (160, 160-I) generated by the image processor (170) solely from the digital fluorescent color image (112); - a digital output color image (160, 160-II) generated by the image processor (170) from only the digital white light color image (114); and - a digital output color image (160, 160-III) generated by the image processor (170) from a combination of the digital fluorescent color image (114) and the digital white light color image (112); wherein the image processor (170) is further configured to - receive a display selection signal (900) from a selector device (165); and - selecting said at least one digital output color image (160, 160-I, 160-II, 160-III) from said set for output according to said display selection signal (900).

2. The image processor (170) of claim 1, wherein the image processor (170) - comprising a set (143) of color conversion functions (140, 140-I, 140-II, 140-III), each color conversion function being configured to map the color of an input pixel to a different color of an output pixel; and - configured to calculate said digital output color image (160, 160-I) generated only from said digital fluorescent color image (112) by applying at least one color conversion function (140-I) of said set (143) of color conversion functions to said fluorescent color image (112); o computing the digital output color image (160, 160-I) generated solely from the digital white light color image (114) by applying at least one color conversion function (140-II) of the set (143) of color conversion functions to the white light color image (114); o computing the digital output color image (160, 160-I) generated by a combination of the digital fluorescent color image (114) and the digital white light color image (112) by applying at least one color conversion function (140-I, 140-II) of the set (143) of color conversion functions to The white light color image (114), The fluorescent color image (112) and / or The combination of the digital fluorescent color image (114) and the digital white light color image (112).

3. The image processor (170) according to claim 1 or 2, wherein the image processor (170) is configured to output at least two digital output color images in the set (1000) simultaneously.

4. An image processor (170) according to any one of claims 1 to 3, wherein the digital white light color image (114) represents a reflectance image of the object (106) under illumination having a fluorescence excitation spectrum of at least one fluorophore (116), and / or the digital fluorescence color image (112) represents the fluorescence emission of the at least one fluorophore.

5. An image processor (170) according to any one of claims 1 to 3, wherein the digital white light color image (114) represents a reflectance image of the object (106) under white light illumination, and / or the digital fluorescence color image (112) represents the fluorescence emission of the at least one fluorophore.

6. The image processor (170) of any one of claims 1 to 3, wherein the digital white light color image (114) and the digital fluorescent color image (112) represent reflectance images of the object (106) under white light illumination.

7. The image processor (170) according to any one of claims 1 to 6, wherein the first imaged spectrum (202) and the second imaged spectrum (222) are complementary to each other.

8. The image processor (170) of any one of claims 1 to 7, wherein the color conversion function (14) is a 3x3, 6x3 or 3x6 matrix.

9. A medical fluorescence observation device (100), such as a fluorescence microscope or a fluorescence endoscope, comprising: An image processor (170) according to any one of claims 1 to 8; a fluorescent color camera (111) configured to record the digital fluorescent color image (112); and A white light color camera (110) is configured to record the digital white light color image (114).

10. The medical fluorescence observation device (100) according to claim 9, wherein the medical fluorescence observation device (100) comprises a selector device (165), and the selector device (165) is configured to be operated by a user of the medical fluorescence observation device (100) and to generate the display selection signal (900) according to the operation of the user.

11. The medical fluorescence observation device (100) according to claim 9 or 10, wherein the medical fluorescence observation device (100) comprises at least one display (132, 182) connected to the at least one image processor (170), and the display is configured to receive and display the at least one digital output color image (160) output by the image processor (170).

12. The medical fluorescence observation device (100) according to any one of claims 9 to 11, wherein the medical fluorescence observation device (100) is a surgical fluorescence microscope.

13. A computer-implemented image processing method for a fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the computer-implemented image processing method comprising the following steps: - retrieving a digital white light color image (114) of the object (106) recorded in the first imaged spectrum (202); - retrieving a digital fluorescent color image (112) of the object (106) recorded in a second imaged spectrum (222), the second imaged spectrum (222) overlapping with the fluorescence emission spectrum (226) of at least one fluorophore (116) and different from the first imaged spectrum (202), the first imaged spectrum (202) and the second imaged spectrum (222) both overlapping with the visible spectrum (212); and - outputting at least one digital output color image (160, 160-I, 160-II, 160-III) of a set (1000) of digital output color images (160), the set comprising: - a digital output color image (160, 160-I) generated by the image processor (170) solely from the digital fluorescent color image (112); - a digital output color image (160, 160-II) generated by the image processor (170) from only the digital white light color image (114); and - a digital output color image (160, 160-III) generated by the image processor (170) from a combination of the digital fluorescent color image (114) and the digital white light color image (112); The computer-implemented image processing method further comprises the following steps: - receiving a display selection signal (900) from a selector device (165); and - selecting said at least one digital output color image (160, 160-I, 160-II, 160-III) from said set for output according to said display selection signal (900).

14. A computer program product or a computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of claim 13.

15. A method for operating a medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope, the method comprising the following steps: - recording the digital fluorescence color image (112) in a second imaged spectrum (222) using a fluorescence color camera (111); as well as - recording a digital white light color image (114) in the first imaged spectrum (202) using a white light color camera (110); and - performing a computer-implemented image processing method as claimed in claim 13.

Images