Method, processor, and medical fluorescence observation apparatus for switching images

Abstract

The present invention relates to an image processor (170) for a medical fluorescence observation device (100) such as a fluorescence microscope or a fluorescence endoscope and / or a computer-implemented image processing method. A digital white light color image (114) of an object (106) recorded in a first imaging spectrum (202) and a digital fluorescence color image (112) of the object (106) recorded in a second imaging spectrum (222) are acquired. The second imaging spectrum (222) overlaps with the fluorescence emission spectrum (226) of at least one phosphor (116) and is different from the first imaging spectrum (202). Both the first imaging spectrum (202) and the second imaging spectrum (222) overlap with the visible spectrum (212). To facilitate the handling of the medical observation device (100) and to promote the transition to new fluorescence imaging technology for the user, a digital output color image (160, 160-I) generated by the image processor (170) from only the digital fluorescence color image (112), a digital output color image (160, 160-II) generated by the image processor (170) from only the digital white light color image (114), and a digital output color image (160, 160-III) generated by the image processor (170) from a combination of the digital fluorescence color image (112) and the digital white light color image (114) are output, at least one digital output color image (160, 160-I, 160-II, 160-III) of a set (1000) of digital output color images (160). To switch between different output images, a display selection signal (900) is received from a selector device (165), and at least one digital output color image (160, 160-I, 160-II, 160-III) is selected from the output group according to the display selection signal (900).

Description

Technical Field

[0001] The claimed subject matter relates to a computer-implemented image processing method and an image processor for generating one or more digital output color images of an object for display in a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope. The claimed subject matter further relates to a method of operating such a medical fluorescence observation device and to the medical fluorescence observation device itself.

Background Art

[0002] In existing medical fluorescence observation devices, fluorescence and anatomical images are recorded by one camera. Such a configuration can be used for fluorescence imaging of pPIX after administration of 5-ALA to reveal tumors. The selection of the optical filter is such that the camera captures both the fluorescence of the red wavelength and a part of the excitation light of the blue wavelength. The excitation light reveals a part of the anatomical structure of the tissue, and the fluorescence indicates the tumor. The optical filters used today are standardized for pPIX imaging. Such a configuration can also be used for other phosphors such as ICG.

[0003] This type of fluorescence imaging is particularly helpful in neurosurgical oncology, but still has significant drawbacks. The blue fluorescence excitation light used to visualize the anatomical structure of the tissue has a very low intensity, and the anatomical structure is shown only in a blue monochromatic color tone, so the visualization provided is insufficient. In particular, it is very difficult or even impossible to see bleeding under blue light. Therefore, this configuration is not optimal for live surgery guidance.

[0004] Other medical fluorescence observation devices use a grayscale camera to record fluorescence and a color camera to record white light images. In such a configuration, the fluorescence image and the white light image are digitally mixed, and the fluorescence is marked with a pseudo-color. The drawback here is that the pseudo-color does not accurately represent the color of the fluorescence that would be perceived by the human eye.

Summary of the Invention

Problems to be Solved by the Invention

[0005] In view of the above, there is a need to provide an apparatus and method with improved image display in order to facilitate the handling of observation methods in medicine and to help migrate users from different visualization techniques.

Means for Solving the Problems

[0006] This need is addressed by an image processor for a medical fluorescence observation apparatus such as a fluorescence microscope or a fluorescence endoscope. This image processor: - obtains a digital white light color image of the object recorded in a first imaging spectrum, - obtains a digital fluorescence color image of the object recorded in a second imaging spectrum that overlaps the fluorescence emission spectrum of at least one phosphor, is different from the first imaging spectrum, and both the first imaging spectrum and the second imaging spectrum overlap the visible spectrum, and - outputs at least one digital output color image from a set of digital output color images including: - a digital output color image generated by the image processor from only the digital fluorescence color image, - a digital output color image generated by the image processor from only the digital white light color image, - a digital output color image generated by the image processor from a combination of the digital fluorescence color image and the digital white light color image, and is configured to: The image processor is further configured to: - receive a display selection signal from a selector device, and - select at least one digital output color image from the set for output according to the display selection signal. ​It is an image processor configured as follows.

[0007] Furthermore, this need is addressed by a computer-implemented image processing method for medical fluorescence observation devices such as fluorescence microscopes or fluorescence endoscopes. This computer-implemented image processing method comprises: - obtaining a digital white light color image of an object recorded in a first imaging spectrum; - obtaining a digital fluorescence color image of the object recorded in a second imaging spectrum, the second imaging spectrum overlapping the fluorescence emission spectrum of at least one phosphor, being different from the first imaging spectrum, and both the first imaging spectrum and the second imaging spectrum overlapping the visible spectrum; and a digital output color image, - a digital output color image generated by an image processor from only the digital fluorescence color image, - a digital output color image generated by an image processor from only the digital white light color image, - a digital output color image generated by an image processor from a combination of the digital fluorescence color image and the digital white light color image, outputting at least one digital output color image from a set of digital output color images including; including, The computer-implemented image processing method further comprises - receiving a display selection signal from a selector device, and - selecting at least one digital output color image from the output set according to the display selection signal. This is a computer-implemented image processing method.

[0008] The above-described image processor and computer-implemented image processing method facilitate the handling of a medical observation device. This is because the device can be customized to display certain digital output color images in a coloring scheme that the user is more familiar with or that can accommodate different image recording configurations. A user-dependent display selection signal controls which image among a set of digital output color images is to be displayed.

[0009] The above solution can be further improved by the following features. These features can be added and combined independently of each other and each has its own advantageous technical effect. Each of the following features can be used to improve one of the above methods and / or to improve one of the above devices, regardless of whether a particular feature is only mentioned in relation to a certain method or only mentioned in relation to a certain device.

[0010] For example, the image processor can be configured to output at least two digital output color images of the output set simultaneously. This facilitates the comparison of different image styles and / or different color schemes or color conversions, and thus the detection of certain features of interest.

[0011] The at least two output color images can be displayed, for example, side by side or in a grid so as not to overlap. The image processor can also be configured to determine and / or change the position of the output color images on the display in response to a signal from a pointer device such as a mouse, joystick, mouse piece, foot pedal, track pad, or digital pen, etc., according to the user's selection. For example, this allows the user to manually overlay two or more images and directly compare, for example, the spatial extent of image features in different images.

[0012] Since the first spectrum and the second spectrum each represent only a part of the visible spectrum, the colors of the materials and tissue types recorded in the digital white light color image and the digital fluorescence color image may not appear natural and / or may not be clearly distinguishable from each other, considering the perceptual characteristics of the human eye. To adjust the colors, color conversion functions can be used. Different color conversion functions can be used to highlight different materials or tissue types or to affect the colors in the color space in a specific way. This will be explained in more detail again later.

[0013] In one advantageous aspect, the image processor can include a set of color conversion functions, each color conversion function can be configured to map the color of an input pixel to a different color of an output pixel, and the image processor can be configured to calculate a digital output color image generated only from the digital fluorescence color image by applying at least one color conversion function of the set of color conversion functions to the fluorescence color image; calculate a digital output color image generated only from the digital white light color image by applying at least one color conversion function of the set of color conversion functions to the white light color image; and calculate a digital output color image generated from a combination of the digital fluorescence color image and the digital white light color image by applying at least one color conversion function of the set of color conversion functions to a combination of the white light color image, the fluorescence color image, and / or the digital fluorescence color image and the digital white light color image. The selectability of such digital output color images facilitates the interpretation of the image data.

[0014] Digital white light color images typically record the anatomy of the surgical field by recording reflectance, but can also be used to record fluorescence, or both fluorescence and reflectance, of an object illuminated by a fluorescence excitation spectrum. Fluorescent color images can record the fluorescence of one or more phosphors, but can also be used to record a reflectance image of the anatomy when fluorescence is not used. By separately employing a digital camera for fluorescence on the one hand and a camera for white light images on the other hand, a more color-accurate natural-color output image can be provided regardless of the content recorded in the digital fluorescent color image and the digital white light color image. The digital output color image in this configuration mainly includes fluorescence, for example, either fluorescence excitation light or fluorescence emission as illumination, so the expression "digital fluorescent output color image" is used. Therefore, the digital fluorescent output color image is merely a special case of the digital output color image that can be generated from digital white light color images and digital fluorescent color images.

[0015] Accordingly, according to one embodiment, a digital white light color image can represent a reflectance image of an object under illumination using the fluorescence excitation spectrum of at least one phosphor, and / or a digital fluorescent color image can represent the fluorescence emission of at least one phosphor. Typically, the fluorescence excitation spectrum is limited to wavelengths with high light absorption, resulting in colored illumination. For example, illumination by the fluorescence excitation spectrum of 5-ALA / pPiX appears blue to the human eye. This combination corresponds to an image recorded by a single color camera that captures both fluorescence excitation and fluorescence emission.

[0016] According to another embodiment, a digital white light color image can represent a reflectance image of an object under white light illumination, and / or a digital fluorescent color image can represent the fluorescence emission of at least one phosphor. As will be explained in more detail below, this combination enables faithful representation of both a reflectance image that looks natural in, for example, anatomical surgery and a fluorescence image that looks clear in a certain target area.

[0017] According to yet another embodiment, the digital white light color image and the digital fluorescence color image can represent a reflectance image of an object under white light illumination. This combination, which will be discussed in more detail below, can more accurately analyze the image content and improve color representation and resolution.

[0018] In one embodiment, an image processor can be part of a medical fluorescence observation device. The medical fluorescence observation device may include a white light color camera for recording one or more digital white light color images and a fluorescence color camera for recording one or more digital fluorescence color images. The digital white light color image can include a plurality of first pixels, and each first pixel can have a first set of color space coordinates. Further, the first imaging spectrum can overlap the fluorescence emission spectrum of at least one of one or more fluorescent phosphor materials. In the digital fluorescence color image, one or more fluorescent phosphor materials can be represented by a second imaging spectrum. The digital fluorescence color image can include a plurality of second pixels, and each second pixel can have a second set of color space coordinates. The digital fluorescence output color image can have a plurality of output pixels. The image processor can be configured to calculate the color of the output pixels by applying a color conversion function to an input combination of the first set of color space coordinates of the first pixels and the second set of color space coordinates of the second pixels.

[0019] A computer-implemented image processing method can include generating a digital fluorescence output color image from a digital white light color image and a digital fluorescence color image, and calculating the color of output pixels by applying a color conversion function to an input combination of a first set of color space coordinates of first pixels and a second set of color space coordinates of second pixels.

[0020] The above-described image processor and computer-implemented image processing method can display the fluorescence of the phosphor more clearly. Both the digital white light color image and the digital fluorescence color image contain information regarding fluorescence in different spectra, i.e., the first imaging spectrum and the second imaging spectrum. The input combination has more color bands than each of the digital white light color image and the digital fluorescence image, and thus individually contains more spectral information. The additional spectral information is a result of the first spectrum and the second spectrum being different.

[0021] The input combinations of the first set and the second set correspond to multi-spectral images of one or more fluorescent-emitting phosphors. By using the enhanced spectral resolution obtained by processing the digital white light color image and the digital fluorescence image together as a multi-spectral image, individual color conversions can be applied to different tissue types and fluorescence rates. This further enables more reliable detection of fluorescence and discrimination between different types of fluorescence, including, for example, discrimination between the excitation fluorescence of the phosphor added to the subject and the autofluorescence of substances inherent to the subject.

[0022] The input combination has more color bands than each of the digital white light color image and the digital fluorescence image, and thus individually contains more spectral information. The additional spectral information is a result of the first spectrum and the second spectrum being different.

[0023] The input combination of the first set and the second set corresponds to a multispectral image of one or more fluorescent phosphors. By processing a digital white light color image and a digital fluorescence image together as a multispectral image using enhanced spectral resolution, individual color conversions can be applied to different tissue types and fluorescence rates. This further enables more reliable detection of fluorescence and discrimination between different types of fluorescence, including, for example, discrimination between the excitation fluorescence of the phosphor added to the subject and the autofluorescence of the material inherent to the subject.

[0024] The image processor can be configured to perform any of the above processing steps. The terms "image processor" and "data processing device" can be used synonymously.

[0025] For example, one embodiment may relate to an image processor for a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope, the processor being configured to acquire a digital white light color image of a subject recorded in a first imaging spectrum, the digital white light color image including a plurality of first pixels, each of the first pixels including a first set of color space coordinates in a first set of color bands; configured to acquire a digital fluorescence color image of the subject recorded in a second imaging spectrum, the digital fluorescence color image including a plurality of second pixels, each of the second pixels including a second set of color space coordinates in a second set of color bands; the second imaging spectrum overlapping at least one phosphor fluorescence emission spectrum, the second imaging spectrum being different from the first imaging spectrum, both the first imaging spectrum and the second imaging spectrum overlapping the visible spectrum; configured to generate a digital output color image from the digital white light color image and the digital fluorescence color image, the digital output color image including a plurality of output pixels; the image processor being configured to calculate the color of the output pixels by applying a color conversion function to an input combination of the first set of color space coordinates of the first pixels and the second set of color space coordinates of the second pixels, the application of the color conversion function depending on the color space coordinates within the input combination.

[0026] Another embodiment may relate to a computer-implemented image processing method for a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope. The computer-implemented image processing method includes the steps of obtaining a digital white light color image of an object recorded in a first imaging spectrum, the digital white light color image including a plurality of first pixels, each of the first pixels including a first set of color space coordinates in a first set of color bands; obtaining a digital fluorescence color image of the object recorded in a second imaging spectrum, the digital fluorescence color image including a plurality of second pixels, each of the second pixels including a second set of color space coordinates in a second set of color bands; the second imaging spectrum overlapping at least one phosphor's fluorescence emission spectrum, the second imaging spectrum being different from the first imaging spectrum, and both the first imaging spectrum and the second imaging spectrum overlapping the visible spectrum; generating a digital output color image from the digital white light color image and the digital fluorescence color image, the digital output color image including a plurality of output pixels; the color of the output pixels being calculated by applying a color conversion function to an input combination of the first set of color space coordinates of the first pixels and the second set of color space coordinates of the second pixels, the color conversion function being applied depending on the color space coordinates within the input combination.

[0027] At least one fluorescent phosphor can be included in an object imaged by a medical observation device, particularly a white light color camera and a fluorescence color camera. The object may include or consist of biological tissue.

[0028] At least one of the digital white light color image and the digital fluorescence color image can consist of a fluorescence excitation spectrum, or can include a reflectance image of an object illuminated by light including a fluorescence excitation spectrum, or can represent something equivalent thereto. Preferably, this reflectance image is included in the digital white light color image in addition to the fluorescence of at least one phosphor. Thus, the first imaging spectrum may overlap the fluorescence excitation spectrum.

[0029] By the above-described image processor and computer-implemented image processing method, it is possible to perform different color conversions for different colors on a target. By depending the application of the color conversion function on the input combination of two sets of color space coordinates, a more accurate color conversion method becomes possible. By color-dependent conversion, for example, different color regions can be assigned to different fluorescent colors and reflected colors under fluorescent excitation illumination.

[0030] According to another aspect, the computer-implemented image processing method can include the step of demosaicking at least one of a digital white light color image and a digital fluorescent color image. By demosaicking, color artifacts can be avoided. It is preferable to perform demosaicking before any color conversion function is applied.

[0031] The computer-implemented image processing method can include the step of normalizing at least one of a digital white light color image and a digital fluorescent color image. By normalization, the combination process of the digital white light color image and the digital fluorescent color image becomes easier.

[0032] According to one aspect, the computer-implemented image processing method can include the step of aligning a digital white light color image and a digital fluorescent color image. By alignment, image features existing in both the digital white light color image and the digital fluorescent color image are represented in the same size and orientation in each image. After alignment, the digital white light color image and the digital fluorescent color image match each other. Therefore, the pixels at the same position in the two aligned images are corresponding pixels.

[0033] Specifically, the digital white light color image and the digital fluorescent color image may already be aligned with each other when they are acquired by the image processor. Alternatively, the image processor can be configured to align the digital white light color image and the digital fluorescent color image with each other.

[0034] Alignment should be performed before applying the color conversion function, preferably after demosaicing. In the aligned digital white light image and digital fluorescence image, a first pixel including a first set of color space coordinates and a second pixel including a second set of color space coordinates are corresponding pixels, and / or the output pixel and the first and second pixels are corresponding pixels.

[0035] In one embodiment, the fluorescence medical observation device can be operated in various operation modes, in which different image modalities are assigned to both or instead of the digital fluorescence color image and the digital white light color image in addition to the images representing the fluorescent phosphor.

[0036] For example, in one such operation mode, a fluorescence color camera can record a digital white light color image, particularly a reflectance color image under the same illumination as a standard light source. Such a fluorescence color image can complement the reflectance image recorded by a white light color camera. Thus, the fluorescence color camera is not necessarily limited to recording only fluorescence images. However, the fluorescence camera may be limited to recording only images in a second imaging spectrum without hardware change, for example, by replacing one or more filters. In another operation mode, the digital white light color image can represent a reflectance image of an object illuminated by a standard light source, and the digital fluorescence color image can represent one or more fluorescent phosphors. In all these operation modes, the generation of the output color image may be the same.

[0037] According to another advantageous embodiment of a computer-implemented image processing method and / or an image processor, the first imaging spectrum and the second imaging spectrum are complementary to each other, i.e., they do not overlap. In such a configuration, crosstalk between fluorescence and white light is avoided. On the one hand, an accurate rendition of the fluorescence of at least one phosphor is maintained, and on the other hand, an accurate rendition of the reflectance of the object is maintained. Further, by having complementary spectra for each of the digital fluorescence color image and the digital white light color image, there is no redundant special information, which facilitates the connection process of input combining the two images to form a multi-spectral image.

[0038] The digital white light color image can be represented in a first color space using at least three first color bands. The digital fluorescence color image can be represented in a second color space including at least three second color bands. The first color space and the second color space are preferably the same such that the first color bands and the second color bands are also the same. However, this is not essential. The first color space and the second color space may be different.

[0039] The digital output color image can be generated in a third color space including at least three third color bands. The third color space may be the same color space as the first color space and / or the second color space, or may be a different color space.

[0040] Any of the first color space, the second color space, and / or the third color space may be an RGB color space including three color bands of R, G, and B, or any other color space, such as HSV, LAB, CIELAB, XYZ, CMYK, a multispectral color space, or a hyperspectral color space, that is, adopting four or more color bands. The term "color" is used to indicate coordinates in a color space, that is, a set of color space coordinates in the color bands of the color space, and the color bands do not necessarily represent saturation, and may represent other aspects of color such as saturation, luminance, or lightness, as in the HSV, LUV, or LAB color spaces. In this regard, the term "color band" is used synonymously with the dimension of the color space throughout this text.

[0041] According to one aspect of a computer-implemented image processing method and / or an image processor, the first color band and the second color band are mapped onto a third color band of a digital output color image using a color conversion function. The color conversion function may be non-linear, but is preferably a linear transformation. The color conversion function can include a color conversion matrix or other types of linear functions. To map the first color band and the second color band onto the third color band, the color conversion matrix can be applied to the first color band and the second color band. Preferably, each color band of the first color band (set) and each color band of the second color band (set) are input into the linear transformation as separate color bands. In particular, each color band of the first color band and each color band of the second color band may be input into the linear transformation simultaneously.

[0042] The color conversion function can be stored in the memory of an image processor or a medical fluorescence observation device. As will be further described below, two or more color conversion functions can be provided.

[0043] The color conversion matrix can have a dimension of number X in one direction and a dimension of number Y in the other direction. The number X may be the sum of the amount of the first color band, i.e., the number of color bands in the set of the first color bands, and the amount of the second color band, i.e., the number of color bands in the set of the second color bands. The number Y may be the number of color bands in the third color space. In other words, one dimension of the color conversion matrix may correspond to the sum of the dimensions of the color spaces of the digital white light color image and the digital fluorescence image. The other dimension of the color conversion matrix may correspond to the dimension of the color space of the digital output color image. For example, when the first, second, and third color spaces are each an RGB color space, the color conversion matrix can have dimensions of 6×3, or 3×6, or 3×3.

[0044] The color conversion matrix can be obtained from combinations of various matrices that are continuously applied to the combined set.

[0045] For example, a first 6×1 intermediate matrix can be applied to the digital white light color image and the digital fluorescence color image. This can be regarded as reducing the color information in the combined set to intensity values. Such intensity values can correspond, for example, to the fluorescence intensity of a specific phosphor. Next, a second 1×3 intermediate matrix can be used to map the intensity values to colors in the digital output color image. And the application of these two matrices corresponds to the application of a 6×3 matrix. Of course, these two intermediates.

[0046] Therefore, the third color band, i.e., the color band of the digital output color image, can be obtained from a linear combination, particularly an addition, of the color bands of the digital white light color image and the color bands of the digital fluorescence color image. Such a linear combination or addition can be regarded as a color conversion function by which two colors of pixels in the digital white light color image and the digital fluorescence color image are mapped to the color of a pixel in the output image.

[0047] According to one aspect, the digital fluorescence color image and the digital white light color image are processed together as a multispectral image, and the color band of the multispectral image is formed or constituted by the complementary color bands of the digital fluorescence color image and the digital white light color image. This leads to an improvement in color accuracy, especially when the fluorescence image represents the reflected white light recorded in the second imaging spectrum and can thus complement the white light information in the digital white light color image. By using this approach, the white light information is included in the color band of the digital fluorescence color image and is thus provided with a finer color granularity than the color granularity that only the digital white light color image would provide. When the digital fluorescence color image represents fluorescence emission, processing the digital white light color image and the digital fluorescence color image as a multispectral image can faithfully represent the object under both the reflected white light and the fluorescence. In particular, when the digital fluorescence color image represents the reflectance image of the object under the same illumination as the digital white light color image, additional spectral information can be obtained.

[0048] According to other advantageous embodiments, the first imaging spectrum can include a first sub-band in a color band of the color space, the second imaging spectrum can include a second sub-band in this color band, and the first sub-band and the second sub-band are preferably complementary to each other. The color space can be the color space of the digital white light color image, the color space of the digital fluorescence color image, or preferably the color space of the digital output color image. It is preferred that the sub-bands of one color band together complete each color band. This ensures that no useful spectral information is lost.

[0049] The first sub-band and the second sub-band can be present in two or more color bands of each color space. The sub-bands in a color band can further be divided into a plurality of different wavelength bands that are distinct from each other.

[0050] Preferably, the second imaging spectrum includes IR wavelengths. This allows for capturing fluorescence emission in the near-infrared region or, if a digital fluorescence color image is not used for fluorescence capture, additional information regarding anatomical structures can be added. Further, the IR wavelengths can include spectral information that enables better discrimination between specific tissue types and the specific color conversion functions assigned to them.

[0051] The image processor may be configured to generate output pixels by simultaneously applying color conversion functions to both the first pixel and the second pixel.

[0052] According to one embodiment, the image processor can include at least two different color conversion functions. The image processor can be configured to select one of at least two different color conversion functions depending on the input combination, or in the method, one of at least two different color conversion functions can be selected depending on the input combination.

[0053] Various color conversion functions can be used for various tissue types having a predetermined color range. The predetermined color range can correspond to one or more regions in a color space, each region being characterized by a set of color space coordinates. The color range can be determined for each tissue type in a calibration process.

[0054] To assign different color conversion functions to different tissue types, an image processor can include at least two different sets of target bindings. Each of at least two different color conversion functions can be assigned to at least two different sets of different target bindings. The various target bindings of a set correspond to the colors that are converted using the color conversion function assigned to that set. Each set of target bindings can represent, for example, different tissue types and / or can represent the fluorescence emissions of different phosphors, and each target binding within the set can correspond to the color or fluorescence assigned to this tissue type during calibration.

[0055] In one embodiment, to determine whether an input binding is included in a given set of target bindings, an image processor can be configured to compare the combination of inputs to the target bindings of the different sets of target bindings.

[0056] In one embodiment, an image processor can be configured to select a color conversion function from a set of at least two different color conversion functions according to the combination of a first set of color space coordinates of a first pixel and a second set of color space coordinates of a second pixel.

[0057] The processor can be configured to select the color conversion assigned to the set of target bindings that includes the input binding. This color function can then be applied to the input binding.

[0058] In one embodiment, the image processor is - a given set of target bindings that can represent a tissue containing oxygenated blood, such as arterial tissue, where at least one target binding corresponds to the color representing oxygenated blood; - a given set of target bindings that can represent a tissue containing deoxygenated blood, such as venous tissue, where at least one target binding corresponds to the color representing deoxygenated blood; - A set of predetermined target bindings, where at least one target binding corresponds to a color representing the gray matter of a living brain; - A set of predetermined target bindings, where at least one target binding corresponds to a color representing the white matter of a living brain; - A set of predetermined target bindings, where at least one target binding corresponds to a color representing the fluorescence of a phosphor; - A set of predetermined target bindings, where at least one target binding corresponds to a color representing the autofluorescence of a biological tissue; - A set of predetermined target bindings, where at least one target binding corresponds to a natural color representing the color of a biological tissue illuminated by a standard light source, having at least one of.

[0059] The target binding maps the color represented in the set of input bindings to another output color. The target bindings in one set can be determined, for example, by spectral decomposition, principal component analysis, and / or color calibration.

[0060] For example, a set of target bindings can be used to map the color in a reflectance image of an object under illumination of a fluorescence excitation spectrum to a (more) natural color. The fluorescence excitation spectrum includes wavelengths shorter than or preferably consisting of the fluorescence emission wavelengths of at least one phosphor. Furthermore, the fluorescence excitation spectrum preferably has a peak at or in the region of the wavelength at which the light absorption of one or more phosphors is maximum. Thereby, the object is brightly illuminated by the fluorescence excitation spectrum. For example, when 5-ALA / pPiX is used as a phosphor, the fluorescence excitation spectrum has a strong blue hue, and the reflectance image of the object under this illumination has a bluish tint. A color conversion function, in particular a set of target bindings, can be used to map the bluish coloration to a more natural-looking color.

[0061] According to one embodiment, the image processor can be configured to operate selectively in one of a first operation mode and a second operation mode. The first operation mode includes an image combining step, and the second operation mode includes a color conversion step and an image combining step. In the image combining step, the image processor is configured to generate a digital output color image by combining a digital white light color image and a digital fluorescence color image; and in the color conversion step, the image processor is configured to apply at least one color conversion function from the group including a first color conversion function applied to the digital white light color image and a second color conversion function applied to the digital fluorescence color image.

[0062] The computer-implemented image processing method can be configured to be executed selectively in either a first operation mode or a second operation mode. Here, in the first operation mode, an image combining step is executed, and in the second operation mode, a color conversion step and an image combining step are executed; in the image combining step, a digital output color image is generated by combining a digital white light color image and a digital fluorescence color image, and in the color conversion step, at least one color conversion function from the group including a first color conversion function applied to the digital white light color image and a second color conversion function applied to the digital fluorescence color image is applied.

[0063] By providing two selectable operation modes, in both cases, it becomes possible to provide a digital output color image corresponding to what is seen through the eyepiece of the medical observation device in the first operation mode.

[0064] By providing two color conversion functions that operate separately on a digital white light color image and a digital fluorescence color image respectively, the color conversion of the digital white light color image can be separated from the color conversion applied to the digital fluorescence color image. As a result, each of these two images can be optimized independently for viewing before they are combined to form a digital output color image. According to one aspect, the first color conversion function and / or the second color conversion function are configured to shift the (recorded) color to a (different) color in the color space in each of the digital white light color image and the digital fluorescence image. The shift is predetermined by the color conversion function with respect to at least one of the shift amount and the shift direction in the color space.

[0065] For example, the natural color of blood containing oxygen or blood that has lost oxygen may be captured only incompletely in each of the first imaging spectrum and the second imaging spectrum. This may be particularly true when the reflectance spectrum of the blood extends to wavelengths that are not recorded in the first imaging spectrum or the second imaging spectrum respectively. Therefore, when blood is recorded in a digital white light color image or a digital fluorescence image, its color may not appear natural, making it difficult for the user or even an automatic image processing device to recognize blood vessels. By using the first color conversion function and / or the second color conversion function, the (recorded) color, here the color of the blood, can be shifted to a more natural color in the individual digital white light color image or fluorescence color image.

[0066] In other examples, by using various color conversion functions, the colors of the recorded oxygenated blood and deoxygenated blood can be shifted differently. Again, both of these colors are incompletely represented in the first imaging spectrum and / or the second imaging spectrum. The color of the deoxygenated blood from which the recorded oxygen has been lost is further shifted towards blue, while the color of the oxygenated blood is further shifted towards red. This may require different amounts and / or directions of shift for the two colors recorded in the color space; one color may be shifted by a color conversion function differently from the other color with respect to at least one of amount and direction in the color space.

[0067] The above examples are of course applicable to any other kind of tissue, such as fluid tissue like lymph, or solid tissue like bone, muscle and / or nerve tissue.

[0068] The converted colors can correspond to natural colors, pseudo-colors such as neon colors, or "hyper-real" colors in which at least one of the color appearance parameters, namely hue, chroma, saturation, lightness and luminance, is modified compared to the natural color in order to enhance visibility and contrast.

[0069] The natural or actual color in this context corresponds to the color that a CIE standard observer would perceive under a standard light source such as CIE illuminant A, B or C, or other standardized white light illuminations.

[0070] The first color conversion function and / or the second color conversion function can be configured to convert the colors in the respective digital white light color image or fluorescent color image into colors that do not lie in the respective first imaging spectrum and / or the second imaging spectrum. Thus, the color conversion function can be used to correct imaging defects resulting from the limited first imaging spectrum and / or the second imaging spectrum.

[0071] In other examples, the first color conversion function and / or the second color conversion function can be configured to shift the white point of a digital white light color image and / or a fluorescent color image to a predetermined position, particularly to another white point. The white point to which the white point is shifted can correspond, in particular, to a standard white point defined by a standard light source such as a CIE light source. Thereby, the color space can be centered.

[0072] Furthermore, the first color conversion function and / or the second color conversion function can be configured to perform white balance adjustment or color balance adjustment on each of the digital white light color image and / or the fluorescent color image. In color balance, the intensity of colors is adjusted so as to correctly render at least some colors, particularly intermediate colors. In white balance, the colors are adjusted to make a white object appear white and not colored.

[0073] According to another aspect, the first color conversion function and / or the second color conversion function can be configured to expand adjacent regions or fragmented regions in the color space into larger regions in the digital white light color image and / or the fluorescent color image, respectively. Such a color conversion function enables better visualization of the subtle differences between adjacent colors as the adjacent colors move further apart. For example, when the region of the fluorescent color of a specific phosphor is expanded, the nuances of the fluorescence become more visible. The expansion of the region will of course often be accompanied by an appropriate color shift, and as a result, the expansion of the region occurs around natural or unnatural colors.

[0074] According to yet another aspect, the first color conversion function and / or the second color conversion function can be configured to shift all colors in the color space of each of the digital white light color image and / or the fluorescent color image.

[0075] Any of the above-described color conversion functions can be combined to form a single color conversion function and / or any of the above-described color conversion functions can be sequentially applied to each color image of a digital white light color image and / or a fluorescent color image.

[0076] It may be even more advantageous if the first color conversion function and / or the second color conversion function is configured to be applied according to the color of individual pixels in a digital white light color image and / or a digital fluorescent color image. Thereby, generally, different colors can be treated differently.

[0077] According to another aspect, a plurality of different first color conversion functions and / or second color conversion functions can be provided, and the processor can be configured to determine, according to the color of a pixel, a first color conversion function and / or a second color conversion function among the plurality of different first color conversion functions and / or second color conversion functions to be applied to the pixel. The plurality of first color conversion functions and / or second color conversion functions can include any of the above-described color conversion functions. According to this aspect, a targeted color dependency correction can be applied to each digital white light color image and / or fluorescent color image. For example, the area around some colors in a digital white light color image can be enlarged to make the color difference more visible. Other colors, such as the colors of bones and / or nerves, can simply be shifted to be more natural. Other colors, and thus the tissues associated with this color, can be highlighted by being converted to neon colors.

[0078] The above computer-implemented image processing method is executable when implementing a method for operating a medical fluorescence observation device. The above-described image processor can be part of a medical fluorescence observation device such as a fluorescence microscope or a fluorescence endoscope, and in particular, can be a fluorescence microscope or a fluorescence endoscope configured for a surgical operation such as a neurosurgical operation, an orthopedic surgical operation, and / or an ophthalmic surgical operation. The fluorescence microscope may be, for example, a laboratory microscope used for a biopsy.

[0079] Medical fluorescence observation devices such as fluorescence microscopes or fluorescence endoscopes can include an image processor configured to perform any of the above-described steps. Further, the medical fluorescence observation device can have a fluorescence color camera for recording a digital fluorescence color image and a white light color camera for recording a digital white light color image.

[0080] Since the fluorescence intensity may be relatively low, the fluorescence color camera can be operated with a higher integration time.

[0081] A method for operating a medical fluorescence observation device can include the steps of recording a digital white light color image using a white light color camera and recording a digital fluorescence color image using a digital fluorescence color camera. The medical fluorescence observation device can include a digital fluorescence color camera and a white light color camera. In particular, both the fluorescence color camera and the white light color camera can record images of one or more fluorescent-emitting phosphors. The white light color camera can further record a reflectance image of an object illuminated by a fluorescence excitation spectrum.

[0082] To reduce post-processing, the fluorescence color camera and the white light color camera can have overlapping coaxial and / or preferably the same field of view.

[0083] The medical fluorescence observation device can be a stereoscopic device or a planar view device. In one embodiment, a digital fluorescence color camera and a white light color camera can be provided in each stereoscopic channel. Alternatively, a medical fluorescence observation planar view device can be provided with only one white light color camera and only one digital fluorescence color camera.

[0084] According to one embodiment, the medical fluorescence observation device can include a selector device that is operated by a user of the medical fluorescence observation device and is configured to generate a display selection signal.

[0085] The selector device may be a hardware device such as a dial, knob, or switch, a software device such as an interactive graphical element that may be similar to a dial, knob, or switch, or a combination of a software device and a hardware device. The selector device can be used to generate a display selection signal. The selector device can be configured to generate a plurality of display selection signals, each of which can be uniquely assigned to a specific color conversion function or a specific combination of color conversion functions, and a specific combination of a digital fluorescence color image and a digital white light color image.

[0086] The medical fluorescence observation device can include at least one display connected to at least one image processor, and the display is configured to receive and display at least one digital output color image output by the image processor.

[0087] In another arrangement of the medical stereoscopic fluorescence observation device, the fluorescence color camera may be provided within one stereoscopic channel, and the white light color camera may be provided within another stereoscopic channel. In such an arrangement, the fluorescence color camera may be configured to record a (white light) reflectance image in the second imaging spectrum. In the case of a (white light) reflectance image, stereoscopy is provided by this arrangement, but strictly speaking, stereoscopy is limited to the overlapping portion of the first imaging spectrum and the second imaging spectrum, and it is rarely recognized by a human observer. When fluorescence emission is recorded, the fluorescence color camera provides a plan view image of the fluorescence emission of the object, while the white light color camera provides a plan view of the reflectance of the object.

[0088] A medical fluorescence observation device may include an optical color separation assembly, which is configured to split the light entering the color separation assembly into a first imaging spectrum and a second imaging spectrum. The color separation assembly can include optical elements such as a beam splitter, particularly a dichroic beam splitter and / or optical filters such as a fluorescence filter that blocks light other than fluorescence emission and a white light filter that blocks fluorescence emission.

[0089] The medical fluorescence observation device can further include an illumination assembly, which is preferably adjustable by including, for example, LEDs or OLEDs with multiple different colors or other light sources that emit light in multiple different spectral bands and can be individually turned on and off.

[0090] To facilitate post-processing and enable the recorded color space coordinates of the digital white light image and the digital fluorescence color image to be compared with each other, it is preferable to record the digital fluorescence color image and the digital white light color image simultaneously and / or use the same exposure time and / or use the same gain and / or the same white balance and / or color correction. Furthermore, the gain of the digital fluorescence color camera and the gain of the digital white light color camera can be maintained at a certain ratio or automatically adapted. Among these possibilities, it is most preferable that the white light color camera and the fluorescence color camera are synchronized with respect to at least one of the exposure time and the exposure gain. However, the fluorescence color camera can be operated with a higher integration time, thereby compensating for low fluorescence intensity. The white light color camera and / or the fluorescence color camera are preferably a CCD camera or a CMOS camera. The fluorescence camera and the white light color camera are preferably the same.

[0091] According to another aspect, a phosphor-dependent set of different color conversion functions can be provided and stored, for example, in an image processor or a medical fluorescence device. Each phosphor-dependent set represents a different combination of a first imaging spectrum and a second imaging spectrum, i.e., a different set of filters used in a color separation assembly that is required when different phosphors or combinations of different phosphors are used. Examples of phosphors used in this context are 5-ALA / pPIX, fluorescein, and ICG. Each of these phosphors requires excitation at a different wavelength and fluoresces at a different wavelength. Thus, in a surgical environment where ICG is used, the first imaging spectrum and the second imaging spectrum are different from those in a surgical environment where 5-ALA / pPIX is used, and a different color conversion matrix is required. Accordingly, the sub-bands of the first imaging spectrum and the sub-bands of the second imaging spectrum in the color bands of each color space are different for each phosphor case.

[0092] The phosphor-dependent set can be selected automatically or manually from such a plurality of sets according to the phosphor used or the setting of the color separation assembly. For example, a medical fluorescence observation device can be configured to automatically detect the configuration or setting of the color separation assembly and select a color conversion matrix according to this setting. For example, when the optical filter of the color separation assembly for using ICG is replaced with that for using 5-ALA / pPIX, the medical fluorescence observation device can automatically select the color conversion matrix for this 5-ALA / pPIX.

[0093] The subject matter recited in the claims also relates to a computer-readable medium and a computer program including instructions for causing a computer to perform computer-implemented image processing in any of the above-described embodiments.

[0094] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0095] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent descriptions of corresponding methods, where a block or apparatus corresponds to a step or a feature of a step. Similarly, aspects described in the context of a step also represent descriptions of corresponding blocks or items or features of corresponding apparatuses.

[0096] Hereinafter, the present invention will be illustratively described with reference to embodiments and drawings. The combinations of configurations shown in these embodiments should not be regarded as limiting. For example, in the above, configurations of embodiments having technical effects that are not required in a particular application, for example, as described, may be omitted. Conversely, when the technical effects associated with this particular configuration are required in a particular application, the above-described configuration that is not part of the embodiments described later can be added.

[0097] Throughout this specification and the drawings, the same reference numerals are used for elements that correspond to each other with respect to function and / or structure.

Brief Description of the Drawings

[0098]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0099] Referring to FIG. 1, an exemplary embodiment of the present invention will be described.

[0100] FIG. 1 schematically shows a medical fluorescence observation apparatus 100. The medical fluorescence observation apparatus 100 may be a fluorescence microscope or a fluorescence endoscope. The difference between a microscope and an endoscope is mainly that in an endoscope (not shown), the object 106 is observed through an optical fiber that has reached the vicinity of the object 106 to be investigated, for example, by insertion into the body, while in a microscope, the objective lens 174 is directed towards the object. The medical fluorescence observation apparatus in FIG. 1 is a microscope, but the following description also applies to an endoscope. The medical fluorescence observation apparatus 100 may be a medical fluorescence observation apparatus used in surgery. The medical fluorescence observation apparatus 100 may also be a medical fluorescence observation apparatus used in a laboratory, such as a laboratory microscope. The object 106 may consist of or include a biological tissue 107.

[0101] The object 106 may include one or more phosphors 116, 118. At least one phosphor may be a phosphor naturally contained in the object. For example, bone and blood contain phosphors. At least one phosphor can also be added to the object 106, for example, by injecting it into the biological tissue 107. Examples of phosphors that can be added to the object 106 are ICG, fluorescein, and / or 5-ALA. 5-ALA is synthesized into pPIX / pPIX intracellularly.

[0102] The medical observation apparatus 100 is a fluorescence apparatus. That is, the medical fluorescence observation apparatus is configured to observe, record, and preferably also excite the fluorescence of one or more phosphors 116, 118.

[0103] The medical fluorescence observation apparatus 100 may be a stereoscopic apparatus as exemplarily shown in FIG. 1. Thus, the medical observation apparatus 100 can have two identical sub-assemblies 101L and 101R for each of the two stereoscopic channels. Since the two sub-assemblies 101L, 101R are identical in function and structure, the following describes the right sub-assembly 101R, but the same applies to the left stereoscopic channel 101L.

[0104] Alternatively, the medical fluorescence observation device 100 may be a planar viewing device. In this case, only one of the two sub-assemblies 101L, 101R can be present. Therefore, the following description also applies equally to the medical planar fluorescence observation device 100.

[0105] The medical fluorescence observation device 100 can be used to generate at least one digital white light color image 114 representing a reflectance image of the object 106 over the visible light range. The visible light range or visible spectrum includes wavelengths from about 310 nm to about 1100 nm, or from about 380 nm to 750 nm, or from about 450 nm to about 700 nm. When a fluorescence spectrum or when two or more phosphors are used, the fluorescence spectra of the phosphors 116, 118 are preferably omitted from the spectrum recorded in the digital white light color image 114. Thereby, when fluorescence is present, it is ensured that only the reflected light is included in the digital white light color image 114. For example, when 5-ALA / pPIX is used as the phosphor, the fluorescence spectrum from about 625 nm to about 650 nm may not be recorded in the digital white light color image 114.

[0106] When light of a specific wavelength is used to excite fluorescence, the spectrum including or consisting of these wavelengths may also not be recorded or displayed in the digital white light color image 114. For example, when 5-ALA / pPIX is used as the phosphor, fluorescence can be excited by illuminating the object 106 with light having a wavelength from about 380 nm to about 450 nm. For fluorescein and ICG, as well as other fluorescent substances, known ranges different from those of 5-ALA / pPIX apply to the excitation spectrum and the emission spectrum.

[0107] By not recording the fluorescence excitation spectrum and the fluorescence emission spectrum, the digital white light color image 114 preferably represents the reflectance of the object 106, i.e., the white light image of the object 106 that a human observer would see. Thus, the digital white light color image 114 can be regarded as a true color image. However, in some embodiments, at least a part of the emission spectrum or the emission spectra of some phosphors may be recorded in the digital white light color image 114.

[0108] Furthermore, the digital imaging system 102 can be used to generate a digital fluorescence color image 112 of the object 106. The digital fluorescence color image 112 represents the fluorescence emission of one or more phosphors 116, 118. Thus, it is preferred that the digital fluorescence color image 112 does not record wavelengths outside the emission spectra of the one or more phosphors.

[0109] Both the digital white light color image 114 and the digital fluorescence color image 112 are color images. They are recorded using at least three color bands, i.e., the primary colors of the color space. For example, the digital white light color image 114 and the digital fluorescence color image 112 may be recorded in the RGB color space using the three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescence color image 112 may be recorded in different color spaces respectively, and / or may represent multi-spectral or hyperspectral color images. The digital white light color image 114 and the digital fluorescence color image 112 do not need to be recorded in the same color space, but it is advantageous to record them in the same space.

[0110] The digital white light color image 114 and the digital fluorescence image 112 include pixels 150. In a color space such as the RGB color space, each color of the pixel 150 is represented by a set of three integers, where each integer indicates the intensity of one of the primary colors R, G, B. For example, the strongest red color can be indicated by the set of three integers {255, 0, 0}. The darkest green color can be indicated by {0, 255, 0}, and the darkest blue color can be indicated by {0, 0, 255}. Therefore, the RGB color space is a three-dimensional space, and the CMYK color space would be a four-dimensional space. Color can be regarded as a point in the color space having color space coordinates such as {0, 0, 255}. Similarly, a multi-spectrum or hyperspectrum color space having n color bands is an n-dimensional color space, and each color is represented by a set of n values.

[0111] The spectrum recorded and displayed in the digital white light color image 114, the first imaging spectrum, and the spectrum recorded in the digital fluorescence color image 112, the second imaging spectrum, are complementary to each other, that is, they do not overlap, except for inevitable filter leakage. Preferably, both of them represent the complete visible light spectrum.

[0112] More specifically, the medical observation device 100 can include a digital imaging system 102 for generating the digital fluorescence color image 112 and the digital white light color image 114. The digital imaging system 102 can include a white light color camera 110 and a fluorescence color camera 111.

[0113] The white light color camera 110 is configured to record a digital white light color image 114. In particular, the white light color camera 110 can be configured to generate a stream of the digital white light color image 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images over the entire visible spectrum at the wavelengths indicated above. The white light color camera 110 can be a CCD, CMOS, or multi-spectral camera or hyperspectral camera.

[0114] The fluorescence color camera 111 is configured to record a digital fluorescence image 112. In particular, the fluorescence camera 111 can be configured to generate a stream of the digital fluorescence color image 112 in the form of a digital video stream. The fluorescence color camera 111 can be configured to record the digital fluorescence color image 112 only in the fluorescence spectra of one or more phosphors 116, 118. The fluorescence camera 111 can be configured to record digital fluorescence images only in one or more narrow optical bands. These narrow bands should overlap the fluorescence spectra of one or more phosphors 116, 118 in which fluorescence is to be recorded. The fluorescence spectra of the phosphor 116 and the second phosphor 118 are preferably at least partially, preferably completely separate, so that the fluorescence camera 111 can record light from two separate fluorescence bands spaced apart from each other.

[0115] The fluorescence color camera 111 can be a CCD camera, CMOS camera, or multi-spectral camera or hyperspectral camera. The white light color camera 110 and the fluorescence color camera 111 are preferably of the same type, but this is not essential. Usually, since the intensity of fluorescence is very low, the fluorescence color camera 111 can have a higher integration time.

[0116] The respective fields of view 184 of cameras 110 and 111 are preferably aligned, or even more preferably, coincide and are coaxial. Thus, advantageously, cameras 110 and 111 provide the same field of view 184 with the same viewpoint and focal length. As a result, in images 112 and 114 generated by different cameras 110 and 111, the same display of the object 106 is obtained. Both cameras 110 and 111 can use the same objective lens 174.

[0117] If the viewpoints and fields of view cannot be optically matched, they can be matched by image processing by applying a matching routine or an alignment routine to the digital images 112 and 114, as further described below. Registration can also be performed when cameras 110 and 111 have the same viewpoint and field of view.

[0118] The two cameras 110 and 111 preferably operate synchronously. Specifically, the exposure times can be synchronized. Thus, the medical fluorescence observation device 100 can be configured to generate a digital white light color image 114 and a digital fluorescence image 112 simultaneously.

[0119] The gains of the two cameras 110 and 111 are preferably synchronized, that is, adjusted simultaneously in the two cameras 110 and 111. Further, the ratio of the gain applied in camera 110 to the gain applied in camera 111 may be constant even if the gain varies. Gamma correction and color adjustment or white balance may be turned off or kept constant.

[0120] Any of the means described above facilitates the comparison, connection processing, and / or combination of the two images 112 and 114.

[0121] To separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescence color image 112, i.e., to separate the reflectance spectrum from the fluorescence spectrum, the medical observation device 100 can include an optical color separation assembly 176. The color separation assembly 176 can include optical elements such as a beam splitter 192, which may be dichroic. The color separation assembly 176 can further or alternatively include an optical white light filter 188 and / or an optical fluorescence filter 190.

[0122] The fluorescence filter 190 is preferably configured to transmit light of the fluorescence spectra of one or more phosphors 116, 118 and block light outside the fluorescence spectra.

[0123] The fluorescence filter 190 may be configured as a band - pass filter including one or more pass bands. Each pass band should overlap the fluorescence emission spectra of each phosphor 116, 118 for which fluorescence is recorded. Since the fluorescence filter 190 is in the optical path between the beam splitter 192 and the fluorescence camera 111, only the wavelengths of the pass bands of the fluorescence filter 190 are transmitted to the fluorescence camera 111.

[0124] The white light filter 188 is preferably configured to block light of the fluorescence spectra of one or more phosphors 116, 118. The white light filter 188 may be configured to block light of the fluorescence excitation spectra.

[0125] The white light filter 188 is preferably configured as a band - stop filter, and its stop band corresponds to or at least includes the pass bands of the fluorescence filter 190. The white light filter 188 is disposed in the optical path between the beam splitter 192 and the white light camera 110. Thus, the white light camera 110 records only wavelengths that are outside the stop band of the white light filter 188 and thus also outside the pass bands of the fluorescence filter 190.

[0126] Either the white light filter 188 or the fluorescence filter 190 may be an adjustable filter.

[0127] If the beam splitter 192 is a dichroic beam splitter, in this case, since optical spectral filtering is already incorporated in the dichroic beam splitter, at least one of the filters 188, 190 can be omitted. The above description regarding the passband and stopband should be applied mutatis mutandis to the dichroic beam splitter 192.

[0128] Accordingly, the white light color camera 110 records the digital white light color image 114 in the first imaging spectrum, and the reflectance spectrum, which is different from the second imaging spectrum, is recorded as the fluorescence spectrum by the fluorescence camera. The wavelengths included in the first imaging spectrum and the second imaging spectrum are determined by the filter settings of the color separation assembly 176.

[0129] The medical fluorescence observation device 100 may further include an illumination assembly 178, which is configured to illuminate the object 106 through the objective lens 174, and preferably, through the objective lens 174, the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 can be configured to selectively generate white light, i.e., light uniformly distributed over the entire visible spectrum, and fluorescence excitation light including only light of wavelengths that stimulate the fluorescence of at least one phosphor 116, 118. The illumination light generated by the illumination assembly 178 can be supplied to the objective lens 174 using the illumination beam splitter 180.

[0130] The illumination filter 179 can be provided according to a phosphor and a fluorescence-specific excitation spectrum of the phosphor. For example, when using 5-ALA / pPIX as the phosphor, the illumination filter can have a transmittance of 90% to 98% up to a wavelength of 425 nm, a transmittance of 0.5% to 0.7% at wavelengths of 450 nm to 460 nm, a transmittance of 0.1% or less at wavelengths of 460 nm to 535 nm, and the transmittance may be substantially zero at wavelengths exceeding 535 nm.

[0131] Instead of, or in addition to, the illumination filter 179, the illumination assembly 178 can include, for example, an adjustable light source including a plurality of differently colored LEDs or OLEDs.

[0132] The medical fluorescence observation device 100 can further include an image processor 170. The image processor 170 can be a hardware module such as a microprocessor, or a software module. The image processor 170 can also be a combination of both a hardware module and a software module, for example, by using a software module configured to be executed on a specific processor such as a vector processor, a floating-point graphics processor, a parallel processor, and / or a plurality of processors. The image processor 170 can be a part of a general-purpose computer 186, such as a PC.

[0133] The image processor 170 is configured to acquire a digital white light color image 114 and a digital fluorescence image 112. For example, the image processor 170 can be configured to acquire the digital white light color image 114 and the digital fluorescence image 112 from the memory 194 and / or directly from the cameras 110, 111. The memory 194 can be a part of the image processor 170 or can be provided elsewhere in the medical fluorescence observation device 100.

[0134] The image processor 170 is further configured to calculate a digital output color image 160 from a digital white light color image 114 and a digital fluorescence image 112. The digital output color image 160 is a color image represented in a color space. The color space of the digital output color image may be different from the color space of either the digital white light color image 114 or the digital fluorescence image 112. However, the color space of the digital output color image 160 is preferably the same color space as the color spaces of the digital white light color image 114 and the digital fluorescence image 112.

[0135] The image processor 170 includes at least two operating modes configured to operate selectively. For example, a user (not shown) can manually select which of the two operating modes is to be executed. A selector device 165 can be provided to select an operating mode from at least two available operating modes. The selector device 165 can be a mechanical device such as a dial or a switch, or a button or other element on a graphical user interface displayed on a display such as displays 132, 182, which will be described in more detail below.

[0136] As shown in FIG. 1, the image processor 170 can have a first operating mode A, a second operating mode B, and optionally a third operating mode C. Alternatively, the image processor 170 can be configured to operate in only a single operating mode, for example, one of modes A, B, and C.

[0137] In the first operation mode A, the image processor 170 is preferably configured to generate a digital output color image 160 representing what is seen through the eyepiece 104. To achieve this, in operation mode A, the image processor 170 can be configured to combine the digital fluorescence color image 112 and the digital white light color image 114 to generate the digital output color image 160. In particular, for each corresponding pixel 150 in the digital fluorescence color image 112 and the digital white light color image 114, the image processor 170 can be configured to additionally combine or add the values of the digital fluorescence color image 112 and the digital white light color image 114 in each color band of their color spaces. From this combination, the color at the output pixel of the digital output color image 160 is generated.

[0138] In the second operation mode B, the image processor 170 is first configured to apply the color conversion function 140 to at least one of the digital white light color image 114 and the digital fluorescence color image 112, and then to generate the digital output color image 160 from the digital white light color image 114 and the digital fluorescence color image 112. Thus, in the second operation mode B, after at least one of the digital white light color image 114 and the digital fluorescence color image 112 is color-converted, they are combined. By switching between the first operation mode A and the second operation mode B, the user can switch between a view that faithfully renders what is seen through the eyepiece and a view that can highlight various features of the object 106.

[0139] The color conversion function 140 is configured to map the color of one input pixel, or two or more input pixels, to a different color in the output pixel. Therefore, the color conversion function changes the input color in a predictable manner when applied. The color conversion function 140 may be any type of function, such as a one-dimensional or n-dimensional interpolation function. However, the color conversion function 140 is preferably a linear conversion function. Specifically, the color conversion function 140 may be a color conversion matrix 142. One dimension of the color conversion matrix 142 may be the sum of the number of color bands in the digital fluorescent color image 112 and the number of color bands in the digital white light color image 114. Another dimension of the color conversion matrix 142 may correspond to the number of color bands in the digital output color image 160 in another direction of another direction of the matrix.

[0140] In another embodiment, the color conversion matrix 142 may have a first dimension corresponding to the number of color bands in the digital white light color image 114 or the digital fluorescent color image 112, depending on which of the two color conversion matrices is applied, and a second dimension corresponding to the number of color bands in the digital output color image 160.

[0141] If all the images 112, 114 are in the RGB color space, the dimensions of the color conversion matrix 142 are preferably a 3×3 matrix, or a 6×3 or 3×6 matrix if the color conversion matrix is configured to operate simultaneously on the digital white light color image and the digital fluorescent color image. The color conversion matrix may be the result of applying two or more matrices successively. For example, the color conversion matrix 142 can be obtained by first applying a 6×1 color conversion matrix and then a 1×3 color conversion matrix, which has the same effect as applying a 6×3 color conversion matrix.

[0142] The second operation mode B itself can include different sub - modes. Each sub - mode can apply a different color conversion function 140 to at least one of the digital white - light color image 114 and the digital fluorescence color image 112 before the digital fluorescence color image 112 is combined with the digital white - light color image 114. In FIG. 1, by way of example only, three sub - modes B - I, B - II, B - III are shown. Of course, more or fewer than three sub - modes may be provided. Different color conversion functions 140 can be stored in the memory 194.

[0143] For example, the first color conversion function 140a can be applied only to the digital white - light color image 114, and the second color conversion function 140b can be applied only to the digital fluorescence color image 112. The first color conversion function 140a and the second color conversion function 140b can be applied independently of each other, and the color conversion of the digital white - light color image 114 can be separated from the color conversion of the digital fluorescence color image 112. For example, only one of the first color conversion function 140a and the second color conversion function 140b can be applied to each of the images 112, 114, or both can be applied to their respective images.

[0144] In one embodiment, the user can select from various first color conversion functions 140a and / or various second color conversion functions 140b.

[0145] For example, in operation mode B - I, the first color conversion function 140a can be used, and this first color conversion function 140a is configured to convert the color in the digital white - light color image 114 to a color that is not located within the second imaging spectrum. In this case, the natural color or a part of the object 106 that is partially filtered and thus distorted by the color separation assembly 176 and not accurately recorded in the digital white - light color image 114 can be converted by the first color conversion function 140a to a color that most closely corresponds to a more natural color.

[0146] In other additional or alternative examples, the first color conversion function 140a can be used in an additional operation mode that converts colors in the operation mode B-I or the digital white light color image 114 to pseudo colors or different hues. Thereby, colors related to certain tissues can be highlighted. For example, the colors of nerves, arterial blood, and / or venous blood can all be converted to different pseudo colors.

[0147] By using the selector device 165, the user can quickly assign different pseudo colors to different types of tissues. In one operation mode, the color of arterial blood can be converted to neon red, in another operation mode, the color of venous blood can be converted to neon blue, and in yet another operation mode, the color of nerve tissue can be converted to neon yellow. Other operation modes can use combinations of these modes. If the first color conversion function 140a assigns different hues instead of pseudo colors, the contrast can be improved. For example, the first color conversion function 140a can spread the red color in the digital white light color image 114 over a wider range, thus making the changes in blood oxygenation smaller and easier for the user to see.

[0148] In still other additional or alternative examples, the first color conversion function may be used in the operation mode B-I or an additional operation mode for, for example, color balance adjustment and / or white balance adjustment of the digital white light color image 114. In white balance, the overall color mixture is changed, but intermediate colors, namely gray, black, and white, are maintained as intermediate colors. In color balance, colors including intermediate colors are adjusted to correspond to the intermediate colors.

[0149] In yet other additional or alternative examples, the first color conversion function 140a can be used, for example, in operation mode B-I or an additional operation mode, to shift the white point of the digital white light color image 114 to a predetermined position in the color space. By adjusting the white point, it becomes possible to adjust the filter settings of various illuminations and color separation assemblies 176.

[0150] According to one example, at least one of the first color conversion functions 140a described above can be applied to the digital white light color image 114. According to other examples, at least two of the first color conversion functions 140a described above can be sequentially applied to the digital white light color image 114. Alternatively or additionally, two or more of the color conversion functions 140a described above may be combined into a single color conversion function 140a.

[0151] The above description regarding the various first color conversion functions 140a also applies mutatis mutandis to the various second color conversion functions 140b, and the only difference between these two functions is that the second color conversion 140b operates on the digital fluorescent image 112 and is configured to operate on different colors.

[0152] In operation mode B-II, for example, one or more color conversion functions 140b can be applied only to the digital fluorescent image, and the color conversion function 140a is not applied.

[0153] In operation mode B-III, for example, one or more color conversion functions 140a can be applied to the digital white light color image, and one or more color conversion functions 140b can be applied to the digital fluorescent color image 112.

[0154] It is possible to provide any third operation mode C in which the third color conversion function 140c is applied simultaneously to the digital white light color image 114 and the digital fluorescent color image 112, and as a result of the application of the color conversion function 140c, the digital output color image 160 is directly obtained.

[0155] When other types of phosphors having different excitation wavelengths and / or fluorescence wavelengths are used, it is necessary to provide different color conversion functions 140a, 140b, 140c because the combination of filter settings in the color separation assembly 176, and thus the first imaging spectrum and the second imaging spectrum, need to be adapted to the excitation wavelength and / or fluorescence wavelength.

[0156] Thus, in one embodiment, the medical observation device 100 or the processor 170 can be configured to store a plurality of different sets 143 of the color conversion functions 140a, 140b and / or 140c. The different sets 143 each include one or more of the above-described color conversion functions 140a, 140b and / or 140c. The different sets 143 each correspond to the use of different phosphors and thus represent different filter settings of the color separation assembly 176. For example, the first set 143a can be used for ICG as a phosphor, while the second set 143b can be used for 5-ALA / pPiX as a phosphor, and of course, additional sets may be used to correspond to further phosphors or combinations of phosphors.

[0157] The medical fluorescence observation device 100 can be adjusted for different phosphors by reconfiguring the color separation assembly 176, for example, by replacing its own optical elements such as the filter 190 and / or the filter 188 or the dichroic beam splitter 180. The selection of the appropriate set 143 of the color conversion function 140 can be automatically performed, for example, when the medical observation device 100 is configured to automatically detect the settings of the color separation assembly 176.

[0158] Alternatively or additionally, the medical observation device 100 can include a filter setting selector device 168 by which a user can manually select a set 143 of color conversion functions 140 to be applied to the digital white light color image 114 and / or the digital fluorescence color image. The filter setting selector device 168 can be part of a mechanical and / or graphical user interface. For example, when the filter setting selector device 168 is operated to position a, set 143a is selected. At position b, set 143b is selected, and so on.

[0159] In another embodiment, the filter setting selector device 168 allows a user to select different image components of the digital white light color image 114 and the digital fluorescence color image 112. For example, at one position of the filter setting selector device 168, only the excitation fluorescence of one or more phosphors is shown. At another position, only autofluorescence can be shown. Also, at other positions of the filter setting selector device, a combination of a reflectance image of an object illuminated only with the fluorescence excitation spectrum and the excitation fluorescence of at least one phosphor 116, 118 may be shown.

[0160] The digital output color image 160 can be displayed on a display 132 integrated with the medical fluorescence observation device 100. For example, the display 132 can be incorporated into the eyepiece or ocular lens 104 of the medical fluorescence observation device 100. The display 132 can also display a graphical user interface for operating the medical observation device 100.

[0161] The medical fluorescence observation device 100 can include a direct optical path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In such a case, the display may be a translucent display 132 disposed in the direct optical path 134, or the display may be projected into the direct optical path 134. The beam splitter 136 can be provided to split light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light can be directed towards the eyepiece 104.

[0162] Alternatively, the medical fluorescence observation device 100 may not have a direct optical path 134 and may only display an image from the integral display 132. As yet another option, the medical fluorescence observation device may not have any display at all.

[0163] The medical fluorescence observation device 100 can include an output interface 172 that can connect one or more (external) displays 182. For this purpose, the output interface 172 can include a standardized connector and a data transmission protocol such as USB, HDMI, DVI, DisplayPort, Bluetooth, and / or others. The external display can be a monitor, 3D goggles, eyeglasses, etc. Any combination of external displays can be connected to the output interface 172. Any of the displays 182 can display a graphical user interface for operating the medical observation device 100.

[0164] The computer 186 or the image processor 170 is connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission lines may be wired or wireless, or may be partially wired or partially wireless. The computer 186 and / or the image processor 170 need not be entirely incorporated into the medical fluorescence observation device 100, but can be physically separated from the digital imaging system 102. For this purpose, the digital imaging system 102 and the computer 186 and / or the image processor 170 can be connected to a network such as a LAN, WLAN or WAN, and at least one display 182 is also connected to this network.

[0165] According to one variant, the medical fluorescence observation device 100 may be stereoscopic, but includes only two cameras, with one camera in each stereoscopic channel. In one stereoscopic channel, a fluorescence color camera 111 is used and is configured to selectively record the white light reflectance as well. In the other stereoscopic channel, a white light color camera 110 is used. With such a configuration, a white light color image for stereoscopic viewing is provided when fluorescence is not used, and a white light color image for planar viewing and a fluorescence color image for planar viewing are provided when fluorescence is used. The above and below descriptions equally apply to this configuration.

[0166] FIG. 2 shows an example of the first operation mode A.

[0167] The reference numeral 200 is a quantitative example of the first imaging spectrum 202 because it is respectively recorded by the digital white light camera 110 and / or represented in the digital white light color image 114. The intensity I over the wavelength / color λ is shown normalized. The first imaging spectrum 202 preferably extends over at least the visible spectrum 212.

[0168] As a mere example, the color space in which the first imaging spectrum 202 is recorded is an RGB color space having three primary colors or color bands 204, 206, and 208. One primary color 204 is blue, another primary color 206 is green, and the third primary color 208 is red. The sensitivities of the sensors of the white light color camera 110 in the different primary colors 204, 206, 208 are adjusted so as to make the sensitivity over the visible spectrum 212 as constant as possible.

[0169] If a color space other than RGB is used, the number, position, and / or width of the color bands may be different. Otherwise, there is no significant difference from the RGB color space.

[0170] The first imaging spectrum 202 does not include the fluorescence excitation light and fluorescence emission spectrum of at least one phosphor 116, 118. Accordingly, the first imaging spectrum 202 can include at least one blocking band 210 that coincides with the fluorescence emission of at least one phosphor whose fluorescence is to be recorded by the fluorescence color camera 111. The blocking band 210 is generated, for example, by the white light filter 188. The number, width, and / or position of the blocking band 210 depend on the number and type of phosphors observed in the object 106.

[0171] At reference numeral 220, a second imaging spectrum 222 is shown as being recorded by the fluorescence color camera 111 and / or as represented in the digital fluorescence color image 112. As a mere example, the color space in which the second imaging spectrum 222 is recorded is also an RGB color space. The sensitivities of the sensors of the fluorescence color camera 111 in the different primary colors 204, 206, 208 are adjusted so as to make the sensitivity over the visible spectrum 212 as constant as possible.

[0172] The spectra 202, 222 need not be recorded in the same color space, but it is preferable that they be recorded in the same color space.

[0173] The second imaging spectrum 222 can include one or more passbands 224. The number, position, and / or width of the passbands depend on the number and types of phosphors used. At least one passband 224 preferably corresponds to at least one stopband 210. At least one passband is generated, for example, by the fluorescence filter 190.

[0174] The first imaging spectrum 202 and the second imaging spectrum 222 are complementary to each other. Preferably, they complement each other to cover the whole or most of the visible spectrum 212.

[0175] Each passband 224 of the second imaging spectrum 222 preferably overlaps with the fluorescence emission spectra 226, 228 of the phosphors 116, 118 whose fluorescence is to be recorded and also overlaps with one or more primary colors 204, 206, 208 of the color spectrum. For example, the fluorescence emission spectrum 226 of one phosphor 116 can overlap with all of the three primary colors 204, 206, 208. The sensor of the fluorescence camera 111 that records the color band 204 records only a small portion 230 of the fluorescence emission spectrum 226. The sensor of the fluorescence camera 111 that records the color band 206 records most of the fluorescence emission spectrum 226, while the sensor of the fluorescence camera 111 that records the color band 208 also records only a small portion 236 of the fluorescence emission spectrum 226 here.

[0176] In contrast, the fluorescence spectrum 228, if present in a particular case, is recorded only by the fluorescence camera 111 that records the color band 208.

[0177] Some phosphors, such as 5-ALA / pPiX, can have a very broad fluorescence emission spectrum 226. In such cases, since too many wavelengths would be missing in the white light reflectance image, it may not be desirable to expand the passband 224 to cover the entire fluorescence emission spectrum. Thus, the passband 224 may not cover the entire fluorescence emission spectrum 226. As a result, some of the fluorescence can be included in the first imaging spectrum 202 and recorded in the digital white light color image 114. When recorded under a standard light source as a white light reflectance image, the intensity of the fluorescence is much weaker than the intensity of the reflected white light, so this does not impair the digital white light color image 114. Further, any fluorescence recorded in the digital white light color image can be used to complement the fluorescence information in the digital fluorescence color image.

[0178] In FIG. 2, of the fluorescence emission spectrum 226, the portions included in the first imaging spectrum 202 are indicated by reference numerals 260 and 262, respectively. Reference numeral 260 indicates a wavelength of the fluorescence emission spectrum 226 that is shorter than the cut-off wavelength of the passband 224. Reference numeral 262 indicates a wavelength of the fluorescence emission spectrum 226 that is longer than the cut-off wavelength of the passband 224.

[0179] The fluorescence emission in the first imaging spectrum 260 can be captured or recorded in all color bands 204, 206, 208. The first portion 260 of the fluorescence emission spectrum 226 in the first imaging spectrum can overlap, for example, color bands 204 and 206. The second portion 262 can overlap color bands 206 and 208.

[0180] When the digital white light color image 114 is used not to record the white light reflectance image but to record only fluorescence, for example, when the object is not illuminated by a white light standard light source but by a fluorescence excitation spectrum containing only wavelengths shorter than the wavelengths in the fluorescence emission spectrum, the digital white light color image can also represent the fluorescence image of the object 106. In such a case, the fluorescence information is included in spectral bands that do not overlap in both the digital white light color image 114 and the digital fluorescence color image 112.

[0181] The fluorescence excitation light illuminating the object can be blocked in the color separation assembly 176 by, for example, appropriate filters 188 and / or 190 and / or an appropriate dichroic beam splitter 192. However, in another embodiment, the fluorescence excitation light can be recorded in at least one of the digital white light color image 114 and the digital fluorescence color image 112. In the latter case, some reflectance information that can be used in post-processing is preserved.

[0182] In its simplest form, the combination 240 of the digital white light color image 114 and the digital fluorescence color image 112 is an additional combination. For example, the values in each color band of the digital white light color image 114 and the fluorescence color image 112 are added at each corresponding pixel 150. The corresponding pixel 150 may be a pixel located at the same location within both images 112, 114 when the images 112, 114 are registered and of the same size. If both images 112, 114 are RGB images, the values in the RGB color bands of the pixel 150 in the digital white light color image are {R1, G1, B1}, and the values in the RGB color bands of the corresponding pixel 150 in the digital fluorescence color image are {R2, G2, B2}, then the values in the RGB color bands of the corresponding pixel 150 in the digital output color image are {R1 + R2, G1 + G2, B1 + B2}. Since the color of the pixels in the output image is different from the color of the pixels in the images 112, 114, the combination 240 consequently results in a color transformation. Thus, the combination 240 is an example of the color conversion function 140. The combination 240 can be executed using the color conversion matrix 142.

[0183] Reference numeral 250 indicates a qualitative rendition of the spectrum 252 of the digital output image 160. In short, the spectrum 252 of the digital output image 160 is the sum of the first imaging spectrum 202 and the second imaging spectrum 222.

[0184] Alternatively, the individual spectra 202 and 222 are treated as a single combined multi - spectrum having a number of color bands corresponding to the sum of the color bands in the first imaging spectrum 202 and the second imaging spectrum 222. In other words, the digital fluorescence color image 112 and the digital white light color image 114 are processed together as a single (virtual) multi - spectrum image. Instead of adding the color space coordinates in each color band of the images 112, 114, a combined set is formed and processed while keeping the color bands of the images 112, 114 separate.

[0185] This corresponds to operation mode C (Figure 1): In operation mode C, color conversion functions 140, 140c are applied to both the digital white light color image 114 and the digital fluorescence color image 112 to obtain a digital output color image 160 whose spectrum is indicated by reference numeral 250 in Figure 2.

[0186] Referring to Figure 3, this will be described below. For the sake of brevity, Figure 3 shows the spectrum of Figure 2 in the RGB color space and shows only a single passband 224 and stopband 210.

[0187] The digital white light color image 114 recorded at the first imaging spectrum 202 is composed of signals R1, G1, and B1 that respectively represent the intensity I of each color band. Due to the stopband 210, the signal G1 results from two separated wavelength bands. However, the sensor that records G1 cannot distinguish these two wavelength bands.

[0188] The digital fluorescence color image 112 recorded at the second imaging spectrum 222 is composed of signals R2, G2, B2 for each color band.

[0189] Since the first imaging spectrum 202 and the second imaging spectrum 222 are complementary to each other, each color band is divided into two signals, and each signal is transmitted from different cameras 110, 111 and represents or corresponds to a sub-band of the color band. Color band 204 is divided into R1 and R2 respectively. Color band 206 is divided into G1 and G2 respectively, and color band 208 is divided into B1 and B2 respectively. It is preferable that there is no overlap between the various signals in the color band.

[0190] Thus, even if the first imaging spectrum 202 and the second imaging spectrum 222 are recorded in the same color space using the same type of color camera, a multi-spectral image of the object 106 is produced, which in this case consists of six color bands R1, R2, G1, G2, B1, B2. This is schematically shown by reference numeral 300. It should be noted that this has nothing to do with whether the fluorescence camera actually records fluorescence emission or reflectance.

[0191] In other words, at least one color band 204, 206, 208 of at least the color space of the digital output color image 160 is subdivided into two sub-bands R1, R2, G1, G2, B1, B2, and one of these two sub-bands is included in the digital fluorescence color image 112, and the other of the two sub-bands is included in the digital white light color image 114. The two sub-bands within a color band preferably do not overlap. They are preferably complementary. Most preferably, together they at least substantially complete their respective color bands. A sub-band may include two separate spectral bands or may consist of two separate spectral bands. Each sub-band can be considered a color band in itself.

[0192] The color conversion function 140 reflects the subdivision of the color band into sub-bands determined by the stop band 210 and the pass band 224. Since the width and / or position of the sub-bands determine how much light each of the cameras 110, 111 collects in the respective sub-band, the color conversion function 140c needs to be adjusted for each different filter setting of the color separation assembly 176.

[0193] Thus, the digital white light image 114 and the digital fluorescence image 112 can be processed together as an image composed of the sum of the color bands in the two images 112 and 114. This provides improved color resolution.

[0194] For example, the color conversion function 140c can generate the digital output color image 160 by being applied in the form of a linear transformation using the color conversion matrix 142.

[0195] When the digital output color image 160 is represented in the RGB color space, a 6×3 or 3×6 color conversion matrix 142 is applied to the combined digital fluorescence image 112 and digital white light image 114, or to their component signals R2, G2, B2, R1, G1, B1 respectively, to obtain the RGB signals R * , G * , B * . The matrix coefficients C11, C12 ··· C63 can be determined experimentally.

[0196] It should be noted that the color conversion matrix 142 is substantially obtained as a result of sequentially applying a plurality of matrices.

[0197] In FIG. 3, the input combination, or equivalently, the input combination set {R1, R2, G1, G2, B1, B2} is formed from the set of color space coordinates {R1, G1, B1} within the color bands R, G, B of the (first) pixel of the digital white light color image 114 and the set of color space coordinates {R2, G2, B2} within the preferably corresponding (second) pixel of the digital fluorescence color image 112. The input combination set corresponds to the pixels of the multispectral image. Next, the color conversion matrix is applied to the input combination. No additional memory is required to form the input combination. The input combination can logically be generated, for example, by combining pointers to the color space coordinates of the first and second pixels. Of course, the color space coordinates of these two pixels can also be copied to the memory that physically stores the input combination.

[0198] Next, the operation mode B will be described.

[0199] FIG. 4 shows a schematic reflectance spectrum 400 of the biological tissue of object 106, where the term tissue includes both fluid tissue and solid tissue. For example, the reflectance spectrum 400 may correspond to the reflectance spectrum of blood containing oxygen.

[0200] The spectral sensitivities R, G, B of the R, G, B sensors of the sample RGB color camera are also shown in FIG. 4.

[0201] In human perception, tissues such as blood containing oxygen are perceived as having a natural color, i.e., red, under white light illumination such as a CIE light source. Therefore, although the reflectance spectrum 400 spreads widely over the visible light spectrum, it is perceived as a single natural color. In the first imaging spectrum 202 where the digital white light color image 114 is recorded, the wavelength λ in the stop band 210 is not recorded. Therefore, the reflectance spectrum 400 represented in the digital white light color image 114 does not faithfully represent the natural color of the tissue.

[0202] The same applies to the second imaging spectrum 222 and the digital fluorescence color image 112, where distortion also occurs in the reflectance spectrum 400.

[0203] The same is true when the spectrum 400 is not the reflectance spectrum but the fluorescence emission spectrum 226 recorded in both the digital white light color image 114 and the digital fluorescence color image 112 as shown in FIG. 2. Since the peak intensity and / or a larger portion of the fluorescence energy of the fluorescence emission spectrum is within the stop band 210, it is recorded by the fluorescence camera.

[0204] Therefore, when recording only fluorescence using both the fluorescence color camera 111 and the white light color camera 110, both cameras correspond to the case of recording the white light reflectance. In the former case, the spectral fluorescence information is mainly recorded by the fluorescence color camera 111 and supplemented by the spectral information recorded by the white light color camera 110; in the latter case, the spectral white light reflectance information is mainly recorded by the white light color camera 110 and supplemented by the spectral information recorded by the fluorescence color camera 111.

[0205] FIG. 5 shows a schematic rendition of a CIE1932 color space diagram in which color 500 is shown. Color 500 can represent, for example, the natural color perceived by a standard observer when viewing a tissue having a reflectance spectrum 400 (FIG. 4) illuminated by CIE light source A, B, or C or other standard light sources.

[0206] Since wavelengths within the stop band 210 are not recorded, color 500 is represented within the digital white light color image 114 as the (recorded) color 502 in the digital white light color image 114. To more faithfully represent color 500, the recorded color 502 is converted to color 500 by a color conversion function 140 which may be the first color conversion function 140a or the second color conversion function 140b. For example, the color conversion may be a shift or transformation in the LMS color space or tristimulus coordinates.

[0207] The color conversion function 140 can be determined, for example, by color calibration using known colors on a color card, known filter settings, and known illumination conditions. Color 500 may be of wavelengths that are within the stop band 210 and thus not within the first imaging spectrum 202. Along with the shift from 502 to 500, the entire color space may be shifted.

[0208] In some applications, it may be beneficial if the color 500 to which color 502 is converted by the color conversion function is not a natural color, but a pseudo-color or a color that is close to but visually different from the natural color. This can be used to shift this color and the associated tissue type away from other tissues or fluorescent colors close to color 500, thereby highlighting this type of tissue to the viewer's eye. For example, color 500 in the case of oxygenated blood may be a "hyper-real" color, such as a red that is brighter than the natural color of oxygenated blood, or a red pseudo-color such as neon red. Of course, such a conversion can also be performed on any other color or tissue type.

[0209] Due to the stop band 210, the position of the natural white point 504 is shifted to the recorded white point 506. Thus, other color conversion functions 140, such as color conversion function 140c, can shift the recorded white point 506 to the natural white point 504 or bring it closer to the natural white point 504. In this regard, the white point is treated in the same way as any other color.

[0210] A shift of a single color may correspond to a shift of the entire color space, i.e., all colors are shifted by the same amount.

[0211] Using a shift of the entire color space by the same amount and in the same direction, the digital white light color image 114 can be white balanced or color balanced.

[0212] Alternatively, different colors or different sets of colors may be transformed or shifted differently with respect to at least one of the amount and direction of the shift. As an example, for instance, the recorded color 508 of blood that has lost oxygen can be shifted further into the blue color range to become color 510. The shift from color 508 to color 510 may be different from the shift from color 502 to color 500. Thus, instead of transforming the entire color space, the color conversion function 140 can be configured to produce a color-dependent conversion. Different color conversion functions 140 can be applied to different (recorded) colors 502, 504, 508.

[0213] Another example of the color conversion function 140, for example 140d, can expand the region 512 in the color space into a larger region 514. In the region 514, since the distance between colors is larger than that in the region 512, the difference between colors becomes easier to see. The application of such a color conversion function can be color-dependent. In this case, only the region 512 of a predetermined color is expanded. Since this color conversion function 140 may include a color shift, the region 514 is not only expanded but also moved to different regions of the color space.

[0214] Any of the above color conversion functions 140 can only convert a selected subset of the color appearance parameters. The color appearance parameters are hue, chroma, saturation, lightness, and luminance. For example, when converting color 502 to color 500, only the hue can be changed, or only the chroma and lightness can be changed by the color conversion function 140.

[0215] Any combination of the above color conversion functions 140a can be applied. Although the expression "image" is used in the above description, it should be understood that the color conversion function 140 is applied to the pixel-level image, that is, each pixel 150 of the digital white light color image 114. The color of the pixel 150 can determine which color conversion function 140 is applied or which combination of color conversion functions 140 is applied. This can be automatically done by the processor 170.

[0216] FIG. 8 shows an example of color-dependent color conversion. The digital white light color image 114 and the digital fluorescence color image 112 may both be reflectance images of the object 106 in this example. However, this is not essential, and the digital fluorescence image 112 may alternatively be a fluorescence image or a reflectance image that only includes, for example, the fluorescence of the phosphor.

[0217] Images 112 and 114 show three different types of tissues 852a, 852b, and 852c. Of course, there may be more or fewer types of tissues in images 112 and 114. Various types of tissues may not be clearly visible in one or both of images 112 and 114. For example, tissue 852a may be an arterial blood vessel containing oxygenated blood, tissue 852b may be a venous blood vessel containing deoxygenated blood, or when fluorescence is recorded in a digital fluorescence image, the tumor and 852c may be the background, for example, in neurosurgery, it may be nerve tissue.

[0218] Pixel 150a1 of the digital white light color image 114 is located in the region of tissue 852a. The color space coordinates of pixel 150a1 are {R 1,1 , G 1,1 , B 1,1} when the RGB color space is assumed for illustrative purposes. Pixel 150a2 of the digital white light color image 114 is located in the region of tissue 852b. The color space coordinates of pixel 150a2 are {R 1,2 , G 1,2 , B 1,2}. Pixel 150a3 of the digital white light color image 114 is located in the region of tissue 852c. The color space coordinates of pixel 150a3 are {R 1,3 , G 1,3 , B 1,3}. The color space coordinates are recorded in the first imaging spectrum 202.

[0219] In the digital fluorescence image 112, pixel 150b1 is located in tissue 852a, preferably the pixel corresponding to pixel 150a1, pixel 150b2 is located in tissue 852b, preferably the pixel corresponding to pixel 150a2, and pixel 150b3 is located in tissue 852c, preferably the pixel corresponding to pixel 150a3. The corresponding pixels are preferably located at the corresponding locations in images 112 and 114 or at the corresponding locations of the patterns previously identified using a pattern recognition algorithm in images 112 and 114.

[0220] The color space coordinates of pixel 150b1 are, assuming an RGB color space here, {R 2,1 , G 2,1 , B 2,1}. The color space coordinates of pixel 150b2 are {R 2,2 , G 2,2 , B 2,2}. The color space coordinates of pixel 150b3 are {R 2,3 , G 2,3 , B 2,3}. These color space coordinates are recorded in the second imaging spectrum 222 and thus contain spectral information different from the color space coordinates of the corresponding pixels 150a to 150c in the digital white light color image. If the color spaces of images 112, 114 are different, they can be converted to a common color space.

[0221] As shown in FIG. 3, an input combination (set) 846 of color space coordinates may be formed for each of pixels 150a1 to 150a3 and 150b1 to 150b3. Thus, the color space coordinates of pixels 150a1 and 150b1, {R 1,1 , G 1,1 , B 1,1} and {R 2,1 , G 2,1 , B 2,1} form the input combination {R 1,1 , R 2,1 , G 1,1 , G 2,1 , B 1,1 , B 2,1}. The color space coordinates of pixels 150a2 and 150b2, {R 1,2 , G 1,2 , B 1,2} and {R 2,2 , G 2,2 , B 2,2} form the input combination {R 1,2 , R 2,2 , G 1,2 , G 2,2 , B 1,2 , B 2,2}. The color space coordinates of pixels 150a3 and 150b3, {R1,3, G1,3, B 1,3} and {R 2,3 , G 2,3 , B2,3} forms the input combination {R 1,3 , R 2,3 , G 1,3 , G 2,3 , B 1,3 , B 2,3}. It should be noted that each combination 846 substantially corresponds to the input combination {R1, R2, G1, G2, B1, B2} in FIG. 2. Each different input combination 846 corresponds to a different color.

[0222] A set 143 including at least two, that is, a plurality of color conversion functions 140 may be provided, for example, may be stored in the image processor 170. The set 143 can include, for example, different color conversion functions 140-I, 140-II, 140-III shown in FIG. 5. Instead of a set, a single color conversion function 140 can be provided. In one embodiment, the color conversion functions 140-I, 140-II, 140-III can correspond to the color conversion functions 140a, 140b, 140c.

[0223] If only a single color conversion function 140 is provided, its application can depend on the input combination 846 in FIG. 8, that is, the color space coordinates included therein. For example, if the input combination is a specific color conversion function 140 or belongs to a predetermined set 856 of target combinations 858 assigned to a specific color conversion function 140, this specific color conversion function 140 is applied to the input combination. Otherwise, the color conversion function 140 is not applied. In this case, the color conversion function 140 can be applied only when the input combination 846 represents a color that is typical for organization 852a but not typical for organization 852b or organization 852c. Because the color conversion function 140 is assigned to the color representing organization 852a or is assigned to the color representing organization 852a.

[0224] When two or more color conversion functions 140, for example, color conversion functions 140-I, 140-II, 140-III are provided, whether any of the color conversion functions 140-I, 140-II, 140-III is applied, or whether any arbitrary color conversion function is applied, may depend on the input combination 846, that is, the set of color space coordinates of the input combination. Each set 856 is assigned to a different color conversion function.

[0225] For example, when the input combination 846, or more precisely, the target combination 858 corresponding to the input combination 846, is included within a predetermined set 856 of the target combination 858, the color conversion function 140-I assigned to this specific set 856 can be applied. If the input combination 846 is included within a second predetermined set of the target combination 858 and this target combination 858 preferably does not overlap with the first set, the color conversion function 140-II assigned to this set can be applied. When the input combination 846 is included within a third predetermined set 856 of the target combination, the third color conversion function 140-III can be applied or no color conversion function is applied. Alternatively, if the input combination 846 is not included in any set 856, it may not be necessary to apply a color conversion function or a specific color conversion function can be applied. Thus, the tissue 852a can undergo a color conversion different from that of the tissue 852b and / or the tissue 852c, as described with respect to colors 500, 508, 504 in FIG. 5, and each of the colors 500, 508, 504 can be represented by a different input combination.

[0226] Each set 856 represents a group of different colors. Each set 856 can in particular represent a specific type of tissue indicating the colors included in this set. For example, one set 856 can include a range of brighter red colors to represent oxygenated blood, another set 856 can include a range of bluish-red colors to represent deoxygenated blood, another set 856 can include a range of grayish-pink colors to represent the gray matter of a living brain, and another set 856 can include a range of whitish-pink colors to represent the white matter of a living brain. Any number and combination of sets can be used. However, in order to have a one-to-one assignment between the color conversion function and the input connection 846, the target connection 858 should be included in only one set 856.

[0227] The various sets 856 and their target connections 858 can represent different types of fluorescence when the digital white light color image 114 and the digital fluorescence color image 112 record only fluorescence. For example, one set 856 can include target connections that map the fluorescence of a phosphor such as 5-ALA / pPiX to the output color. Another set 856 can include target connections 858 that represent the autofluorescence of biological tissue and map their colors represented by the input connections to the reflectance color of the biological tissue, or the naturally appearing autofluorescence color, or a pseudo color. Still another set 856 can include target connections 858 that represent the reflectance color of an object under illumination of a fluorescence emission spectrum and map these to more natural colors.

[0228] The target connection 858 can be determined empirically in a calibration process.

[0229] At least one set 856 of the target connections 858 can be stored in or included in the image processor 170, for example, as a look-up table.

[0230] Input connection {R 1,1 ,R 2,1 ,G1,1 , G 2,1 , B 1,1 , B 2,1} is assigned and the applied color conversion function 140-I results in color space coordinates {R * 1 , G * 1 , B * 1} to which the input combination is converted. The output pixel 150c1 is preferably a pixel corresponding to at least one of the pixels 150a1 and 150b1. The input combination {R 1,2 , R 2,2 , G 1,2 , G 2,2 , B 1,2 , B 2,2} is assigned and the applied color conversion function 140-II results in color space coordinates {R * 2 , G * 2 , B * 2} to which the input combination is converted. The output pixel 150c2 is preferably a pixel corresponding to at least one of the pixels 150a2 and 150b2. The input combination {R 1,3 , R 2,3 , G 1,3 , G 2,3 , B 1,3 , B 2,3} is assigned and the applied color conversion function 140-III results in color space coordinates {R * 3 , G * 3 , B * 3} to convert the input combination. The output pixel 150c1 is preferably a pixel corresponding to at least one of the pixels 150a3 and 150b3. If a color conversion function is not assigned to the input combination and as a result, color conversion is not applied to this input combination, then as described with reference to FIG. 2, the color space coordinates in the digital output color image 160 can be calculated by adding the color space coordinates within the combination that are in the same color band.

[0231] Since the input combination includes many more color bands than each of the images 112, 114, the accuracy of the color conversion that depends on the combination is consistent with the accuracy of the detection of the tissue type. In practice, by accurately calibrating different predetermined sets 856 of the target combination 858 that determines which color conversion function is used, the color conversion that depends on the tissue type can be integrated into the color conversion process.

[0232] FIG. 6 is a schematic diagram of an imaging method executable as a computer-implemented method, for example, executed on an image processor 170.

[0233] In any step 600, for example, using a white light color camera 110, a digital white light color image 114 is recorded. In any step 602, for example, using a fluorescence color camera 111, a digital fluorescence color image 112 is recorded. Since the images 112, 114 may be obtained from the memory, steps 600 and 602 are optional. As described above, the cameras 110, 111 should be synchronized with respect to the exposure time and locked to each other with respect to the gain, that is, maintain the same gain ratio. The gamma can be set to a fixed value, and it is preferable that any automatic color adjustment is turned off.

[0234] In any step 604, each of the images 112, 114 can be demosaicked.

[0235] At any step 606, one or both of the images 112 or 114 can be registered by matching them so that the same image features are geometrically identical in each of the images 112, 114 with respect to position, size, and orientation. After registration, each pixel in the digital white light color image 114 has the corresponding pixel in the digital fluorescence color image 112. Preferably, the corresponding pixels are located at the same location in each of the images 112, 114.

[0236] In one operating mode, for example operating mode B, and optionally also in operating mode C, the color conversion is performed in any color conversion step 608. Step 608 is omitted, for example, in operating mode A. In step 608, the above-described color conversion function 140 or a combination of color conversion functions 140 is applied to at least one of the digital white light color image 114 and the digital fluorescence image 112.

[0237] In the image combining step 610, the digital white light color image 114 and the digital fluorescence color image 112 are combined to generate a digital output color image 160. This can be done in all operating modes A, B, C. As described above, the images 112, 114 may be simply added, for example, as shown in FIG. 2, for the combination. Such addition can be performed by adding the color space coordinates of the two images located within the same color band. If the digital white light color image 114 has color space coordinates {R1, G1, B1} in the color space and the corresponding pixel in the digital fluorescence color image 112 has color space coordinates {R2, G2, B2}, the color space coordinates {R * , G * , B *} of the corresponding pixel in the output color image can be calculated as R * = R1 + R2, G * = G1 + G2, B * = B1 + B2.

[0238] In operation mode C, the color conversion function 140 can be applied to both the digital white light color image 114 and the digital fluorescence image 112, as shown in FIG. 3, to generate a digital output color 160. In step 610, the color conversion function can be selected from a predetermined set of color conversion functions 140 according to the input combination and applied to the input combination as described with reference to FIG. 8.

[0239] In step 612, post-processing can be performed. For example, the digital output color image 160 can be homogenized, its contrast can be enhanced, and / or a color space conversion, such as from RGB to sRGB, can be performed, and / or gamma correction can be performed. It is important to note that the color conversion in steps 608 and / or 610 is not gamma correction.

[0240] In step 612, the digital output image 160 is displayed.

[0241] Some embodiments relate to a microscope, particularly a fluorescence microscope, that includes a system as described in connection with one or more of FIGS. 1-6. Alternatively, the microscope may be part of a system as described in connection with one or more of FIGS. 1-6, or may be connected to a system as described in connection with one or more of FIGS. 1-6.

[0242] The medical fluorescence observation device 100 can be configured to select which of a set 143 of different color conversion functions 140, such as 140-I, 140-II, 140-III, is applied to at least one of the digital fluorescence color image 112 and the digital white light color image 114. More specifically, the input selector can be configured to perform this selection.

[0243] Furthermore, the medical fluorescence observation device 100 can be configured to select which digital output color image 160 is to be displayed on the displays 132, 182, such as an eyepiece, VR goggles, and / or an external monitor. Even further, the selector device can be configured to select which of the digital output color images 160 is to be displayed on the displays 132, 182. Next, this will be described with reference to FIG. 9. In FIG. 9, the image processor 170 is configured to apply one or more sets 143 of color conversion functions 140-I, 140-II, 140-III to the digital fluorescence color image 112 and / or the digital white light color image 114 as described above. Any number of different color conversion functions 140 and / or sets 143 can be used.

[0244] When each color conversion function 140-I, 140-II, 140-III is applied to the digital fluorescence color image 112 and / or the digital white light color image 114, for example, in the color conversion step 608, different digital output color images 160-I, 160-II, 160-III are generated as a result. For example, when the color conversion function 140-I is applied to the digital fluorescence color image 112 and / or the digital white light color image 114, the digital output color image 160-I is generated, and as a result of applying the color conversion function 140-II, the digital output color image 160-II is generated, and so on.

[0245] The selector device 165 is shown as being connected to the image processor 170. The display selection signal 900 can be transmitted from the selector device 165 to the image processor 170 via any type of wireless and / or wired transmission path.

[0246] The selector device 165 is part of the hardware such as a knob or a switch. However, the selector device 165 may also be a software device such as a graphical interactive element that can be operated by a user in the same way as a hardware switch and may exist at least partially in the image processor 170. The selector device may be a combination of hardware and software. In this case, the software part may exist at least partially in the image processor 170.

[0247] As described above, the selector device 165 can be used to select different operation modes of the medical fluorescence observation device 100. The display selection signal 900 can represent at least a part of these different operation modes.

[0248] In FIG. 9, each different operation mode includes the application of a different color conversion function 140 or a set of different color conversion functions 140. For example, using the selector device 165, six different operation modes such as Mode I, Mode II, Mode III, Mode I+II, Mode I+III, and Mode II+III can be selected. In Mode I, the color conversion function 140-I is applied. In Mode II, the color conversion function 140-II is applied. In Mode III, the color conversion function 140-III is applied. In Mode I+II, the color conversion functions 140-I and 140-II are applied. In Mode I+III, the color conversion functions 140-I and 140-III are applied. In Mode II+III, the color conversion functions 140-II and 140-III are applied. Of course, there may also be a Mode I+II+III in which the color conversion functions 140-I, 140-II, and 140-III are applied.

[0249] In mode I+II and other modes where two or more different color conversion functions are selected for operation on the digital white light color image 114 and / or the digital fluorescence color image 112, the color conversion functions may be stacked on top of each other and applied sequentially. For example, in mode I+II, first, the color conversion function 140-I is applied to the digital white light color image 114 and / or the digital fluorescence color image 112, and then the color conversion function 140-II is applied to the result.

[0250] In another variation shown in FIG. 10, the selector device 165 can be used to select which of the different digital output color images 160-I, 160-II, and 160-III is to be displayed on the displays 132, 182. If I is selected by the selector device, the digital output color image 160-I is displayed; if II is selected by the selector device, the digital output color image 160-II is displayed, and so on. Further, if I+II is selected, the digital output color images 160-I and 160-II are displayed simultaneously in separate portions of the displays 132, 182, for example, side by side. If II, III is selected, the digital output color images 160-II and 160-III are displayed simultaneously in separate portions of the displays 132, 182. This can continue for any number and combination of the color conversion functions 140 and the digital output color images 160.

[0251] Therefore, an output set 1000 of different digital output color images 160 can be generated from the set 143 of color conversions. The image processor 170 is configured to select at least one digital output color image from the output set 1000 for display in response to a display selection signal 900. The selector device 165 can be configured to generate a plurality of different display selection signals 900, where each selector signal can be uniquely assigned to a digital output color image 160 or a combination of digital output color images 160 of the output set 1000. The image processor is preferably configured to display the digital output color image 160 assigned to the last received display selection signal 900.

[0252] When two or more digital output color images 160 are selected to be displayed, the displayed digital output color images 160 are preferably synchronized, i.e., generated from the same digital white light color image 114 and digital fluorescence color image 112.

[0253] In one embodiment, the digital output color image 160-I can be generated only from the digital fluorescence color image 112 that can specifically represent the fluorescence emission of at least one phosphor 116, as described above. In this case, the color conversion function 140-I can be applied only to the fluorescence color image 112, and this color conversion function enhances the contrast of the fluorescence.

[0254] In another embodiment, the digital output color image 160-II can be generated from a combination of the digital white light color image 114 and the digital fluorescence color image 112, as described above. The digital white light color image 114 represents the reflectance image of the object 106 under illumination of the fluorescence excitation spectrum, and the digital fluorescence color image 112 represents the fluorescence emission of at least one phosphor. In this case, the color conversion function 140-II can be applied only to the white light color image 114. Such a color conversion function can be configured to shift the color in the digital white light color image to a more natural color.

[0255] In yet another embodiment, the digital output color image 160-III can be generated from a combination of the digital white light color image 114 and the digital fluorescence color image 112. Here, the digital white light color image 114 can represent the reflectance image of the object 106 under white light illumination, and the digital fluorescence color image 112 can represent the fluorescence emission of at least one phosphor. As described above, a separate color conversion function 140 can be applied to the digital fluorescence color image 112 and the digital white light color image 114, or the color conversion function can be applied to only one of the digital fluorescence color image 112 and the digital white light color image 114, or to a combination of the digital fluorescence color image 112 and the digital white light color image 114.

[0256] In the combined view, any two of these three images can be combined according to the user's selection.

[0257] FIG. 7 shows a schematic diagram of a system 700 configured to implement the method described herein. The system 700 includes a microscope 710 and a computer system 720. The microscope 710 is configured to image and is connected to the computer system 720. The computer system 720 is configured to implement at least a part of the method described herein. The computer system 720 may be configured to execute a machine learning algorithm. The computer system 720 and the microscope 710 may be separate entities, or may be integrated within a common housing. The computer system 720 may be part of the central processing system of the microscope 710 and / or the computer system 720 may be part of a sub-component of the microscope 710, such as a sensor, actuator, camera or illumination unit of the microscope 710.

[0258] The computer system 720 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system distributed across various locations such as local clients and / or one or more remote server farms and / or data centers, with one or more processors and one or more storage devices). The computer system 720 may include any circuit or combination of circuits. In one embodiment, the computer system 720 may include one or more processors, which can be of any type. As used herein, a processor may be, for example, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, but is not limited thereto. Other types of circuits that may be included in the computer system 720 may be custom circuits, application specific integrated circuits (ASICs), etc., for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 720 may include one or more storage devices that may include one or more storage elements suitable for specific applications, such as main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives for handling removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 720 may include a display device, one or more speakers, and a controller that may include a keyboard and / or a mouse, a trackball, a touch screen, a voice recognition device, or any other device that enables a user of the system to input information to and receive information from the computer system 720.

[0259] Some or all of the steps may be performed by (or using) a hardware device such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the critically important steps may be performed by such a device.

[0260] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is executable by a non-transitory recording medium, which is, for example, a digital recording medium such as a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, in which electronically readable control signals are stored that cooperate (or are capable of cooperating) with a programmable computer system to implement each method. Thus, the digital recording medium may be computer-readable.

[0261] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system such that any of the methods described herein are implemented.

[0262] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to implement any of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0263] Another embodiment includes a computer program stored on a machine-readable carrier for implementing any of the methods described herein.

[0264] Thus, in other words, embodiments of the present invention are computer programs having program code for implementing any of the methods described herein when the computer program is executed on a computer.

[0265] Thus, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) including a stored computer program for implementing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium to be recorded is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein including a processor and a storage medium.

[0266] Thus, another embodiment of the present invention is a data stream or signal sequence representing a computer program for implementing any of the methods described herein. The data stream or signal sequence may be configured, for example, to be transferred via a data communication connection, such as the Internet.

[0267] Another embodiment includes a processing means, for example, a computer or a programmable logic device configured or adapted to implement any of the methods described herein.

[0268] Another embodiment includes a computer having an installed computer program for implementing any of the methods described herein.

[0269] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0270] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to implement any of the methods described herein. Generally, and advantageously, the methods are implemented by any hardware device.

Description of the Reference Numerals

[0271] 100 Medical fluorescence observation device 101L Stereoscopic sub-assembly for the left channel 101R Stereoscopic sub-assembly for the right channel 102 Digital imaging system 104 Eyepiece 106 Object of investigation 107 Biological tissue 110 White light color camera 111 Fluorescence color camera 112 Digital fluorescence color image 114 Digital white light color image 116 Phosphor 118 Second phosphor 132 Integral display / internal display 134 Direct optical path 136 beam splitter 140 color conversion function 140a to 140d various color conversion functions 140-I, 140-II, 140-III various color conversion functions 142 color conversion matrix 143 set of color conversion functions 143a to 143c various sets of color conversion functions 150 pixel 150a1 to 150a3 pixels in a digital white light color image 150b1 to 150b3 pixels in a digital fluorescence color image 150c1 to 150c3 pixels in a digital output color image 160 digital output color image 165 selector device 168 filter setting selector 170 image processor 172 output interface 174 objective lens 176 color separation assembly 178 illumination assembly 179 illumination filter 180 illumination beam splitter 182 display 184 field of view 186 computer 188 white light filter 190 fluorescence filter 192 dichroic beam splitter 194 memory 196 data transmission line 200 spectrum 202 first imaging spectrum 204, 206, 208 primary colors or color bands 210 stop band 212 visible spectrum or visible light range 220 spectrum 222 second imaging spectrum 224 pass band Fluorescence emission spectrum of phosphor 226 Fluorescence emission spectrum of another phosphor 228 Portion of fluorescence emission spectrum in color band 204 Portion of fluorescence emission spectrum in color band 206 Portion of fluorescence emission spectrum in color band 208 Combination of images Spectrum Portion of fluorescence emission spectrum in the first imaging spectrum Portion of fluorescence emission spectrum in the first imaging spectrum Multi - spectral image from digital white - light color image and fluorescence color image Reflectance spectrum Converted color Recorded color Natural white point Recorded white point Recorded color Converted color Recorded color area Target color area Recording of digital white - light color image Recording of digital fluorescence color image Demosaicing Registration Color conversion step Image combination step Post - processing Display System Microscope Computer system Input combination Tissue types 852a - 852b Set of different target combinations Target combination Display selection signal Output set of digital output color images λ wavelength I intensity A first operation mode of the processor B second operation mode of the processor Coefficients C11, C12, …, C63 of the color conversion matrix Example of RGB color space R, G, B, R1, G1, B1, R2, G2, B2, R * , G * , B * (Intensity) signals in each color band or sub - band of the color space

Claims

1. An image processor (170) for a medical fluorescence observation device (100) such as a fluorescence microscope or fluorescence endoscope, wherein the image processor (170) is - A digital white light color image (114) of the object (106) recorded in the first imaging spectrum (202) is acquired. - The system is configured to acquire a digital fluorescence color image (112) of the object (106) recorded in a second imaging spectrum (222), The second imaging spectrum (222) overlaps with the fluorescence emission spectrum (226) of at least one phosphor (116) and is different from the first imaging spectrum (202), and both the first imaging spectrum (202) and the second imaging spectrum (222) overlap with the visible spectrum (212). - The image processor (170) is configured to output at least one digital output color image (160, 160-I, 160-II, 160-III) from a set (1000) of digital output color images (160), The aforementioned at least one digital output color image (160, 160-I, 160-II, 160-III) is, - A digital output color image (160, 160-I) generated by the image processor (170) solely from the digital fluorescent color image (112), - A digital output color image (160, 160-II) generated by the image processor (170) solely from the digital white light color image (114), - A digital output color image (160, 160-III) generated by the image processor (170) from the combination of the digital fluorescent color image (112) and the digital white light color image (114), Includes, The aforementioned image processor (170) -Receives a display selection signal (900) from the selector device (165), - The system is further configured to select at least one digital output color image (160, 160-I, 160-II, 160-III) from the set for output in response to the display selection signal (900). Image processor (170).

2. The aforementioned image processor (170) - Includes a set (143) of color conversion functions (140, 140-I, 140-II, 140-III) respectively, configured to map the color of an input pixel to a different color of an output pixel, -By applying at least one color conversion function (140-I) from the set of color conversion functions (143) to the digital fluorescent color image (112), the digital output color image (160, 160-I) generated solely from the digital fluorescent color image (112) is calculated. -By applying at least one color conversion function (140-II) from the set of color conversion functions (143) to the digital white light color image (114), the digital output color image (160, 160-I) generated solely from the digital white light color image (114) is calculated. - At least one color conversion function (140-I, 140-II) of the set of color conversion functions (143) is, In the aforementioned digital white light color image (114), The aforementioned digital fluorescent color image (112) and / or The combination of the aforementioned digital fluorescent color image (112) and the aforementioned digital white light color image (114) The system is configured to calculate the digital output color image (160, 160-I) generated from the combination of the digital fluorescent color image (112) and the digital white light color image (114) by applying the system. The image processor (170) according to claim 1.

3. The image processor (170) is configured to simultaneously output at least two digital output color images of the set (1000). The image processor (170) according to claim 1.

4. The aforementioned digital white light color image (114) represents a reflectance image of the object (106) under illumination by the fluorescence excitation spectrum of at least one phosphor (116), and / or The aforementioned digital fluorescent color image (112) represents the fluorescence emission of the at least one phosphor. The image processor (170) according to claim 1.

5. The aforementioned digital white light color image (114) represents a reflectance image of the object (106) under white light illumination, and / or The aforementioned digital fluorescent color image (112) represents the fluorescence emission of the at least one phosphor. The image processor (170) according to claim 1.

6. The aforementioned digital white light color image (114) and the aforementioned digital fluorescent color image (112) represent reflectance images of the object (106) under white light illumination. The image processor (170) according to claim 1.

7. The first imaging spectrum (202) and the second imaging spectrum (222) are complementary to each other. The image processor (170) according to claim 1.

8. The aforementioned color conversion function (140) is a 3x3, 6x3, or 3x6 matrix. The image processor (170) according to claim 1.

9. A medical fluorescence observation device (100) such as a fluorescence microscope or fluorescence endoscope, wherein the medical fluorescence observation device (100) is An image processor (170) according to any one of claims 1 to 8, A fluorescent color camera (111) configured to record the aforementioned digital fluorescent color image (112), A white light color camera (110) configured to record the aforementioned digital white light color image (114), A medical fluorescence observation device (100) including [a specific component].

10. The medical fluorescence observation device (100) includes a selector device (165) which is operated by a user of the medical fluorescence observation device (100) and is configured to generate a display selection signal (900) in response to the user's operation. A medical fluorescence observation device (100) according to claim 9.

11. The medical fluorescence observation apparatus (100) includes at least one display (132, 182) connected to at least one image processor (170), the display being configured to receive and display at least one digital output color image (160) output by the image processor (170). A medical fluorescence observation device (100) according to claim 9.

12. The aforementioned medical fluorescence observation device (100) is a surgical fluorescence microscope. A medical fluorescence observation device (100) according to claim 9.

13. A computer-implemented image processing method for a fluorescence observation device (100) such as a fluorescence microscope or fluorescence endoscope, wherein the computer-implemented image processing method is - A step of acquiring a digital white light color image (114) of the object (106) recorded in the first imaging spectrum (202), - A step of acquiring a digital fluorescence color image (112) of the object (106) recorded in a second imaging spectrum (222), Includes, The second imaging spectrum (222) overlaps with the fluorescence emission spectrum (226) of at least one phosphor (116) and is different from the first imaging spectrum (202), and both the first imaging spectrum (202) and the second imaging spectrum (222) overlap with the visible spectrum (212). - The computer-implemented image processing method includes the step of outputting at least one digital output color image (160, 160-I, 160-II, 160-III) from a set (1000) of digital output color images (160), The aforementioned at least one digital output color image (160, 160-I, 160-II, 160-III) is, - A digital output color image (160, 160-I) generated by the image processor (170) solely from the aforementioned digital fluorescent color image (112), - A digital output color image (160, 160-II) generated by the image processor (170) solely from the digital white light color image (114), - A digital output color image (160, 160-III) generated by the image processor (170) from the combination of the digital fluorescent color image (112) and the digital white light color image (114), Includes, The aforementioned computer-implemented image processing method is - The step of receiving a display selection signal (900) from a selector device (165), - A step of selecting at least one digital output color image (160, 160-I, 160-II, 160-III) from the set for output in accordance with the display selection signal (900), A computer-implemented image processing method further includes the following.

14. A computer program product or computer-readable medium, wherein when the program is executed by a computer, the computer includes an instruction causing the computer to perform the method described in claim 13.

15. A method for operating a medical fluorescence observation device (100), such as a fluorescence microscope or fluorescence endoscope, wherein the method is: - A step of recording a digital fluorescence color image (112) in a second imaging spectrum (222) using a fluorescence color camera (111), - A step of recording a digital white light color image (114) in a first imaging spectrum (202) using a white light color camera (110), - A step of performing the computer-implemented image processing method described in claim 13, A method that includes this.

Images