Data processing device and computer-implemented method for combining two images and superimposed colors using uniform color space

Abstract

The invention relates to a data processing device (170) and a computer-implemented method for calculating an output color of an output pixel (150c) in an output color image (160) by mixing a background color of a first pixel (150a) in a background color image (114) with a superimposed color in accordance with a light intensity value (204) of a second pixel (150b) in a second image (112). In order to achieve a color mixing suitable for human color perception, so as to retain all visually recognizable information without introducing artifacts, the following steps are carried out: background color coordinates (202) of a background color in a uniform color space (400), superimposed color coordinates (200) of a superimposed color in the uniform color space (400), and light intensity values (204) are retrieved. The background color has a predetermined maximum brightness value (440) and a predetermined minimum brightness value (442) in a uniform color space. The light intensity value is within a range extending from and including a predetermined maximum light intensity value (446) and extending to and including a predetermined maximum light intensity value (448). Further, a luminance line (402) is calculated in the uniform color space (400), the luminance line (402) extending from the superimposed color coordinates (200) to an end point (406) along points having chromaticity of the superimposed color coordinates (200). The brightness at the endpoint is lower than the brightness at the superimposed color coordinates. A first point (408) on the luminance line (402) is determined, the ratio of the difference between the predetermined maximum brightness value and the brightness of the background color coordinates (202) to the difference between the brightness of the background color coordinates (202) and the predetermined minimum brightness value corresponds to the ratio of the length (410) of the brightness line between the first point (408) and the superimposed color coordinates (200) to the length (412) of the brightness line between the end point (406) and the first point (410). Then, a superimposed color line (404) is calculated, the superimposed color line (404) extending linearly from the background color coordinate (202) to the first point (408). Further, a second point (414) on the superimposed color line (404) is determined, the ratio of the difference between the predetermined maximum light intensity value and the light intensity value to the difference between the light intensity value and the predetermined minimum light intensity value corresponds to the ratio of the length (420) of the superimposed color line (404) between the second point (414) and the first point (408) to the length (422) of the superimposed color line (404) between the background color coordinate (202) and the second point (414). A color coordinate (206) of a second point (414) in the uniform color space (400) is output as an output color of an output pixel (150c).

Description

Technical Field

[0001] The present invention relates to a data processing device and a computer-implemented method for calculating an output color of an output pixel in an output color image by mixing a background color of a first pixel in a background color image with an overlay color according to a light intensity value of a second pixel in a second image. Background Art

[0002] In some applications, it is desirable to blend two images and to highlight in the resulting image information contained in one of the two images in a predetermined superimposed color, the predetermined superimposed color being, for example, a false color or pseudo color.

[0003] One such application is in fluorescence imaging, as used in medical observation devices such as microscopes and endoscopes. Here, a background color image can be recorded to provide a reflected image of an object. For example, a reflected image can be used to show the anatomical structure of the area where surgery is taking place. The object may contain or may be artificially provided with one or more fluorophores that emit fluorescence when excited. These fluorophores can be used to highlight areas of interest. For example, some fluorophores can accumulate in tumors so that tumors can be more easily detected during surgery through their fluorescence. Other fluorophores can be used to highlight blood vessels. The fluorescence image can be recorded as a second image using only wavelengths that are contained in the fluorescence emission spectrum of at least one fluorophore. In this configuration, background information and fluorescence information are contained in two different images. In order to gather this information in a single image, the background image and the fluorescence image are combined (or "mixed"). This allows surgeons or other medical personnel to identify fluorescent areas within the background image. It is desirable that the fluorescent areas of interest are immediately recognizable. Therefore, the fluorescence is assigned to an overlay color, which can correspond to the natural color of the fluorescence, or to a pseudo color.

[0004] Blending the background color image with the second image and the overlay color in a way that all information is visually preserved and no artifacts are introduced is a challenging process. Since the intensity of fluorescence is typically much lower than that of reflected light, special care must be taken to ensure that the combination of the fluorescence image and the background image preserves as much information as possible from both images.

[0005] However, the present invention is not limited to combining images where the background image is a reflection image and the second image is a fluorescence image. There are some situations where two reflection images of the same object (but recorded using different spectra or wavelengths) need to be mixed. For example, other applications may require combining a thermal, microwave or radar image with a background image.

[0006] In all of these cases, it is desirable to retain as much information as possible from the blending of the background image and the secondary image using an additive color. Summary of the invention

[0007] To meet this requirement, the data processing device is configured to retrieve background color coordinates of a background color in a uniform color space, the background color having a predetermined maximum brightness value and a predetermined minimum brightness value in the uniform color space; retrieve superimposed color coordinates of a superimposed color in the uniform color space; retrieve light intensity values, the light intensity values ​​being within a range extending from and including the predetermined maximum light intensity value and extending to and including the predetermined maximum light intensity value; calculate a brightness line in the uniform color space, the brightness line extending from the superimposed color coordinates to an endpoint, the brightness at the endpoint being lower than the brightness at the superimposed color coordinates, the brightness line extending along a point having a chromaticity of the superimposed color coordinates; determine a first point on the brightness line, wherein the predetermined The ratio of the difference between the maximum brightness value and the brightness of the background color coordinates to the difference between the brightness of the background color coordinates and a predetermined minimum brightness value corresponds to the ratio of the length of the brightness line between the first point and the superimposed color coordinates to the length of the brightness line between the endpoint and the first point; calculating a superimposed color line extending straightly from the background color coordinates to the first point; determining a second point on the superimposed color line, wherein the ratio of the difference between the predetermined maximum light intensity value and the light intensity value to the difference between the light intensity value and a predetermined minimum light intensity value corresponds to the ratio of the length of the superimposed color line between the second point and the first point to the length of the superimposed color line between the background color coordinates and the second point; and outputting the color coordinates of the second point in a uniform color space as the output color of the output pixel.

[0008] The above needs are also solved by a computer-implemented method, which includes the following steps: retrieving background color coordinates of a background color in a uniform color space, the uniform color space having a predetermined maximum brightness value and a predetermined minimum brightness value; retrieving superimposed color coordinates of a superimposed color in the uniform color space; retrieving light intensity values, the light intensity values ​​being within a range extending from and including the minimum light intensity value and extending to and including the maximum light intensity value; calculating a brightness line in the uniform color space, the brightness line extending from the superimposed color coordinates to an endpoint, the brightness at the endpoint being lower than the brightness at the superimposed color coordinates, the brightness line extending along a point having a chromaticity of the superimposed color coordinates; determining a first point on the brightness line, wherein the predetermined maximum brightness value and the minimum brightness value are within a range extending from and including the minimum light intensity value and extending to and including the maximum light intensity value. The ratio of the difference between the maximum brightness value and the brightness of the background color coordinates to the difference between the brightness of the background color coordinates and a predetermined minimum brightness value corresponds to the ratio of the length of the brightness line between the first point and the superimposed color coordinates to the length of the brightness line between the endpoint and the first point; calculating a superimposed color line extending straightly from the background color coordinates to the first point; determining a second point on the superimposed color line, wherein the ratio of the difference between the maximum light intensity value and the light intensity value to the difference between the light intensity value and the minimum light intensity value corresponds to the ratio of the length of the superimposed color line between the second point and the first point to the length of the superimposed color line between the background color coordinates and the second point; and outputting the color coordinates of the second point in a uniform color space as an output color.

[0009] Using the above pixel-by-pixel combination of the background color image, the second image, and the superimposed color, several problems associated with the traditional blending process are solved. First, the color transition is better adapted to the physical characteristics of human visual perception; the color transition appears more natural. Second, oversaturation is prevented. Third, non-linear dependencies that may lead to unforeseen results are avoided.

[0010] A (visually) uniform color space is a color space in which geometric distances correspond to perceived color differences. A uniform color space may be a color space from the group of color spaces comprising tristimulus color spaces such as the CIE 1931 color space, and color spaces derived from CIELUV, CIELAB, HSLuv, IPT color space, and OKLab.

[0011] The predetermined maximum brightness value and the predetermined minimum brightness value may depend on the chromaticity of the background color. For a given chromaticity, the maximum brightness value and the minimum brightness value are determined by the color space or the color gamut of the display device (if the display device is considered). In this document, the terms color and color coordinates are used synonymously. Colors are represented by color space coordinates. In different color spaces, the color space coordinates for the same color are different.

[0012] The maximum light intensity value and the minimum light intensity value can be determined by a color space or a predetermined color gamut of an alternative device, in particular for neutral colors. For example, in a uniform color space or color gamut, the maximum light intensity value can correspond to white and the minimum light intensity value can correspond to black. The maximum light intensity value and the minimum light intensity value can also be determined by a camera recording the second image.

[0013] The term "output" includes sending output data and allowing access to output data, for example allowing access to a memory in which the color coordinates of the second point are stored.

[0014] The terms "data processing device", "image processor" or "processor" are used synonymously herein.

[0015] In the following, further features of the invention are described. Each of the following features is advantageous in itself and can be independently combined with any other following features.

[0016] Each of the following features can be used independently to improve a data processing device or a computer-implemented method, even if the corresponding feature is only mentioned in the context of a data processing device or only in the context of a computer-implemented method. More specifically, a data processing device can be configured to perform each method step described below, even if the data processing device is not specifically mentioned in connection with that method step.

[0017] According to one aspect, for a plurality of or all pixels in the background image, the second image, and / or the output image, the ratio of the brightness at the endpoints of the brightness line to the brightness of the superimposed color may be constant. For example, the ratio may be set at a value between 0.1 and 0.4, meaning that the brightness at the endpoints is between 10% and 40% of the brightness of the superimposed color. The ratio may be adjusted by a user and / or stored in a memory of a data processing device.

[0018] The ratio of the brightness of the superimposed color coordinate to the brightness at the endpoint of the brightness line can, in particular, be independent of the background color coordinate of the first pixel and / or the light intensity value of the second pixel. Preferably, the ratio is constant for all pixels in the background color image and the second image.

[0019] In one embodiment, the endpoints of the brightness line may be within the color gamut of a predetermined display device. This configuration may ensure that the endpoints may be properly displayed on the predetermined display device. The predetermined display device may be an external monitor, such as a virtual reality (VR) device, a light emitting diode (LED) or organic light emitting diode (OLED) screen or projector, or a monitor integrated into the eyepiece of a medical viewing device.

[0020] For example, if an endpoint calculated using a predetermined ratio of the brightness of the superimposed color coordinates falls outside the gamut, the endpoint may be automatically shifted along the brightness line to a location where the brightness line crosses the limit of the gamut.

[0021] According to another embodiment, the brightness line may be a straight line or a curved line in the uniform color space. If one dimension of the uniform color space is brightness, the brightness line may extend parallel to this dimension.

[0022] If the superimposed color coordinates are set in a non-uniform color space, they should be transformed to a uniform color space before or at the time of retrieval. The data processing device can be configured to perform such a color space transformation. The function used for the color space transformation can be standard, or experimentally determined.

[0023] The non-uniform color space can, for example, be an RGB color space such as RGB, sRGB, AdobeRGB, Adobe WhiteGamut RGB, Rec.2100, DCI-P3, Rec.2020, Rec.709, Rec.601 or ACES. Other non-uniform color spaces are, for example, HSV, HSL, YUV and YCbCr and their derivatives.

[0024] If the background color coordinates are in a non-uniform color space, preferably, before or upon retrieval, the background color coordinates are transformed into a uniform color space. The data processing apparatus may be configured to perform such a transformation.

[0025] The color coordinates of the second point on the superimposed color line can be transformed from a uniform color space to a non-uniform color space before output. This may be preferred if the display device cannot handle color coordinates in a uniform color space.

[0026] In one embodiment, the background color image and the second image can be registration images. In the registration image, the direction, form, size and position of the pattern are consistent. The registration image is consistent and therefore contains corresponding pixels. Corresponding pixels, that is, if the background color image and the second image are registered, the pixels representing the same area of ​​the pattern have the same position in the background color image and the second image. This does not necessarily mean that the two images have the same pixel resolution. A pixel in one image can correspond to multiple pixels in another image. For example, if one of the images has a higher resolution, a first pixel can correspond to multiple second pixels, or a second pixel can correspond to multiple first pixels. However, preferably, the background color image and the second image have the same format and the same number of pixels. For example, both images can be 1k, 2k, 4k or 8k images.

[0027] The registration may be performed by a computer implemented method and / or a data processing device.

[0028] It is further preferred that the position of the output pixel corresponds to the position of at least one of the first pixel and the second pixel.The output color image may have the same format and number of pixels as any one of the background color image and the second image.

[0029] Retrieving the background color coordinates may include retrieving the first pixel including the background color coordinates. Retrieving the first pixel may further include retrieving the background color image. For example, the background color image is loaded into the memory of the data processing device in its entirety or on a pixel-by-pixel basis.

[0030] Retrieving the light intensity value may include retrieving a second pixel, wherein the second pixel includes the light intensity value. Further, retrieving the second pixel may include retrieving a second image. For example, the second image is loaded into a memory of the data processing device in its entirety or on a pixel-by-pixel basis.

[0031] In one embodiment, the second image can be one of a grayscale image and a color image. If the second image is a grayscale image, the grayscale value may already correspond to the light intensity value. This value can then be used to determine the position of the second point on the superimposed color line. The second image can be recorded by a grayscale camera or a color camera.

[0032] If the second image is a color image, retrieving the light intensity value may include calculating the light intensity value from the color coordinates of the second pixel. This calculation may be part of retrieving the light intensity value. The calculation of the light intensity value may be performed in any color space, i.e. in a non-uniform color space or a uniform color space, using a standard function or using an experimentally determined function.

[0033] The calculation of the brightness line and the superimposed color line as described above is preferably repeated for multiple or all pixels in the background color image and the second image. The superimposed color coordinates are preferably independent of the first and second pixels. For example, the superimposed color coordinates can be a constant value that can be changed by the user.

[0034] As initially explained, one area of ​​application of the data processing device and / or the computer-implemented method may be fluorescence imaging, in which a background color image may be recorded in a first imaging spectrum and may include a plurality of first pixels. For example, the background color image may be a digital white light color image. The second image may be recorded in a second imaging spectrum and may also include a plurality of second pixels. The second imaging spectrum may overlap with the fluorescence emission spectrum of at least one fluorophore. Thus, the fluorescence of at least one fluorophore may be recorded in the second imaging spectrum. For example, the second image may be a digital fluorescence image. Further, the second imaging spectrum may be different from the first imaging spectrum. For example, the first imaging spectrum and the second imaging spectrum may be complementary. Both the first and second imaging spectra may include a plurality of stopbands and passbands. The stopbands in the first imaging spectrum may correspond to the passbands in the second imaging spectrum, and the passbands in the first imaging spectrum may correspond to the stopbands in the second imaging spectrum. In one embodiment, the background color image does not record wavelengths within the fluorescence emission spectrum of at least one fluorophore.

[0035] The first and second imaging spectra may overlap with the visible spectrum.

[0036] The above configuration may be particularly useful in a medical viewing device such as a microscope or an endoscope.The microscope may be a laboratory microscope or a surgical microscope.

[0037] A medical observation device, such as a microscope or an endoscope, may include the data processing device of any of the above embodiments. The medical observation device may also include a white light color camera configured to record a background image and a fluorescence camera configured to record a second image. The fluorescence camera may be a grayscale camera or a color camera. The term "color camera" includes CCD, CMOS, multispectral and hyperspectral cameras, and all other types of cameras that record color images.

[0038] The operating method of the medical observation device may include the computer-implemented method in any of the above embodiments, and may also include the steps of: recording a background color image of the object using a white light color camera; and recording a second image of the object using a fluorescent camera.

[0039] The claimed subject matter also relates to a computer-readable medium and a computer program comprising instructions that cause a computer to perform the computer-implemented image processing in any of the above-described embodiments.

[0040] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0041] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding device.

[0042] Below, the present invention will be described exemplarily with reference to embodiments and with reference to the accompanying drawings. The combination of features shown in the embodiments should not be considered restrictive. For example, features of the embodiments that have technical effects such as those explained above but are not required in a specific application can be omitted. On the contrary, if a technical effect associated with this specific feature is required in a specific application, the above-mentioned features that are not part of the embodiments described below can be added. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Throughout the description and drawings, the same reference numerals are used to indicate elements that correspond to each other in function and / or structure.

[0044] In the accompanying drawings,

[0045] Figure 1 A schematic representation of an embodiment of a medical fluorescence observation device is shown;

[0046] Figure 2 A schematic representation of an example of a first image, a second image, and a superimposed color combined to generate an output image is shown;

[0047] Figure 3 A schematic representation of an example of an imaged spectrum in a first and a second image is shown;

[0048] Figure 4 shows a schematic representation of mixing the colors of a first image with an overlay color based on the light intensity in a second image;

[0049] Figure 5 Shown is a schematic representation of a system in which the present invention may be used. DETAILED DESCRIPTION

[0050] First, refer to Figure 1 Exemplary embodiments of the present invention are described.

[0051] Figure 1 A medical viewing device 100 is schematically shown. The medical viewing device 100 may be a fluorescence microscope or a fluorescence endoscope, the difference between a microscope and an endoscope being that in an endoscope (not shown) the object 106 is viewed via an optical fiber which is brought close to the object 106 to be studied, for example by being inserted into the body, whereas in a microscope the objective lens 174 is directed towards the object 106. Although Figure 1The medical observation device 100 is shown as a microscope, but the following description also applies to an endoscope. The medical observation device 100 can be, for example, a medical observation device used in surgery. The medical observation device 100 can also be a medical observation device used in a laboratory, such as a laboratory microscope. The object 106 to be studied can consist of or include biological tissue 107, but can also consist of or include inorganic substances.

[0052] The object 106 may include one or more fluorophores 116, 118. At least one fluorophore may be a fluorophore naturally contained in the object. For example, bones and blood contain fluorophores. At least one fluorophore may also be added to the object 106, for example by injecting it into the biological tissue 107. Examples of fluorophores that may be added to the object 106 are ICG, fluorescein, and / or 5-ALA. 5-ALA is synthesized in cells as pPIX.

[0053] The medical viewing device 100 may be configured to record the fluorescence of one or more fluorophores 116, 118. This means that the medical viewing device 100 may be configured to view, record and preferably also excite the fluorescence of one or more fluorophores 116, 118.

[0054] The medical observation device 100 may be a stereoscopic device, such as Figure 1 As shown in the example. Therefore, it can include two identical subassemblies 101L and 101R, for each of the two stereo channels. Since the two subassemblies 101L, 101R are identical in function and structure, the following description focuses on the right subassembly 101R, but is completely applicable to the left stereo channel 101L.

[0055] The medical observation device 100 may alternatively be a monoscopic device. In this case, only one of the two subassemblies 101L, 101R may be present. Therefore, for the monoscopic medical observation device 100, the following description also applies.

[0056] In one embodiment, the medical viewing device 100 may be used to generate one or more digital white light color images 114. The digital white light color image may specifically represent a reflection image of the object 106 in the visible light range. The visible light range or visible spectrum includes wavelengths from about 310 nanometers to about 1100 nanometers, or from about 380 nanometers to 750 nanometers, or from about 450 nanometers to about 700 nanometers. Fluorescence spectra or, if more than one fluorophore is used, the fluorescence spectra of the fluorophores 116, 118 are preferably omitted from the spectra recorded in the digital white light color image 114. This ensures that if fluorescence is present, only reflected light is included in the digital white light color image 114. For example, if 5-ALA / pPIX is used as a fluorophore, the fluorescence spectrum from about 625 nanometers to about 650 nanometers may not be recorded in the digital white light color image 114.

[0057] If specific wavelengths of light are used to excite fluorescence, spectra including or consisting of these wavelengths may also not be recorded or represented in the digital white light color image 114. For example, if 5-ALA / pPIX is used as the fluorophore, fluorescence may be excited by illuminating the object 106 with a wavelength between about 380 nanometers and about 450 nanometers. For fluorescein, ICG, and other fluorophores, different but known ranges for excitation and emission spectra apply compared to 5-ALA / pPIX.

[0058] When the fluorescence excitation spectrum and the fluorescence emission spectrum are not recorded, the digital white light color image 114 represents the reflectance of the object 106, that is, the white light image of the object 106, as seen by a human observer. In this configuration, the digital white light color image 114 can be viewed as a natural or true color image.

[0059] The digital imaging system 102 may also be used to generate one or more digital fluorescence images 112 of the object 106. The digital fluorescence image 112 may represent the fluorescence emission of one or more fluorophores 116, 118. The digital fluorescence image 112 may be a grayscale or color image.

[0060] The digital fluorescence image 112 preferably does not record any wavelengths outside of the emission spectrum or spectra of the one or more fluorophores.The digital fluorescence image 112 may also be a reflectance image of the object 106, or contain both fluorescent and reflected light from the object.

[0061] In one embodiment, the digital white light color image 114 and the digital fluorescent image 112 may both be color images. They are recorded using at least three color bands or, equivalently, the primary colors of a color space. For example, both the digital white light color image 114 and the digital fluorescent image 112 may be recorded in an RGB color space using three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescent color image 112 may be recorded in different color spaces, each of which may represent a multi-spectral or hyperspectral color image. The digital white light color image 114 and the digital fluorescent image 112 do not need to be recorded in the same color space, but this is preferred.

[0062] At least one digital white light color image 114 and at least one digital fluorescent image 112 contain pixels 150a, 150b, respectively. In a color space, such as an RGB color space, each color of the pixels 150a, 150b is represented by a color space coordinate, and the color space coordinate forms a triple of three integer numbers. Each integer number indicates the intensity of one of the primary colors R, G, B. For example, the strongest red can be indicated by the triple {255, 0, 0}. The strongest green can be indicated by {0, 255, 0}, for example, and the strongest blue can be indicated by {0, 0, 255}, for example. Therefore, the RGB color space is a three-dimensional space, while the CMYK color space will be a four-dimensional space. A color can be regarded as a point in the color space with a color space coordinate of, for example, {0, 0, 255}. In a more general way, a multispectral or hyperspectral color space with n color bands will correspondingly produce an n-dimensional color space, and each color will be represented by an n-tuple of color space coordinates.

[0063] Examples of the RGB color space are RGB, sRGB, AdobeRGB, Adobe White Gamut RGB, REC.2100, DCI-P3, Rec.2020, Rec.709, Rec.601, and ACES.

[0064] The spectrum recorded and represented in the digital white light color image 114, i.e. the first imaging spectrum, and the spectrum recorded in the digital fluorescent image 112, i.e. the second imaging spectrum, are preferably complementary to each other, i.e. do not overlap except for unavoidable filter leakage. They preferably represent together the complete visible light spectrum, or at least a major part of the visible light range. The first and / or second image may also contain wavelengths outside the visible spectrum, such as near infrared (NIR).

[0065] More specifically, the medical observation device 100 may include a digital imaging system 102 for generating a digital fluorescence image 112 and a digital white light color image 114. The digital imaging system 102 may include a white light color camera 110 and a fluorescence camera 111.

[0066] The white light camera 110 is configured to record digital white light color images 114. In particular, the white light color camera 110 can be configured to generate a stream of digital white light color images 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images across the entire visible spectrum in the wavelengths indicated above. The white light color camera 110 can be a CCD, CMOS, or multispectral or hyperspectral camera.

[0067] The fluorescence camera 111 is configured to record a digital fluorescence image 112. In particular, the fluorescence camera 111 may be configured to generate a stream of digital fluorescence images 112 in the form of a digital video stream. The fluorescence camera 111 may be configured to record the digital fluorescence image 112 only in one or more fluorescence spectra of at least one fluorophore 116, 118. The fluorescence camera 111 may be configured to record the digital fluorescence image 112 only in one or more narrow bands of light. These narrow bands should overlap with one or more fluorescence spectra of the one or more fluorophores 116, 118 whose fluorescence is to be recorded. Preferably, the fluorescence spectra of the fluorophore 116 and the second fluorophore 118 do not overlap, so that the fluorescence camera 111 may record light in two separate fluorescence bands that are spaced apart from each other.

[0068] The fluorescence camera 111 may be a CCD, CMOS or multispectral or hyperspectral camera, or any other type of suitable camera. Preferably, the white light color camera 110 and the fluorescence camera 111 are of the same type, but this is not necessary. Since the intensity of fluorescence is usually very low, the fluorescence camera 111 may have a longer integration time.

[0069] The respective fields of view 184 of the cameras 110, 111 are preferably aligned or even coincident and coaxial. Thus, preferably, the cameras 110, 111 provide exactly the same field of view 184 at exactly the same viewing angle and focal length. This results in exactly the same representation of the object 106 in the images 112, 114 generated by the different cameras 110, 111. The two cameras 110, 111 can use the same objective lens 174. In this configuration, the images 112, 114 can be optically registered, which is registered by not requiring or making image processing easier.

[0070] If the match of the perspective and field of view cannot be generated optically, it can be generated by image processing by applying a matching or registration routine to the digital images 112, 114, as further explained below. If the cameras 110, 111 have exactly the same perspective and field of view, the application of a computer-implemented registration routine may even be desirable.

[0071] Preferably, the two cameras 110 , 111 operate synchronously. Specifically, the exposure time may be synchronized. Therefore, the medical observation device 100 may be configured to generate a digital white light color image 114 and a digital fluorescent image 112 simultaneously.

[0072] Preferably, the gains of the two cameras 110, 111 are synchronized, i.e. adjusted in both cameras 110, 111 at the same time. Furthermore, the ratio of the gain applied in camera 110 to the gain applied in camera 111 may be constant, even if the gain varies. Gamma correction and color adjustment or white balance may be turned off or kept constant.

[0073] Any of the above-described measures may facilitate comparison, joint processing, and / or combining of the two images 112 , 114 .

[0074] In order to separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescence image 112, that is, to separate the reflection spectrum from the fluorescence spectrum, the medical observation device 100 may include an optical color separation component 176. The color separation component 176 may include an optical element such as a beam splitter 192, which may be a dichroic. The color separation component 176 may also or alternatively include an optical white light filter 188 and / or an optical fluorescence filter 190.

[0075] The fluorescence filter 190 is preferably configured to transmit light within the one or more fluorescence spectra of the one or more fluorophores 116 , 118 , and to block light outside of the one or more fluorescence spectra.

[0076] The fluorescence filter 190 can be configured as a bandpass filter including one or more passbands. Each passband should overlap with the fluorescence emission spectrum of the corresponding fluorophore 116, 118 whose fluorescence is to be recorded. Since the fluorescence filter 190 is in the optical path between the beam splitter 192 and the fluorescence camera 111, only wavelengths in the passband of the fluorescence filter 190 are transmitted to the fluorescence camera 111.

[0077] The white light filter 188 is preferably configured to block light in the fluorescence spectrum of one or more fluorophores 116, 118. The white light filter 188 may also be configured to block light in the fluorescence excitation spectrum.

[0078] The white light filter 188 is preferably configured as a band-stop filter, the stop band of which corresponds to or at least contains the pass band of the fluorescence filter 190. The white light filter 188 is located in the light path between the beam splitter 192 and the white light camera 110. Therefore, the white light color camera 110 only records wavelengths outside the stop band of the white light filter 188, and therefore also outside the pass band of the fluorescence filter 190.

[0079] Either the white light filter 188 or the fluorescence filter 190 may be a tunable filter.

[0080] If the beam splitter 192 is a dichroic beam splitter, at least one of the filters 188 and 190 may be omitted, since in this case the optical spectral filtering is already integrated in the dichroic beam splitter. The above description of passband and stopband should apply mutatis mutandis to the dichroic beam splitter 192.

[0081] Thus, white light color camera 110 records digital white light color image 114 in a first imaging spectrum, a reflectance spectrum, which is different from a second imaging spectrum, which is recorded as a fluorescence spectrum by fluorescence camera 111. The wavelengths contained in the first and second imaging spectra are determined by the filter settings of color separation component 176.

[0082] The medical viewing device 100 may also include an illumination assembly 178 configured to illuminate the object 106, preferably through the same objective lens 174 through which the imaging system 102 records the at least one digital image 112, 114. The illumination assembly 178 may be configured to selectively generate white light (i.e., light uniformly distributed across the entire visible spectrum), and fluorescence excitation light, which contains only wavelengths of light that stimulate fluorescence of at least one fluorophore 116, 118. The illumination light generated by the illumination assembly 178 may be fed into the objective lens 174 using an illumination beam splitter 180.

[0083] The illumination filter 179 may be provided according to the fluorophore and its fluorescence-specific excitation spectrum. For example, if 5-ALA / pPIX is used as the fluorophore, the illumination filter 179 may have a transmittance of 90% to 98% up to a wavelength of 425 nanometers, a transmittance between 0.5% and 0.7% between a wavelength of 450 nanometers and 460 nanometers, a transmittance of no more than 0.1% between a wavelength of 460 nanometers and 535 nanometers, and a transmittance of almost zero at wavelengths greater than 535 nanometers.

[0084] Instead of or in addition to the illumination filter 179, the illumination assembly 178 may also include an adjustable light source including, for example, light emitting diodes (LEDs) or organic light emitting diodes (OLEDs) of many different colors.

[0085] The medical observation device 100 may also include a data processing device 170. The data processing device 170 may be a hardware module, such as a microprocessor, or may be a software module. The data processing device 170 may also be a combination of a hardware module and a software module, for example, by using a software module configured to run on a specific processor (such as a vector processor, a floating point graphics processor, a parallel processor and / or a multiprocessor). The data processing device 170 may be part of a general purpose computer 186 (e.g., a PC).

[0086] The data processing device 170 is configured to retrieve the digital white light color image 114 and the digital fluorescence image 112. For example, the data processing device 170 may be configured to retrieve the digital white light color image 114 and the digital fluorescence image 112 from the memory 194 and / or directly from the cameras 110, 111. The memory 194 may be part of the data processing device 170, or reside elsewhere in the medical viewing device 100.

[0087] The data processing device 170 is also configured to calculate a digital output color image 160 from the digital white light color image 114 and the digital fluorescence image 112. The digital output color image 160 is a color image represented in a color space, and the color space of the digital output color image 160 may be different from the color space of any one of the digital white light color image 114 and the digital fluorescence image 112. Preferably, however, the color space of the digital output color image 160 is the same as the color space of the digital white light color image 114 and the digital fluorescence image 112. In order to convert coordinates from one color space to another color space, a standardized function exists or can be determined experimentally or analytically. In the case of fluorescence, the fluorescence intensity, i.e., brightness, can be used for brightness. If the fluorescence image is a grayscale image, the intensity or brightness value is recorded directly in the fluorescence image. If the fluorescence image is a color image, the intensity or brightness can be calculated using a function suitable for this color space.

[0088] Data processing device 170 is preferably configured to generate a digital output color image 160 from digital fluorescence image 112 and digital white light color image 114 .

[0089] In one embodiment, the digital white light color image 114 is used as a background color image, and the fluorescent image 112 is superimposed on the background color image as a second image. In another embodiment, the digital fluorescent color image 112 can be used as a background image, and the second pixel 150b corresponds to a pixel from the digital white light color image 114.

[0090] The data processing device 170 may be configured to combine the second image 112 and the background color image 114 using the color conversion function 140 to generate a digital output color image 160. The generation of the output color image 160 may be performed on a pixel-by-pixel basis, wherein a first pixel 150a of the background color image 114 is combined with a second pixel 150b of the second image 112 using the color conversion function 140 to form an output pixel 150c.

[0091] The first and second pixels 150a and 150b are preferably corresponding pixels. Thus, in the registered images 112, 114, the first and second pixels 150a, 150b may be located at the same position in the respective images 112, 114.

[0092] The number of pixels in the images 112, 114 does not have to be the same. For example, one pixel with a higher resolution in one image 112, 114 may correspond to a plurality of pixels with a lower resolution in the other image 114, 122. Alternatively, in the two images 112, 114, two pixels located at the same position within the same pattern identified by the pattern recognition algorithm may also be considered as corresponding pixels.

[0093] In the context of a surgical procedure, the background color image 114 may represent a background anatomical image on which fluorescence information is superimposed. If the fluorescence recorded in the second image 112 is to be represented in a predetermined superimposed color 152, such as a pseudo color, the color of the output pixel 150c may be calculated using the color of the first pixel 150b, the brightness of the second pixel 150b, and the predetermined superimposed color.

[0094] The data processing device 170 can be configured to output the digital output color image 160 to any kind of peripheral device. In this context, output includes sending and / or allowing access to the digital output color image 160, such as allowing access to a memory, such as the memory 194, where the digital output color image 160 can be stored, or outputting on an output interface 172 of the medical viewing device 100.

[0095] The digital output color image 160 may be displayed on a display 132 that is integral to the medical viewing device 100. For example, the display 132 may be integrated in an eyepiece or eyepiece 104 of the medical viewing device 100. The display 132 may also display a graphical user interface for operating the medical viewing device 100.

[0096] The overlay colors 152 may be stored in the data processing device 170 and / or the computer 186 and may be changed based on user preferences.

[0097] The medical viewing device 100 may include a direct optical path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In this case, the display may be a semi-transparent display 132 located in the direct optical path 134, or the display may be projected into the direct optical path 134. A beam splitter 136 may be provided to split the light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed to the eyepiece 104.

[0098] In another embodiment, the medical viewing device 100 does not necessarily have a direct optical path 134, but may display images using only the integral display 132. As another alternative, the medical viewing device 100 need not have any integrated display at all.

[0099] The medical viewing device 100 may include an output interface 172 to which one or more (external) displays 182 and / or any type of unidirectional or bidirectional wired or wireless data connection may be connected. To this end, the output interface 172 may include standardized connectors and data transmission protocols, such as WLAN, TCP / IP, Ethernet, USB, HDMI, DVI, DisplayPort, Bluetooth and / or others. The external display may be a monitor, 3D goggles, eyepieces and the like. Any combination of external displays may be connected to the output interface 172. Any display 182 may display a graphical user interface for operating the medical viewing device 100.

[0100] The computer 186 or the data processing device 170 may be connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission line may be wired or wireless, or may be partially wired and partially wireless. The computer 186 and / or the data processing device 170 may not be integrally integrated in the medical observation device 100, but may be physically remote from the digital imaging system 102. To this end, the digital imaging system 102 and the computer 186 and / or the data processing device 170 may be connected to a network, such as a local area network (LAN), a wireless local area network (WLAN), or a wide area network (WAN), to which at least one display 182 may also be connected. The network connection may be provided by an output interface 172.

[0101] According to a modification, the medical observation device 100 can be stereoscopic but only include two cameras, one for each stereo channel. In one stereo channel, a fluorescence camera 111 is used and configured to selectively also record white light reflectance, while in the other stereo channel, a white light color camera 110 is used. If fluorescence is not used, this arrangement can provide a stereoscopic white light color image; if fluorescence is used, this arrangement can provide a monoscopic white light color image and a monoscopic fluorescence image. The description above and below also applies to this configuration.

[0102] According to another configuration, each stereo channel may include more than two cameras.

[0103] exist Figure 3 , a quantitative example of a first imaging spectrum 302 is shown as it is recorded by a digital white light camera 110 and / or represented in a digital white light color image 114, respectively. The intensity I across wavelength / color λ is shown as normalized. The first imaging spectrum 302 preferably extends across at least the visible spectrum 312.

[0104] By way of example only, the color space in which the first imaging spectrum 302 is recorded is an RGB color space having three primary colors or color bands 304, 306, and 308. One primary color 304 is blue, another primary color 306 is green, and the third primary color 308 is red. The sensitivity of the sensor of the white light color camera 110 in the different primary colors 304, 306, 308 can be tuned to result in a sensitivity that is as constant as possible across the visible spectrum 312.

[0105] If a color space other than RGB is used, the number, position and / or width of the color bands may be different.

[0106] The first imaging spectrum 302 preferably does not include the fluorescence excitation light and the fluorescence emission spectrum of at least one fluorophore 116, 118. Therefore, the first imaging spectrum 302 may include at least one stop band 310, which corresponds to the fluorescence emission of at least one fluorophore whose fluorescence is to be recorded by the fluorescence camera 111. The stop band 310 is created, for example, by the white light filter 188. The number, width and / or position of the stop bands 310 depends on the number and type of fluorophores to be observed in the object 106.

[0107] The second imaging spectrum 322 is shown as it is recorded by the fluorescence camera 111 and / or represented in the digital fluorescence color image 112, respectively. Merely by way of example, the color space in which the second imaging spectrum 322 is recorded is also an RGB color space. The sensitivity of the sensor of the fluorescence camera 111 in the different primary colors 304, 306, 308 is tuned to result in a sensitivity across the visible spectrum 312 that is as constant as possible.

[0108] The spectra 302, 322 need not be recorded in the same color space, but this is preferred.

[0109] The second imaging spectrum 322 may include one or more passbands 324. The number, position and / or width of the passbands depends on the number and type of fluorophores used. At least one passband 324 preferably corresponds to at least one stopband 310. At least one passband is generated, for example, by a fluorescence filter 190.

[0110] The first imaging spectrum 302 and the second imaging spectrum 322 are complementary to each other. They preferably complement each other to cover the entire, more than, or most of the visible spectrum 312.

[0111] Each passband 324 of the second imaging spectrum 322 preferably overlaps with a fluorescence emission spectrum 326 of a fluorophore 116, 118 whose fluorescence is to be recorded and with one or more primary colors 304, 306, 308 of the color spectrum. For example, the fluorescence emission spectrum 326 of the fluorophore 116 may overlap with all three primary colors 304, 306, 308.

[0112] Next, refer to Figure 2 , explains the process of combining images 112, 114 using superimposed color 152. Data processing device 170 is configured to perform any steps of this process.

[0113] In order to obtain color mixing from the background color of the first pixel 150a and the superimposed color 152 that most closely corresponds to human color perception, the color mixing is performed in a uniform color space, such as the CIELAB color space, CIELUV color space, HSLuv color space, IPT color space, OKLab color space, and tristimulus color space.

[0114] If the digital white light color image 114, which in this example forms the background color image, is recorded in a non-uniform color space, such as an RGB color space, it is converted into a uniform color space at step 210. Therefore, at step 210, a color space conversion into a uniform color space is performed for the first pixel 150a. The background color coordinates 202 of the first pixel 150a in the uniform color space are {X BC , Y BC , Z BC}, where X, Y, Z are, for example, the coordinates L, a, b in the CIELAB color space and L, U, V in the CIELUV color space.

[0115] If it is a color image that has been recorded in a non-uniform color space (e.g., RGB color space), the same operation is performed on the fluorescence image 112 as the second image: a color space conversion is performed on the second pixel 150b in step 210. The color space coordinates of the first and second pixels 150a, 150b should be converted to the same uniform color space. If the fluorescence image is a grayscale image, a color space conversion is not necessary.

[0116] If the fluorescent image is also a color image, the brightness value of the second pixel 150b is calculated. It should be noted that the brightness calculation 212 can also be performed before the color space conversion 210. Therefore, the light intensity value I2 of the second pixel 150b is obtained.

[0117] The superimposed color coordinates 200 of the superimposed color 152 are {X OC , Y OC , Z OC Of course, if the superimposed color coordinates are not in the uniform color space of the first and second pixels 150a, 150b, they can also be transformed into the uniform color space.

[0118] A color conversion using the superimposed color coordinates 200, the light intensity value 204, and the background color coordinates 202 is performed at step 250 to produce output color coordinates 206, i.e., the tuple {X O , Y O , Z O Output pixel 150c is preferably located at the same position in output color image 160 as first and second pixels 150a and 150b are located in their respective images 112 and 114.

[0119] The output color coordinates 206 may then be converted to a color space more suitable for the output device (eg, display 132, 182), for example, again to the RG color space. Thus, the color space transformation 210 is performed again.

[0120] refer to Figure 4 , the color conversion 250 performed by the color conversion function 140 is explained in more detail. Figure 4 Only a uniform color space 400 is shown, in particular, by way of example only, a CIELAB color space.

[0121] First, a brightness line 402 is calculated. Brightness line 402 starts at superposition color space coordinate 200 and extends continuously along points where only brightness changes and chromaticity is constant. In a uniform color space (e.g., CIELAB) where brightness is the coordinate, brightness line 402 extends along the brightness coordinate. In another uniform color space, brightness line 402 may not be a straight line.

[0122] The endpoint 406 of the brightness line 402 is determined by, for example, a predetermined value of brightness related to the brightness of the superimposed color space coordinate 200. For example, the endpoint 406 may be located at a position at which the brightness is 20% of the brightness of the superimposed color space coordinate 200 at the same chromaticity. Of course, any other predetermined ratio may also be used.

[0123] If endpoint 406 is outside of a color gamut 452 of a predetermined display device (eg, one of displays 132 , 182 ) using a predetermined luminance ratio, then endpoint 406 is placed at the intersection of luminance line 402 and the limits of color gamut 452 .

[0124] Next, a first point 408 is calculated on the brightness line 402. The location of the first point 408 along the brightness line 402 corresponds to the location of the brightness 444 of the background color 202 within the range of possible brightness values ​​in the uniform color space 400 or within the gamut 452 of the chromaticity of the background color. This range extends between and includes a predetermined maximum brightness value 440 and a predetermined minimum brightness value 442. The maximum and / or minimum brightness values ​​440, 444 may depend on the chromaticity of the background color.

[0125] More specifically, the ratio of the distance 410 from the first point 408 to the superimposed color space coordinate 200 to the distance 412 from the first point 408 to the end point 406 is proportional to the maximum brightness value 440 and the background color coordinate {X BC , Y BC , Z BC} is the same as the ratio of the difference between the minimum brightness value 442 and the brightness value of the background color coordinates.

[0126] For example, if the maximum brightness value 440 of the chromaticity for the background color is 251 in the color gamut 452, and the minimum brightness value 442 of the chromaticity for the background color is 3 in the color gamut 452, and the brightness of the background color 202 is 134, then the ratio of the length 410 to the length 412 is (251-134) / (134-3). The lighter the background color, the closer the first point 408 is to the superimposed color coordinate 200.

[0127] When the first point 408 is determined, the superimposed color line 404 is calculated. The superimposed color line 404 extends linearly from the background color coordinates 202 to the first point 408 in the arbitrary uniform color space 400. The output color coordinates 206 are determined as the coordinates of a second point 414 located on the superimposed color line 404.

[0128] The position of the second point 414 depends on the light intensity value 204 of the second pixel 150b. The distance between the second point 414 and the background color coordinate 202 corresponds to the relative intensity of the light intensity value 204 associated with the maximum range of intensities. The higher the light intensity value 204, the closer the second point 414 is to the brightness line 404. The range of light intensity values ​​204 is from a maximum light intensity value 446 to a minimum light intensity value 448, wherein the maximum and minimum values ​​are contained within the range.

[0129] In particular, the ratio of the difference between the maximum light intensity value 446 and the light intensity value 204 to the difference between the light intensity value 204 and the minimum light intensity value 448 corresponds to the ratio of the length 420 of the superimposed color line 404 between the second point 414 and the first point 408 to the length 422 of the superimposed color line 404 between the background color coordinate 202 and the second point 414.

[0130] For example, if the maximum light intensity value 446 of the first pixel 150a is 1024, the minimum light intensity value 448 of the first pixel is 0, and the light intensity value of the second pixel 150b is 347, the ratio of the length 420 to the length 422 is (1024-347) / (347-0).

[0131] The maximum light intensity value and the minimum light intensity value may be determined by a predetermined color space or color gamut of the display device, or alternatively by a range of a camera recording the second image. The maximum and / or minimum intensity values ​​446, 448 may represent neutral colors, e.g., in a uniform color space and / or color gamut, the minimum light intensity may correspond to black and / or the maximum light intensity may correspond to white.

[0132] The minimum and maximum brightness values ​​and / or the maximum and minimum intensity values ​​may be stored in the memory 194 .

[0133] Although the medical observation device 100 is described in the previous description, the hybrid method and data processing device 170 are not limited to these devices. For example, the second image may represent reflection, infrared light, microwaves, or other electromagnetic waves outside the visible spectrum instead of representing fluorescence. Further, other image sources such as X-ray or MR images may be used as background or second images.

[0134] Figure 5 A schematic diagram of a system 500 configured to perform the methods described herein is shown. The system 500 includes a microscope 510 and a computer system 520. The microscope 510 is configured to capture images and is connected to the computer system 520. The computer system 520 is configured to perform at least part of the methods described herein. The computer system 520 can be configured to execute a machine learning algorithm. The computer system 520 and the microscope 510 can be independent entities, but can also be integrated in a common housing. The computer system 520 can be part of a central processing system of the microscope 510, and / or the computer system 520 can be part of a subcomponent of the microscope 510, such as a sensor, an actor, a camera or a lighting unit of the microscope 510, etc.

[0135] The computer system 520 may be a local computer device (e.g., a personal computer, a laptop computer, a tablet computer, or a mobile phone) having one or more processors and one or more storage devices, or a distributed computer system (e.g., a cloud computing system, having one or more processors and one or more storage devices distributed in different locations, such as a local client and / or one or more remote server farms and / or data centers). The computer system 520 may include any type of circuit or circuit combination. In one embodiment, the computer system 520 may include one or more processors of any type. As used herein, a processor may mean any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA) (e.g., a microscope or a microscope component (e.g., a camera)), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 520 may be custom circuits, application specific integrated circuits (ASICs), etc., such as, for example, one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 520 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 520.

[0136] Some or all of the method steps may be performed by (or using) a hardware device (eg, a processor, a microprocessor, a programmable computer or an electronic circuit). In some embodiments, one or some of the most important method steps may be performed by such a device.

[0137] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. Implementation may be performed using a non-transitory storage medium (such as a digital storage medium, e.g., a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM and EPROM, EEPROM or flash memory) having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating with) a programmable computer system so that the corresponding method is performed. Thus, the digital storage medium may be computer readable.

[0138] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0139] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

[0140] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0141] In other words, therefore, an embodiment of the inventive method is a computer program with a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0142] Therefore, another embodiment of the present invention is a storage medium (or a data carrier, or a computer-readable medium) comprising a computer program stored thereon, the computer program being for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, comprising a processor and a storage medium.

[0143] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection, for example over the Internet.

[0144] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0145] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0146] Another embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0147] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may collaborate with a microprocessor to perform one of the methods described herein. In general, these methods are preferably performed by any hardware device.

[0148] Reference numerals

[0149] 100 Medical observation equipment

[0150] 101L Stereo subassembly for left channel

[0151] 101R Stereo subassembly for right channel

[0152] 102 Digital Imaging System

[0153] 104 Eyepiece

[0154] 106 Objects to be studied

[0155] 107 Biological tissue

[0156] 110 White light color camera

[0157] 111 Fluorescence camera (color or grayscale)

[0158] 112 Fluorescence / secondary image (color or grayscale)

[0159] 114 White light / background color image

[0160] 116 Fluorophore

[0161] 118 Second fluorophore

[0162] 132 Integrated / internal image display device

[0163] 134 Direct optical path

[0164] 136 Beam Splitter

[0165] 140 color conversion functions

[0166] 150a The first pixel in the background color image

[0167] 150b The second pixel in the second image

[0168] 150c Output pixel in the output color image

[0169] 152 Overlay Color

[0170] 160 Output color image

[0171] 170 Data processing equipment

[0172] 172 Output Interface

[0173] 174 Objective

[0174] 176 Color Separation Component

[0175] 178 Lighting components

[0176] 179 Lighting Filter

[0177] 180 Illumination Beam Splitter

[0178] 182 Image display device

[0179] 184 Field of view

[0180] 186 Computer

[0181] 188 White light filter

[0182] 190 Fluorescence filter

[0183] 192 Dichroic Beam Splitter

[0184] 194 Memory

[0185] 196 Data transmission line

[0186] 200 Overlay color coordinates, {X OC , Y OC , Z OC}, overlay color 202 Background color coordinates, {X BC , Y BC , Z BC}, background color 204 Light intensity value, I2

[0187] 206 Output color coordinates of the output color, {X O , Y O , Z O}, output color 210 Color Space Conversion

[0188] 212 Brightness calculation

[0189] 250 Color Conversion

[0190] 302 First Imaging Spectrum

[0191] 304, 306, 308 color space coordinates

[0192] 310 Stopband

[0193] 312 Visible spectrum or range of light

[0194] 322 Second Imaging Spectrum

[0195] 324 Passband

[0196] 326 Fluorescence emission spectra of fluorophores

[0197] 400 Uniform Color Space

[0198] 402 Brightness Line

[0199] 404 Overlay Color Line

[0200] 406 End point of brightness line

[0201] 408 First point, on the brightness line

[0202] 410 Length between the first point and the superimposed color coordinates

[0203] 412 Length between the endpoint and the first point

[0204] 414 The second point is on the superimposed color line

[0205] 420 Length between the second point and the first point

[0206] 422 The length between the background color coordinate and the second point

[0207] 440 Maximum brightness value

[0208] 442 Minimum brightness value

[0209] 444 Brightness of background color

[0210] 446 Maximum light intensity value

[0211] 448 Minimum light intensity value

[0212] 450 Planes with constant brightness

[0213] 452 Color Gamut

[0214] 500 System

[0215] 510 Microscope

[0216] 520 Computer Systems

[0217] I. Intensity

[0218] I2 light intensity value

[0219] R, G, B primary colors of RGB color space

[0220] X BC , Y BC , Z BC The color space coordinates of the color of the first pixel in uniform color space

[0221] X OC , Y OC , Z OC Color space coordinates of the superimposed color in uniform color space

[0222] X O , Y O , Z O The color space coordinates of the output pixel's color in uniform color space

[0223] λ Wavelength

Claims

1. A data processing device (170) for calculating an output color of an output pixel (150c) in an output color image (160) by mixing a background color of a first pixel (150a) in a background color image (114) with an overlay color according to a light intensity value (204) of a second pixel (150b) in a second image (112), wherein the data processing device is configured to: Retrieving background color coordinates (202) of the background color in a uniform color space (400), the background color having a predetermined maximum brightness value (440) and a predetermined minimum brightness value (442) in the uniform color space; Retrieving the superimposed color coordinates (200) of the superimposed color in the uniform color space (400); retrieving the light intensity value (204) within a range extending from and including a predetermined maximum light intensity value (446) and extending to and including a predetermined maximum light intensity value (448); Calculating a brightness line (402) in the uniform color space (400), the brightness line extending from the superimposed color coordinate (200) to an endpoint (406) at which the brightness is lower than the brightness at the superimposed color coordinate, the brightness line extending along a point having the chromaticity of the superimposed color coordinate (200); A first point (408) on the brightness line (402) is determined, wherein: The ratio of the difference between the predetermined maximum brightness value and the brightness of the background color coordinate (202) to the difference between the brightness of the background color coordinate (202) and the predetermined minimum brightness value Corresponds to a ratio of a length (410) of the brightness line between the first point (408) and the superimposed color coordinate (200) to a length (412) of the brightness line between the endpoint (406) and the first point (410); Calculating a superimposed color line (404) extending linearly from the background color coordinates (202) to the first point (408); A second point (414) on the superimposed color line (404) is determined, wherein: The ratio of the difference between the predetermined maximum light intensity value and the light intensity value to the difference between the light intensity value and the predetermined minimum light intensity value Corresponds to a ratio of a length (420) of the superimposed color line (404) between the second point (414) and the first point (408) to a length (422) of the superimposed color line (404) between the background color coordinate (202) and the second point (414); and The color coordinates (206) of the second point (414) in the uniform color space (400) are output as the output color of the output pixel (150c).

2. The data processing device (170) according to claim 1, wherein the endpoint (406) of the brightness line (402) is located within a predetermined color gamut (452) of a display device (132, 182).

3. The data processing device (170) according to claim 1 or 2, wherein the brightness line (402) is a straight line in the uniform color space (400).

4. The data processing device (170) according to any one of claims 1 to 3, wherein the data processing device (170) is configured to transform the superimposed color coordinates (200) from a non-uniform color space to the uniform color space (400) before retrieving the superimposed color coordinates (200).

5. The data processing device (170) according to any one of claims 1 to 4, wherein the data processing device is configured to transform the background color coordinates (202) from a non-uniform color space to the uniform color space (400) before retrieving the background color coordinates (202).

6. A data processing device (170) according to any one of claims 1 to 5, wherein the data processing device is configured to transform the color coordinates (206) of the second point (414) from the uniform color space to a non-uniform color space before outputting the color coordinates of the second point.

7. The data processing device (170) according to any one of claims 1 to 6, wherein the data processing device is configured to: Retrieving the first pixel (150a) from the background color image (114), the first pixel comprising the background color coordinates (202); The second pixel (150b) is retrieved from the second image (112), the second pixel comprising the light intensity value (204).

8. The data processing device (170) according to any one of claims 1 to 7, wherein the second image (112) is one of a grayscale image and a color image.

9. The data processing device (170) according to any one of claims 1 to 8, wherein the data processing device is configured to: Retrieving the background color image (114), wherein the background color image (114) is recorded in the first imaging spectrum (302), and the background color image includes a plurality of first pixels (150a); Retrieving the second image (112), the second image (112) recorded in a second imaging spectrum (322), the second image comprising a plurality of second pixels (150b); The second imaging spectrum overlaps with a fluorescence emission spectrum (326) of at least one fluorophore (116, 118), wherein the second imaging spectrum is different from the first imaging spectrum, and both the first imaging spectrum and the second imaging spectrum overlap with a visible spectrum (312).

10. A data processing device (170) according to claim 9, wherein the ratio of the brightness at the superimposed color coordinate (200) to the brightness at the end point of the luminous line is independent of the background color coordinate (202) of the first pixel (150a) and / or the light intensity value (204) of the second pixel (206).

11. The data processing device (170) according to any one of claims 1 to 10, wherein the background color image (114) and the second image (112) are registered.

12. A computer-implemented method for calculating an output color of an output pixel (150c) in an output color image (160) by mixing a background color of a first pixel (150a) in a background color image (114) with an overlay color based on a light intensity value of a second pixel (150b) in a second image (112), wherein the computer-implemented method comprises the following steps: retrieving background color coordinates (202) of the background color in a uniform color space (400), the uniform color space having a predetermined maximum brightness value (446) and a predetermined minimum brightness value (448); Retrieving superimposed color coordinates of the superimposed color in the uniform color space (200); retrieving the light intensity value (204), the light intensity value being within a range extending from and including a minimum light intensity value (448) and extending to and including a maximum light intensity value (446); calculating a brightness line (402) in the uniform color space, the brightness line extending from the superimposed color coordinate to an endpoint (406) at which the brightness is lower than the brightness at the superimposed color coordinate, the brightness line extending along a point having the chromaticity of the superimposed color coordinate; A first point on the brightness line is determined (408), where The ratio of the difference between the predetermined maximum brightness value and the brightness of the background color coordinate (202) to the difference between the brightness of the background color coordinate (202) and the predetermined minimum brightness value Corresponds to a ratio of a length (410) of the brightness line between the first point and the superimposed color coordinate (200) to a length (412) of the brightness line between the endpoint and the first point; Calculating a superimposed color line (404) extending linearly from the background color coordinates (202) to the first point; A second point on the superimposed color line is determined (414), wherein a ratio of a difference between the maximum light intensity value and the light intensity value (204) to a difference between the light intensity value (204) and the minimum light intensity value Corresponds to a ratio of a length (420) of the superimposed color line between the second point and the first point to a length (422) of the superimposed color line between the background color coordinate (202) and the second point; as well as The color coordinates (206) of the second point in the uniform color space are output as the output color.

13. A computer program product or a computer readable medium comprising instructions which, when the program is executed by a computer, cause the computer to perform the method of claim 12.

14. A medical observation device (100), such as a microscope or an endoscope, comprising: a white light color camera (110) configured to record a background color image (114); a fluorescence camera (111) configured to record a second image (112); and A data processing device (170) according to any one of claims 1 to 11.

15. A method of operating a medical viewing device (100), such as a microscope or an endoscope, the method comprising the computer-implemented method of claim 12, and further comprising the steps of: Using a white light color camera (110) to record a background color image (114) of the object; A second image (112) of the object is recorded using a fluorescent camera (111).

Images