Method for generating an overlay image and associated image recording device

Abstract

For generating realistic overlay images (1) that are color-true with respect to at least one false color FF(x,y) used to visualize a spectral image (3a, 3b), a method and an associated image recording device (5) are proposed, which make it possible to avoid undesired color shifts of such a false color in the overlay image (1) and, despite the overlay, to largely maintain a high image brightness of a white light image (2) illustrated as the image background and the original image contrast of the white light image (2) in the overlay image (1). To this end, the invention provides that at least one color channel value of the white light image (2), which is complementary to the respective false color to be displayed locally, is specifically reduced locally, and the more the higher the intensity I(x,y) of the spectral image (3a), which is also to be visualized in the overlay image 1, is at this point (x,y).

Description

[0001] The invention relates to a method for generating an overlay image, wherein the overlay image jointly visualizes image signal components of a white light image (WLI) and image signal components of a spectral image. The spectral image can be in particular in the form of a fluorescence image, thus visualizing fluorescent light. In this approach, local intensity values I(x,y) of the spectral image / fluorescence image are visualized in at least one local false color FF(x,y) in the overlay image.

[0002] In particular, several different false colors can be used to visualize the spectral image in the overlay image, with the respective false color then allowing a conclusion to be drawn about the underlying intensity of the original spectral image. If, however, only a single false color is used, the saturation or brightness of this false color, for example, can allow a conclusion to be drawn about the intensity values I(x,y) of the spectral image.

[0003] The white light image, which is also to be at least partially visualized in the overlay image, can be visualized, for example, using grayscale or using true colors in the overlay image. It can further be provided that, to generate the overlay image, local color channel values of the white light image (e.g., R / G / B values when using three color channels: red / green / blue) are offset against the associated local signal values of the spectral image. The spectral image can, for example, be present as a grayscale image, so that the signal values are then gray values (GW). Or, the spectral image can, for example, already be available in a false-color representation, in which case the local signal values can then be, for example, R / G / B values, if the spectral image is present in the false-color representation as an RGB image.

[0004] The invention further relates to an image recording device, which can preferably be configured in the form of / as part of a medical visualization system. This system can, for example, comprise an endoscope, an exoscope, or a microscope as the respective image recording device. Furthermore, the visualization system can also comprise other components, such as a camera control unit and a display device. The image recording device or this visualization system comprises at least one image sensor, in particular at least two image sensors, which is / are configured for the sensory capture of a white light image and / or a spectral image (in particular for the sensory capture of a fluorescence image).The device / system further comprises an image signal processing unit configured to calculate an overlay image (in particular as previously explained) from a white light image and a spectral image, each of which is / was recorded with the at least one image sensor. The device / system can output the overlay image, for example, in the form of a digital overlay image and / or display it on the display device.

[0005] Various types of fluorescence imaging are already known in the prior art, each of which involves capturing a white-light image separately from a fluorescence image. For example, DE 10 2011 016 138 A1, EP 2 505 989 A1, and DE 10 2021 118 427 A1 each describe approaches for graphically visualizing a white-light image and a fluorescence image using a respective (medical) endoscopy device. Publications DE 10 2019 101 773 A1, DE 10 2010 033 825 A1, and DE 10 2010 044 503 A1 describe surgical microscopes capable of simultaneously displaying a white-light image and a fluorescence image. And the publication DE 10 2019 101 777 A1 describes a microscopy method with which two different color images are recorded and each displayed or reproduced in its own gamut, which, however, exclude each other (distinct gamut).

[0006] There are also other imaging approaches in which, in addition to a white-light image, a spectral image, which then only visualizes a specific section of the spectrum used for imaging, is captured separately using sensors. In such approaches, an image generation chain is set up using signal technology that generates a white-light image and, separately from this, a spectral image. The white-light image visualizes wavelengths in the visible wavelength range across a broad band, while the spectral image visualizes a comparatively narrower spectral range, for example, in the near-infrared (NIR).

[0007] If, for example, a fluorophore is used that has an excitation wavelength in the UV and emits a fluorescence wavelength in the NIR, a white light image can be captured using a conventional color image sensor and broadband illumination. Furthermore, the NIR fluorescence wavelength can be captured spatially separately as a spectral image / fluorescence image using an additional monochrome sensor. In such a case, the fluorescence image visualizes an intensity distribution I(x,y) of a fluorescence signal captured by the monochrome sensor, which can, for example, lie in a wavelength range with a bandwidth of less than 100 nm. In this case, the monochrome fluorescence image is typically first recorded as a grayscale image.

[0008] Furthermore, it is already known to visualize such a spectral image using a false color representation. This typically involves using a mapping function that assigns a specific false color to each intensity value I(x,y) or the resulting gray value GW of the spectral image, thus resulting in a colored false color spectral image. It is even possible to use only a single false color, with the brightness and / or saturation of this false color being varied to visualize the different intensity levels in the spectral image.

[0009] In many applications, it is advantageous to visualize the white-light image and the spectral image to the user in the form of an overlay image. One advantage of this approach is that the spatial impression of the recorded scene can be visualized using the (ideally colored) white-light image, and at the same time, a fluorescence signal, for example, can be made visible using at least one false color of the spectral image in the overlay image.

[0010] There are also already approaches to using a mapping function that maps the different intensity values of the fluorescence signal visualized in the fluorescence image onto a predefined scale of different false colors. In this case, the fluorescence signal can be visualized using several false colors according to a predefined false color scale, for example, in the form of a so-called "heat map."

[0011] Especially in medical applications, there is often the problem that individual image areas, especially where surfaces are moist and therefore highly reflective, exceed the maximum displayable brightness range. As a result, these image areas are visualized in white in the white light image, but the actual color information is essentially lost because the white color indicates that all of the color channels used by the image sensor are showing maximum intensity values, a phenomenon known as "clipping." Such original clipping does not always have to be disruptive, but can even improve the spatial impression of the white light image. In image signal processing, "clipping" therefore refers to the exceeding of a permissible signal range that can therefore be displayed in the image, for example due to overloading, i.e. an overshoot of the respective signal.In relation to a white light image, this can be understood as a pixel or a larger image area in which all color channels used show a full signal deflection, i.e. the maximum possible signal value.

[0012] If the white-light image and the fluorescence image are superimposed by simply adding the signals, it may happen that individual image areas of the superimposed image, in which no original clipping was previously observed, now show additional clipping. However, such additional clipping is very disadvantageous because the surgeon viewing the superimposed image can no longer reliably determine the intensity of the fluorescence signal in these image areas, which is precisely what is intended to be visualized using false color.It would therefore be desirable, on the one hand, to retain the original clipping under certain circumstances, especially in image areas that do not show a significant fluorescence signal, but, on the other hand, to avoid additional clipping in the overlay image, which only arises when the two images are overlaid, so that the fluorescence signal is visualized with the correct false color in the overlay image.

[0013] Another technical problem that needs to be solved with overlay images is that, especially when a colored white-light image is overlaid with a spectral image / fluorescence image displayed in at least one false color, an undesirable color shift of the respective false color to be displayed can occur, particularly in those image areas in which, for example, a relevant fluorescence signal was detected by the sensor, which is to be highlighted with a false color in the overlay image. Such a color shift is problematic in practice because the user is supposed to use the false color to infer the sensory-detected intensity of the fluorescence signal. A color shift can therefore result in an incorrect interpretation of the intensity of the fluorescence signal and should therefore at least be limited or avoided entirely.This is especially necessary in medical applications because otherwise an incorrect diagnosis may occur due to an incorrectly visualized false color.

[0014] There are already approaches in the state of the art to avoid the previously described "additional clipping" in an overlay image, as well as any color shift that may occur there, namely so-called "alpha blending": The approach of this method consists in assigning a respective weighting to the respective image data, for example the RGB values of the respective pixel of the white light image and the spectral image, globally (i.e. for the entire image), and in doing so, choosing this weighting precisely so that additional clipping in the generated overlay image is avoided. Therefore, with "alpha blending," a global transparency value is generally used. The brightness of the WLI channel is therefore reduced even when no signal at all is present in the spectral image.

[0015] With such alpha blending, however, it is possible to ensure that the respective signal value of the overlay image in each color channel remains within an available / displayable value range. For example, if the entire white light image is weighted with a factor α1, the spectral image can be added after weighting with the factor α2 = 1-α1 to obtain an overlay image. While this approach avoids additional clipping, it has the critical disadvantage that the global weighting causes image brightness to be lost even in image areas where it is not absolutely necessary. In other words, overlay images generated with this method often show insufficient brightness of the white light background.

[0016] When generating overlay images, it is also generally desirable to avoid any loss of contrast and also any loss of color saturation of the spectral image, in particular any loss of color saturation of the false colors in which the spectral image is visualized in the overlay image.

[0017] Based on this background, the invention is based on the object of proposing an improved method that solves the above-mentioned problem and can deliver true-color overlay images with high color saturation and high contrast. Furthermore, the invention also aims to provide an associated image recording device with which such high-quality overlay images can be reliably generated, particularly for medical applications.

[0018] To achieve the stated object, the features of method claim 1 are provided according to the invention. In particular, to achieve the object in a method as described above, according to the first alternative of claim 1, it is proposed according to the invention that at least one complementary color channel value of the white light image is lowered locally and selectively compared to the remaining local color channel values of the white light image. This lowering does not occur globally (as with alpha blending), but rather in direct or at least indirect dependence on an associated local intensity value I(x,y) of the spectral image. The complementary color channel value is complementary to at least one local false color in which (as mentioned above) the local intensity values I(x,y) of the spectral image / fluorescence image are (to be) visualized.

[0019] In other words, as mentioned at the beginning, the spectral image in the overlay image is visualized using at least one local false color. "Local" here means that this at least one false color can change throughout the overlay image depending on the local intensity value I(x,y) of the original spectral image: For example, the color value (hue) of the false color can change, or, for example, when using only a single green false color, its saturation or brightness value. For a specific location (x,y) within the overlay image and the false color FF(x,y) to be visualized there, a complementary color can be determined at any time that is complementary to this local false color.Accordingly, the complementary color and thus the complementary color channel value of the white light image can also change across the overlay image: If a blue false color is to be visualized at a first location, the red and green color channels of the white light image are complementary to it at this location. However, if a yellow false color is to be visualized at a second location, the blue color channel of the white light image is complementary at this second location. Accordingly, according to the method according to the invention, the red and green color channel values (R and G values) of the white light image are lowered at the first location, and the blue color channel value (B value) is lowered at the second location.

[0020] It is therefore possible (location-dependent) to determine which of the color channels used to display the white light image (typically three color channels: R, G, B) is complementary to the local false color. The color channel value of this complementary color channel can then be lowered locally, i.e., at the location of the said local false color.

[0021] The technical effect achieved by selective, location-dependent attenuation consists, first of all, in the color fidelity of the white light image being somewhat distorted. However, when the white light image is superposed with the spectral image in the overlay image, only those color components of the white light image are accentuated that correspond to the local false color to be displayed according to the spectral image. The complementary colors of the white light image, or more precisely, the color channel value of the corresponding at least one complementary color channel of the white light image, are attenuated by the aforementioned attenuation, which precisely prevents a significant color shift of the local false color to be displayed in the overlay image during the aforementioned superposition.In other words, this ensures that the false colors in the overlay image are visualized true to color, i.e., without significant color shift. This allows the user to reliably infer the underlying intensity of the original spectral image based on the visualized false color, which, among other things, enables more accurate diagnoses in medical applications.

[0022] In the said method, it is preferred if the remaining / residual color channel values (e.g., R and G, provided B is the complementary color channel value) of the white light image are locally and selectively maintained or even increased in direct or indirect dependence on the local intensity value I (x, y) of the spectral image. By such a targeted local increase of the remaining color channel values, while simultaneously lowering the complementary color channel value, the image brightness can be at least approximately maintained locally. This allows the contrast ratios of the white light image to be better preserved, thus resulting in a more realistic overlay image.

[0023] It is further preferred in the method if the at least one complementary color channel value of the white light image is reduced more significantly, depending on the location, the higher the respective local intensity value I(x,y) of the spectral image. Here, too, this reduction does not necessarily have to be a direct function of the local intensity value I(x,y), but rather, as will become clearer, can also be based, for example, on a false color, which in turn depends on the local intensity value I(x,y) (this would then be an indirect function of I(x,y)). For example, false color values of a false color representation of the spectral image can also be used as input variables to determine which color channel value of the white light image at a specific location (x,y) of the overlay image is complementary to the local false color to be visualized at that location.It should be noted that in image areas where the spectral image does not exhibit any appreciable intensity, there is a lower risk of color deviations, so that the complementary color channel value of the white light image needs to be reduced very little or not at all there. However, this is advantageous in order to maintain the color fidelity of these image areas of the white light image in the overlay image. Imagine an image area where the spectral image shows an intensity value of zero: There, the white light image does not need to be adjusted at all, i.e. there is no need for any reduction of the complementary color channel value there either, so that the white light image can be visualized in the overlay image without any color change in these image areas.

[0024] In summary, it can be said that the at least one complementary color channel value of the white light image can be reduced to different degrees, in particular not at all, at different locations / points (x,y) of the overlay image, namely depending on the associated local intensity value I(x,y) of the spectral image.

[0025] It should also be mentioned at this point that the method according to the invention can of course be used to generate a continuous video image data stream of live overlay images, starting from a first stream of live white light images and a second stream of live spectral images. In this case, the intensity I(x,y) of the spectral image can change over time, and accordingly, which color channel value of the white light image is reduced at which points and to what extent will also change from image to image. This is not comparable to time-varying alpha blending, because with alpha blending only a global weighting is changed, but not spatially resolved in at least an indirect dependence on a local I(x,y) of the spectral image.

[0026] It should also be mentioned that, for example, color channel values (R, G, B) of a false-color representation of the spectral image can also be used as input variables in the method to determine which color channel / color channel value of the white light image is complementary to the local false color at a specific location (x, y) of the white light image. In other words, the reduction of the complementary color channel value can then be carried out in direct dependence on color channel values of a false-color spectral image that is to be visualized in the overlay image. However, even in this case, there is at least an indirect dependence on the associated local intensity I(x, y) of the spectral image, because the false colors of the false-color spectral image are a function of the respective local intensity value I(x, y) of the original spectral image, i.e., they directly depend on these intensity values.

[0027] The term "local" can be understood here in particular to mean that the described local manipulation of the respective color channel value of the white light image is performed at a specific location (x,y) within the overlay image, depending on a corresponding intensity value I(x,y) at the same location in the spectral image / fluorescence image. It is understood that for this purpose, the white light image and the spectral image should have the same size for the desired overlay. If this is not originally the case, for example, due to different spatial resolutions of the images, an identical image size can be achieved by appropriately scaling at least one of the two images.

[0028] The method according to the invention can be used, for example, to adjust the local color channel values (R / G / B) of the white light image (2) belonging to those locations (x,y) of the white light image (2) at which high intensity values I(x,y) were sensorily detected in the corresponding spectral image (3a) in such a way that the color channel values (R / G / B) of the white light image (2) that do not correspond to the associated local signal values (R / G / B) of the false color representation FF(x,y) of the spectral image at these locations (x,y), i.e., are complementary to the respective false color, are reduced. This can effectively prevent a relevant color shift (color drift) with respect to the false color to be represented in the thus generated overlay image when the white light image is superimposed on the false color representation of the spectral image.

[0029] The essential technical advantage of the method according to the invention is that a desired false color representation of the spectral image, which can be linked, for example, to an intensity false color scale, is retained. It should be noted that a superposition of identical or similar color components does not lead to a relevant "color drift," but rather only the superposition of a false color with its respective complementary color. However, because the method specifically suppresses these complementary colors locally in the relevant image areas (namely, where relevant intensity I(x,y) is observed in the spectral image) (through the targeted, spatially resolved / local lowering of the complementary color channel values (e.g.By determining the B-value for a yellow false color of the white light image as a function of the corresponding intensity I(x,y) of the spectral image, a color drift in the overlay image with respect to the respective false color to be displayed is very effectively avoided. This approach is equally applicable when only a single false color (e.g., with different saturation and / or brightness) is used to visualize the intensity of the spectral image (e.g., light to dark green) or when multiple false colors (e.g., a continuous false color scale from black through dark blue, turquoise, green to yellow) are used and the intensity I(x,y) of the spectral image is thus visualized, in particular exclusively, via a change in the color value (hue) of the false color.

[0030] For example, consider the HSV color space (H = hue = color value; S = saturation = color saturation; V = value = brightness): The false color representation can, for example, only change the brightness V for a certain H value; or brightness and saturation are kept constant and the hue value is varied depending on the intensity I(x,y) (= tracing a circular arc in the HSV space).

[0031] In both cases, the result is a very realistic and bright overlay image whose white light components in the low-intensity regions I(x,y) of the spectral image can be reproduced virtually unadulterated (because no significant adjustment of the image signal components of the white light image is made there), while the false-color representation of the intensity distribution of the original spectral image in the overlay image is color-true in the sense that a user can still reliably and accurately infer the original intensity I(x,y) in the spectral image based on the visualized false color and / or its saturation and / or its brightness. This significantly improves image perception and enables more precise interpretation of the overlay image, particularly for users who use the method in medical imaging. This can, for example, lead to improved diagnostics.

[0032] Viewed from a slightly different perspective, the inventive approach can also be described as follows (cf. the second alternative in claim 1): To achieve the object, it can be provided (in particular in addition to the method features explained above) according to the second technical alternative of claim 1 that the overlay image jointly visualizes image signal components of a white light image and image signal components of a spectral image, in particular a fluorescence image, and that to generate the overlay image, a respective ratio with which respective local color channel values (for example R / G / B) of the white light image are offset against associated respective local signal values (for example R / G / B or GB) of the spectral image is selected in a location-dependent manner and in direct or indirect dependence on the respective original local intensity value I (x, y) of a respective pixel (x, y) of the spectral image.This ratio, which can be understood as a kind of local weighting between the white light image and the spectral image, can therefore preferably be selected / changed pixel by pixel, in particular as described above in direct or at least indirect dependence on the local intensity value of the spectral image.

[0033] With this approach, the respective weighting between the local color channel values of the white light image and the local signal values of the spectral image can change location-dependently in the overlay image, namely depending on the local original intensity value I (x, y) of the spectral image. This change can be selected pixel-by-pixel or at least for specific image regions individually; that is, this change can change pixel-by-pixel or image region-by-image.

[0034] According to the invention, a suitable weighting between the image signal components of the white light image and the image signal components of the spectral image / fluorescence image can therefore be applied at a specific point in time in the overlay image (unlike global alpha blending). With the inventive approach, this local weighting can change depending on the current intensity distribution I(x,y) of the spectral image just acquired, so that the distribution of the weighting across the overlay image can also change from image acquisition time to image acquisition time. The inventive method thus allows, for example, a temporally and spatially dynamic adaptation of the overlay process when generating a video image data stream (consisting of continuously recalculated overlay images based on a respective first video stream of white light images and a second video stream of spectral images).The dynamics arise from the temporal and / or spatial change of the intensity I(x,y) of the spectral image, because this influences the respective local weighting that is applied locally when the two images are superimposed.

[0035] The original spectral image can have respective signal values / pixel values that correspond to a respective sensor-detected intensity I(x,y) of a spectral light signal, in particular a fluorescence signal. As an intermediate step, however, a false color representation of the spectral image (false color image = FCI) can also be generated in a conventional manner. If a "heat map" is used for this purpose as a mapping function, which maps a respective intensity I(x,y) into a space of false colors (this false color subspace can cover only a single color value or, for example, a continuous range of color values / hue values, as previously explained for the HSV color space), this false color image (FCI) can also have different false color signal values, in particular different false colors / color values / hue values.Thus, the signal values of the spectral image can basically be present as intensity values I(x,y) or, after appropriate image processing, already as false color signal values (e.g. as R / G / B values).

[0036] Color channel values of the white light image (WLI) can be, for example: R / G / B values, i.e. signal values of a respective color channel (= color signal value).

[0037] From a slightly different perspective, the inventive approach can also be described as follows: In a method for generating an overlay image according to claim 1, which can be designed in particular as described above, it can also be provided that the overlay image jointly visualizes image signal components of a white light image and image signal components of a spectral image, in particular a fluorescence image, and that to generate the overlay image, a respective ratio is selected depending on the location, in particular pixel by pixel, with which local color channel values (R / G / B) of the white light image are offset against associated local signal values (R / G / B; GW) of the spectral image.To achieve the object, according to the third technical alternative of claim 1, it can further be provided that the respective ratio is calculated using an overlay function OF(I) = f(I(x,y)), which, in direct or indirect dependence on a respective local intensity value I(x,y) of the original spectral image, specifies the amount by which the respective local color channel value of the white light image is raised / amplified / accentuated or lowered / attenuated / reduced during the said calculation to generate the overlay image. This overlay function can therefore be understood as an "overlay function" that specifies the respective local weighting with which the two images are to be superimposed on one another in different image areas of the overlay image.

[0038] The overlay function OF(I) can therefore, in particular, determine which color channel values (e.g., R / G / B) of the white light image are to be attenuated as complementary color channel values compared to a specific local false color FF(x,y) to be displayed in the overlay image, and / or which of these color channel values of the white light image are to be amplified as non-complementary color channel values or at least not attenuated. With this approach, the local false color FF(x,y) to be displayed can also be displayed in the overlay image with true color, i.e., without a significant color shift. A color shift would be classified as significant if it leads to a significant error with regard to the intensity I(x,y) of the spectral image underlying the false color.

[0039] For example, if the overlay function OF(I) is to generate an RGB representation of the overlay image, it can be specified as a vector of three respective correction functions OF(I)=(ΔR(I), ΔG(I), ΔB(I)). Here, for example, ΔG(I) is the correction function that specifies by which value the color value G of the green color channel of the overlay image is increased or reduced, depending on the intensity I(x,y) of the spectral image. Technically equivalent, the overlay function OF(I) can also be generated using respective correction functions (each per color channel, e.g. for red: ΔR=f(R FF ,G FF ,B FF )) which do not take into account the intensity I(x,y) of the spectral image, but R FF - / G FF - / B FF -values of a false color representation FF(x,y) of the spectral image, where these false colors FF(x,y)=f(I(x,y)) then depend on the intensity I(x,y) of the spectral image, for example: ΔR=f(RFF(I(x,y)),GFF(I(x,y)),BFF(I(x,y))) with e.g. G FF (I(x,y)) the false color value of the green color channel of the false color spectral image (FCI), which is calculated from the local intensity I(x,y) of the spectral image depending on the location.

[0040] The three methods explained above, which can also be used in combination to solve the problem, can be further developed as follows: For example, it can be provided that when calculating the overlay image from the original spectral image, a false-color representation FF(x,y) of the spectral image is generated in the form of a false-color spectral image using a mapping function MF(I). This step can, in particular, be an intermediate step in the calculation of the overlay image. Furthermore, the said false-color representation or the false-color spectral image can also be only partially visualized in the overlay image. For example, in image areas where the original spectral image does not display any significant intensity, the false-color representation can, under certain circumstances, be completely hidden, so that in these image areas of the overlay image, only the original white light image is visualized, possibly without any modification.When using such a mapping function MF(I), it is preferable if it assigns a specific local false color FF(x,y) to a respective local intensity value I(x,y) of the original spectral image. This describes a mathematical mapping between a spatially recorded range of possible intensity values in the spectral image (for example, with an 8-bit resolution in a value range of 0..255) and a false color space in which the respective false color is encoded, such as the HSV color space.

[0041] The mapping function MF(I) can map into a false color subspace (for example, within the aforementioned HSV color space). This false color subspace, which thus represents a subset of an available color space, can, in particular, comprise only a single false color value H, but, for example, different color saturation values S and / or different color brightness values H. If, for example, the intensity of the spectral image is to be represented using the false color green, it is sufficient to change the color saturation value S of this false color in the false color spectral image in order to visualize the intensity distribution in the original spectral image.

[0042] The false color subspace can also comprise several different false color values H. For example, false colors ranging from dark blue to light blue, turquoise, green, and yellow can be used to visualize the intensity of the original spectral image in a "heat map." In such a case, it is preferable if these different false color values span a continuous path within the false color subspace. In this case, a continuous intensity distribution of the original spectral image can be visualized in the overlay image by a continuous color gradient within the false color subspace. This is advantageous for being able to infer the original intensity I(x,y) as accurately as possible from the overlay image.

[0043] A preferred embodiment therefore provides for a visualization system, for example in the form of an endoscope or a microscope (which can be configured in particular as described here), to be configured such that it is set up for the sensory capture of a white light image (WLI) and a spectral image, i.e. in particular a fluorescence image, as well as for visualizing an overlay image. The overlay image jointly visualizes image signal components of the white light image and image signal components of the spectral image. The visualization system is further configured to visualize said overlay image using an overall color space, wherein the white light image is visualized using colors of a VIS color space as a subset of the overall color space, and the spectral image is visualized using false colors of a remaining residual color space of the overall color space (i.e., the VIS color space and the residual color space complement each other to form the overall color space used).Furthermore, the false colors of the residual color space are chosen to be disjoint from the colors of the VIS color space; that is, the false colors are not part of the VIS color space, but lie exclusively in the residual color space. The size of the VIS color space can also change depending on the current VIS image and the tissue visualized therein (for example, if initially no fatty tissue is visible and then yellow fatty tissue is visible, which is to be visualized in the VIS image). It is therefore characteristic that the false color scale (or the color range of the false colors used) never penetrates the VIS color space (i.e., is designed to be disjoint from it).

[0044] This approach can be understood as meaning that the visualization system renders a VIS image in a color representation that occupies a specific portion of the available color space, for example, the color spectrum from green to red. Accordingly, there is then a residual color space, namely the remaining portion of the total available color space. The invention has recognized that it is advantageous for visualization if the false colors used to visualize the spectral image / fluorescence image are disjoint from the portion of the color space used to display the VIS image. This makes it possible, for example, to precisely identify where fluorescence occurs in the scene.

[0045] Preferably, the false color scale can even be designed to complement the VIS color space, i.e., to occupy the entire remaining residual color space. This is advantageous because the false color scale can then have a maximum length / color bandwidth, which in turn is advantageous for being able to resolve as many intensity levels of the fluorescence signal as possible as finely as possible using different colors. If, for example, only the color space from turquoise to yellow were used, the user could potentially recognize three colors: yellow, orange, and green. With the solution proposed here, however, the false color space can range from dark blue through turquoise, green, orange, and yellow, so that the surgeon can distinguish five different false colors and thus five intensity levels on the screen.

[0046] In particular, it can be provided that the spectral image is visualized using a false color scale (color map) that extends continuously within the residual color space. Preferably, the false color scale extends over at least 80% of the remaining residual color space. This allows intensity differences in the fluorescence signal to be visualized using a wide range of false colors from the residual color space. The false color scale is ideally displayed to the user together with the VIS image and / or the overlay image, so that the user can easily infer the visualized intensity of the fluorescence signal based on the scale.

[0047] The visualization system can also be configured to continuously determine the current VIS color space based on image information from a sensor-captured VIS image (and thus automatically also the remaining residual color space) and to define the false color scale based on the determined current VIS color space. For example, if the range of colors in the VIS color space currently captured by the sensor (with the image sensor) increases, perhaps because previously invisible yellow fatty tissue now comes into view, the visualization system can reduce the color bandwidth of the residual space accordingly, for example, transferring yellow as a false color from the residual color space to the VIS color space. This approach always enables optimal false color representation with the largest possible color bandwidth.According to this approach, the bandwidth of the residual color space used for false color representation is dynamically adjusted depending on which color bandwidth is currently required to display the VIS signal in the VIS color space.

[0048] For example, if yellow fatty tissue is visualized in the VIS image, it is advantageous if the false color scale no longer extends into the yellow range. It is particularly advantageous if the visualization system itself continuously determines the occupied VIS color space from a current VIS image (captured by the system's image sensor) and, from this, always determines the available residual color space. The visualization system can then automatically adjust the false color scale so that the false color scale always lies within the currently determined residual color space. This allows the surgeon to visualize different intensity levels or different local signal values of the spectral image using the false color scale with the finest possible color gradation.

[0049] As explained, the previously mentioned mapping function assigns a specific false color to an intensity. The mapping function can be configured to perform this assignment / mapping for all pixels of the false color representation FF(x,y), which is calculated from the intensity distribution I(x,y) of the spectral image using the mapping function: MF(x,y)=f(I(x,y)).

[0050] For example, if the mapping function is to generate an RGB false color representation, it can be described as a vector of three individual mapping functions for the respective color channel value (R / G / B), for example: MF(I)=(R(I),G(I),B(I)), where, for example, R(I) represents the mapping function for the red color channel, which assigns a specific color value R(I) of the red color channel to each intensity value I(x,y).

[0051] The previously described first alternative with only a single false color value can be achieved, for example, by means of a mapping function based on a brightness scale and / or a saturation scale of a specific false color, e.g., green. The second alternative (i.e., when using different false color values), on the other hand, can be implemented, for example, by means of a mapping function based on a, preferably continuous, false color scale. This false color scale can therefore comprise a continuous spectrum of different color values. In the second alternative, the false color spectral image can thus be present, for example, in the form of a colored "heat map"; in the first alternative, as a brightness image in a single green false color.The false color subspace (residual color space) can be selected in a specific way, particularly depending on the color subspace occupied by the white light image (WLI) (VIS color space). Typically, the false color subspace will only cover a portion of the total color space available for displaying the overlay image; the remainder of the color space can then be occupied by the WLI image without causing color overlap, which is advantageous for reliable color interpretation of the overlay image. This will be explained in more detail below.

[0052] At least “partial” visualization can be understood here to mean that only certain sections of the false color spectral image can be visualized in the overlay image or that, for example, certain areas of the false color spectral image can be hidden in the overlay image.

[0053] The false-color spectral image or false-color representation FF(x,y) can preferably visualize the false colors blue, turquoise, green, and yellow, preferably in this order. This allows for a particularly advantageous, precisely resolvable, color visualization of different intensity levels of the original spectral image, especially in medical applications.

[0054] The mapping function MF(x,y) can therefore, in particular, describe a continuous false color gradient in the false color subspace, preferably from blue to turquoise to green to yellow. It is particularly preferred if high intensity values of the spectral image are assigned to a yellow false color and low intensity values of the spectral image are assigned to a blue false color. This corresponds to a color value-intensity inference that is intuitively correctly interpreted by humans. The false color spectral image can also include black as a false color, particularly in those image areas where the intensity values of the spectral image are below an intensity threshold.

[0055] The false-color spectral image or false-color representation FF(x,y) can, for example, also visualize only the false color green and, if necessary, the false color black. This also allows a complex intensity distribution of the original spectral image to be visualized in the overlay image. The mapping function MF(x,y) can therefore describe a continuous brightness and / or saturation gradient in the false-color subspace for at least one false color, in particular for a false color gradient from black to green. It is preferred if high intensity values of the spectral image are assigned to a green false color and low intensity values of the spectral image are assigned to a black false color.

[0056] The method according to the invention can provide for the local color channel values of the white light image to be offset against the respective false color signal values of the false color spectral image, each with a different local weighting, depending on the local original intensity value of the spectral image. This approach is particularly applicable to the aforementioned location-dependent offsetting.

[0057] As already mentioned, a respective local false color of the false-color spectral image (i.e., a false color that is to be used locally to visualize the intensity of the original spectral image in the overlay image) can define a respective local complementary color KF(x,y). In this case, it can be provided that the aforementioned at least one color channel value of the white-light image corresponding to this complementary color KF(x,y) is lowered in a location-dependent manner, in particular pixel-by-pixel. With this approach, a color shift of the respective local false color in the resulting overlay image, which could arise due to the offsetting of the white-light image with the spectral image, can be at least limited or even completely prevented.

[0058] In all of the aforementioned approaches, it is technically equivalent according to the invention if the described lowering of the complementary color channel value and / or the respective calculation and / or the described ratio is not carried out in direct dependence on the intensity value I(x,y), but only in dependence on a false color FF(x,y) to be displayed locally. This is because the false color FF(x,y) to be displayed - more precisely its color value H and / or its saturation S and / or its brightness V - is a function of the original local intensity I(x,y) of the spectral image (because the false color is intended to visualize the intensity distribution of the possibly monochromatic original spectral image / fluorescence image), the type of superimposition in this approach is also carried out spatially resolved in at least indirect dependence on the respective intensity value I(x,y) of the spectral image.This also applies if the false color representation FF(x,y) is a non-linear function of I(x,y), because even then it still depends on I(x,y).

[0059] Of course, the methods according to the invention can also be applied to individual image regions. This means that the entire portion of a spectral image already captured by sensors does not necessarily have to be taken into account; rather, the said superimposition can, for example, be limited to only those image regions in which a significant intensity, and therefore one that can be displayed, was actually captured by sensors in the original spectral image.

[0060] As already indicated, the overlay image can be calculated based on an overlay function OF(x,y) – which can be understood as an "overlay function." This overlay function can, in particular, be predefined and / or customizable by a user (e.g., via a user interface of the image acquisition device used). The overlay function then specifies, directly or at least indirectly dependent on a respective local intensity value I(x,y) of the original spectral image, the amount by which the respective local color channel value of the white light image should be raised / amplified / accentuated or lowered / attenuated / reduced.In this case, it is preferred if a local color channel value of the white light image is raised / amplified / accentuated if this local color channel value corresponds to an associated local false color that is to be realized as an image signal component of the spectral image at this point (x, y) in the overlay image. Furthermore, it may be advantageous if a local color channel value of the white light image is lowered / attenuated / reduced if it is complementary to an associated local false color that is to be visualized as an image signal component of the spectral image at this point (x, y) in the overlay image.

[0061] As already mentioned at the beginning, the white light image and the spectral image / fluorescence image can both be captured using a single image recording device. This device can be configured, in particular, in the form of an endoscope, an exoscope, or a microscope, and will be explained in more detail later. However, this device can comprise multiple image sensors.

[0062] When capturing the two images separately, a variety of approaches are conceivable: For example, the white light image and the spectral image / fluorescence image a. are spatially separated / separately detected by sensors, in particular by means of at least two separate image sensors and / or by means of different color filters at the pixel level of an image sensor, in particular an RGBX sensor or a hyperspectral sensor; or b. The separate sensory capture can be performed at different times, particularly by means of temporally varying illumination. Such temporally separate capture can also be achieved by alternately capturing the two images with one (possibly single) image sensor. c. The separate sensory capture of the two images can also be achieved solely by intelligent signal processing, in particular whereby both images can be captured simultaneously and / or by only a single image sensor.

[0063] All of these approaches for separating a spectral image / fluorescence image from a white light image are fundamentally known from the prior art; however, they can be advantageously used in the inventive approach for generating high-quality overlay images.

[0064] As mentioned, only parts of the spectral image, i.e. certain image sections of the spectral image, can be visualized in the overlay image in a false color representation.

[0065] The respective components of the white-light image can, however, be visualized in the overlay image as a colored white-light background image, preferably in true color, or as a monochrome grayscale background image. Even with a grayscale image, the advantage of the invention, namely that no additional clipping is generated, is retained. Even clipped VIS / WLI image components are still colored.

[0066] The white-light image can accordingly comprise color information from at least two different color channels if it is visualized in color in the overlay image. However, it can also be visualized as a grayscale image.

[0067] The original spectral image, however, can visualize intensity values of a spectral light signal, particularly a fluorescence signal, detected sensorily (with an image sensor). If the invention is used in a minimal solution, the white light image does not necessarily have to be visualized in color in the overlay image, and the spectral image can also be represented, for example, using only a single false color in the visualization image.

[0068] A particularly preferred embodiment of the method provides that, during the calculation of the overlay image, a spatially resolved check, i.e., in particular, pixel-by-pixel, is performed to determine whether the overlay of the image signal components of the white-light image with the image signal components of the spectral image would exceed a value range available for the overlay image and / or the maximum value that can be represented in the overlay image. Should this be the case, the image signal components of the white-light image, in particular the aforementioned at least one complementary color channel value, can be reduced to such an extent that the value range is locally maintained. This approach precisely prevents additional clipping from occurring in the overlay image, as described above and often observed in the prior art.

[0069] It should be noted at this point that original clipping image areas of the white light image that already exceeded / exceeded the displayable value range (particularly in the form of white image spots) can still be displayed in the overlay image. This applies in particular to image areas in which the original spectral image has negligible intensity and is therefore displayed little or not at all. These original clipping image areas can, under certain circumstances, be visualized in a correct false color in the overlay image, provided that the spectral image has a significant intensity corresponding to the respective original clipping image area, so that the false color should then be visualized accordingly in this image area.

[0070] In simple terms, this approach prevents additional "white spots" from occurring in the overlay image in those image regions where a relevant intensity I(x,y) of the spectral image is to be visualized using false color. However, the false color can only be correctly visualized if not all color channels used to visualize the overlay image (locally) have a maximum displayable signal value (which would correspond to the color white in the overlay image). With this approach, an original clipping that already existed in the recorded white light image because individual pixels of the image sensor used showed a full signal deflection can be retained in the overlay image. However, the method according to the invention ensures that no significant "color drift" occurs there with regard to the image signal component of the spectral image visualized using false color in the overlay image.At the same time, additional clipping, which could only occur when the two images are overlaid and could thus distort the false color representation in the overlay image, can also be avoided.

[0071] A further embodiment of the method provides that when calculating the overlay image, the different color channel values of the white light image are raised and / or lowered in a spatially resolved manner, in particular pixel by pixel, in direct or indirect dependence on the local intensity value of the spectral image in such a way that a local overall brightness is at least approximately maintained, i.e. in particular with a relative brightness loss of less than 25%, preferably of less than 10%. The local overall brightness can result in particular from a sum of the color channel values of all color channels of the white light image. This does not necessarily have to be the case, however. This approach can ensure that no significant image brightness of the white light image is lost in the overlay image and / or that a brightness contrast of the white light image is largely maintained locally despite the overlay in the overlay image.

[0072] For example, if the white light image is an RGB image with pixels each having an R, G, and B color channel value, the above condition can be met if it is ensured that even after the respective adjustment of these color channel values of the white light image during the calculation of the overlay image, the sum of these three R / G / B signal values changes by less than 20%, preferably by less than 10%. However, it should be noted that, depending on the image format, image brightness does not necessarily have to be a simple addition of color channel values; rather, a more complex relationship may exist between these variables, which can then be taken into account accordingly.

[0073] A further variant of the method provides that a local intensity threshold I0 is defined for the spectral image and that an inventive reduction of at least one color channel value of the white light image is only carried out locally if the intensity threshold I0 is exceeded in the spectral image at this point (x,y), i.e. if I(x,y) > I0. The intensity threshold I0 should be selected to be positive. In this case, the image areas of the white light image in which the spectral image only shows intensity values below the threshold I0 can be overlaid with the spectral image unchanged with regard to the local color values. This allows the color fastness of the WLI background in the overlay image to be maintained in the image areas that are less relevant with regard to the spectral image. The said color channel value to be reduced can, for example, be the aforementioned complementary color value.

[0074] In other words, it can be provided that no reduction of at least one color channel value (e.g. B value) of the white light image is carried out if the local intensity value I(x,y) = 0 or < I0.

[0075] To achieve the object, as mentioned, an image recording device is also proposed, which can be configured in particular in the form of a medical visualization system and / or can comprise an endoscope, an exoscope, or a microscope. This device, which was already described at the beginning, further comprises, to achieve the object, an image signal processing unit configured to implement a method as described above and / or according to one of the method claims. The image signal processing unit calculates the overlay image from the white light image and the spectral image / fluorescence image, in particular taking into account a previously calculated false-color representation of the spectral image / a false-color spectral image.

[0076] The image recording device can also comprise (in a manner known per se) an excitation light source for generating excitation light. The excitation light can generate spontaneous emission and thus fluorescent light, which is then sensorily detected by the at least one image sensor when the fluorescent image is recorded.

[0077] In particular, the image recording device can comprise, for example, two image sensors, such as a color image sensor for sensory detection of the white light image and a second, in particular monochromatic, image sensor which is configured for spatially separate detection of the spectral image / the fluorescence image.

[0078] The image recording device can further be designed and / or configured analogously as described above with reference to alternatives i) to iii) (see also claim 11).

[0079] The invention will now be described in more detail using exemplary embodiments, but is not limited to these embodiments. Further developments of the invention can be derived from the following description of a preferred embodiment in conjunction with the general description, the claims, and the drawings. In the following description of various preferred embodiments of the invention, elements that correspond in function are given the same reference numerals, even if they have different designs or shapes.

[0080] It shows: Fig. 1 a state-of-the-art approach for spatially separate acquisition of a white light image and a fluorescence image, Fig. 2 shows another approach known from the prior art for the temporally separate acquisition of a white light image and a fluorescence image, Fig. 3 a concept known from the state of the art, the so-called alpha blending, for superimposing a spectral image with a white light image, Fig. 4 a possible implementation of a method according to the invention at the signal processing level, Fig. 5 illustrates previously known problems that can arise when overlaying a fluorescence image with a white light image, Fig. Figure 6 illustrates the application of a method according to the invention to obtain a high-quality overlay image, Fig. 7 illustrates a mapping function with which intensity values of a spectral image can be mapped into a false color space, Fig. 8 illustrates an overlay function that can be used to specify how individual color channel values of the white light image are to be adjusted when overlaid with the spectral image, Fig. 9 illustrates another overlay function, in case the spectral image is to be illustrated with several different false colors, Fig. 10 illustrates the application of the method according to the invention to the superposition of a spectral image, which is visualized with only a single green false color, with a white light image, Fig. 11 illustrates the HSV color space, Fig. 12 illustrates another view of the HSV color space, Fig. 13 illustrates, using a schematic example of a spectral image showing an intensity distribution, how different image areas of the spectral image (for each color channel individually) can be adjusted according to the invention, Fig. 14 illustrates a method according to the invention at the image signal processing level for the case of a false color spectral image which is visualized with only a single green false color, and finally Fig. 15 the application of a method according to the invention at the signal processing level for the case of a false color spectral image which is to be illustrated as an RGB image by means of several different false colors in the overlay image.

[0081] Fig. Figure 1 shows an image recording device 5 comprising a video camera 15 as part of an endoscope with imaging optics 17 that directs light emitted by an object 18 onto a beam splitter 16. The observed scene is illuminated with broadband white light with wavelengths in both the UV and visible ranges. The object 18 also includes fluorophores that are excited by the UV wavelengths to spontaneous emission in the infrared wavelength range (IR). Fluorophores that can be excited with wavelengths in the NIR range can of course also be used.

[0082] The beam splitter 16 directs the IR wavelengths, i.e., the fluorescent light, onto a monochrome image sensor 6b, while the visible wavelengths reach a conventional color image sensor 6a. Thus, the image recording device 5 can capture a white light image 2 using the image sensor 6a and, simultaneously but spatially separated from it, a spectral image 3 in the form of a fluorescent image 4 using the second image sensor 6b. The image recording device 5 further comprises an image signal processing unit 7, which can consist of several electronic components and is configured to calculate a digital overlay image 1 from the recorded white light image 2 and the recorded spectral image 3. This overlay image 1 can then be displayed on the display unit 13.

[0083] The image recording device 5 according to the prior art, which is already known from the prior art, Fig. 1 can be designed according to the invention if, as the star symbol in the Fig. 1, the image signal processing unit 7 is configured to implement a method according to the invention.

[0084] The same applies to the Fig. 2, also known from the prior art, image recording device 5, which differs only in that it is Fig. 1 distinguishes that in Fig. 2, only a single image sensor 6 is used, which, however, captures the white light image 2 and the fluorescent image 4 separately from each other in time. For this purpose, alternating illumination consisting of illumination light from an illumination light source 11 in the visible range and UV excitation light from an excitation light source 12 is used, with these two light sources 11, 12 illuminating the object 18 alternately in time. Accordingly, illumination light 19 and fluorescent light 20 are alternately supplied to the imaging optics 17 of the image recording device 5.

[0085] The Fig. Figure 3 shows a first approach known from the prior art, the so-called alpha blending, with which a white light image 2 and a spectral image 3, for example in the form of a false color spectral image 3b, can be superimposed to form an overlay image 1 by weighted addition at the signal processing level. For this purpose, the white light image 2 is weighted with a factor α1 < 1, while the spectral image 3b is globally weighted with a complementary weighting factor α2 = 1-α1. Although this approach can fundamentally prevent additional clipping in the overlay image 1, the disadvantage of this approach is primarily that image brightness is lost with respect to the components of the white light image, as can be seen from the comparison of the detail in the upper right corner of the Fig. 3 with the original white light image 2.

[0086] In the white light image 2 of the Fig. 3 also shows an original clipping 9, i.e. image areas that are displayed in white because all three RGB color channels of the white light image 2 already show full deflection there. As can be seen from detail A, an unsuitable choice of the weighting factor α1 can also lead to additional clipping 10 here, i.e. white image areas that only arise through the superimposition of the two images 2, 3. In detail B, on the other hand, a reduced image brightness 22 can be seen as a result of the application of alpha blending. Due to the globally reduced image brightness of the white light image 2, it can also be seen in detail C that an original optical reflection 23, which was still visible in the original white light image 2, is no longer visible in the overlay image 1.The result is a representation in the overlay image 1 that provides a deteriorated spatial impression of the observed scene compared to the original white light image 2.

[0087] In order to understand the inventive approach for generating a high-quality overlay image 1, the Fig. 13 is helpful: This shows a schematic illustration of an intensity distribution I(x,y) of a spectral image 3a at the top. As can be seen from the scale, the intensity of a spectrally narrowly defined light, for example, fluorescent light, is spatially recorded / visualized with a resolution of 8 bits, i.e., in a value range of 0..255 in the spectral image 3.

[0088] Using a mapping function MF (I), a false color representation FF(x,y) of the original spectral image 3a can be generated in the form of a false color spectral image 3b. As in Fig. As can be seen in Figure 13, different intensity levels are visualized in five different false colors in the false-color spectral image 3b, namely from black, through blue, turquoise, green, and yellow. The yellow image areas Y in the false-color spectral image 3b thus correspond to those image areas in the original spectral image 3a with the highest intensity.

[0089] In the false-color spectral image 3b, there are therefore different local false colors FF(x,y), for example, the blue false color B and the yellow false color Y. If the white-light image 2 is recorded, for example, as an RGB image with three color channels R / G / B, each of the local false colors of the false-color spectral image 3b can be assigned a respective complementary color channel. For example, the blue color channel B of the white-light image is complementary to the yellow false color Y. With respect to the corresponding complementary color channel values B of the white-light image 2, which are complementary to the yellow local false color Y, the method according to the invention now provides that this color channel value B is locally and selectively lowered compared to the two remaining local color channel values R and G of the white-light image 2.However, this lowering does not occur globally as with alpha blending, but rather indirectly depends on the corresponding local intensity value I(x,y) of the spectral image 3a: For example, for the square image area in the center of the spectral image 3a, which has a high local intensity I and is visualized in the false-color spectral image 3b with the yellow false color Y, it is advantageous if the local color channel value B in the corresponding image area of the white light image 2 is specifically lowered. This can be seen from the "minus" sign in diagram (e), which illustrates a component / correction function ΔB(x,y) of a superposition function OF(I), with the help of which the white light image 2 and the spectral image 3 are specifically superimposed.In the schematic illustration (e), the complementary color channel value B is clearly deliberately lowered in the central square image area, which corresponds to the central area of the original spectral image 3 illustrated in the false color Y.

[0090] A look at diagrams (c) and (d) of the Fig. Figure 13 shows that the remaining color channel values R and G of the white light image 2 are actually increased in the central square image area, which is visualized in the false-color spectral image 3b with the yellow false color Y. This is because the colors red and green are not complementary to the false color Y to be visualized at this point (x, y).

[0091] Fig. 5 shows a further prior art example of an overlay of a white light image 2 with a spectral image 3 in the form of a fluorescence image 4, which is to be visualized as a false-color spectral image 3b using a false-color representation. Image 3b comprises the false colors black, blue, and yellow. The black image areas correspond to low-intensity regions, the blue image areas to medium intensity, and the yellow image areas to high-intensity fluorescent light. In a superposition 27 of the two images 2, 3b, which is common in the prior art, it can happen that additional clipping 10 occurs in image areas (see, for example, detail B) in which a (here: yellow) false color is to be visualized in the overlay image 1. This then results in these image areas no longer being visualized in the correct false color, but in white, as can be seen in detail D of the overlay image 1.At this point, an undesirable color shift 28b has occurred with respect to the false yellow color to be displayed at this point.

[0092] In addition, one can also see in the overlay image 1 of the Fig. 5 shows a further color shift 28a in an image area where a blue false color is actually intended to be visualized. However, in the corresponding white light image 2, the image area there is colored red. Therefore, the addition of the blue false color and the red color from the white light image 2 results in a violet color, which, however, is not included in the false color scale used, from black through blue to yellow. Accordingly, an operator would have difficulty assigning an intensity of the fluorescent light to this image area of the overlay image 1 based on the false color scale.

[0093] The Fig. Figure 6 illustrates how this problem of color shift of a false color can be avoided with the inventive approach: In the left-hand image, one can see a white light image 2, which shows a tissue surface with blood vessels that are visualized in red. The white light image 2 thus uses true colors and gives the surgeon a realistic impression of the observed surgical scene. Furthermore, a false color scale can be seen as a color bar, which continuously ranges from black through blue, turquoise, dark green, light green to yellow and which (comparable to the false color spectral image 3b of the Fig. 5) can be used to visualize the intensity of a spectral image 3, in particular a fluorescence image 4. The overlay image 1 produced by the method according to the invention, which is shown in the right part of the Fig. 6, now shows the original white light image 2 as a colored background image, but at the same time also superimposes a false color representation of a fluorescent light signal according to the original spectral image 3. Unlike in the example of the Fig. 5, no relevant color shift 28 occurs anymore, but as detail D shows, the image areas with high fluorescence signal intensity are visualized in the correct yellow false color. This is achieved by deliberately lowering the blue color channel value B of the left white light image 2 in these image areas. At the same time, the non-complementary red and green color channel values R and G are slightly raised in these image areas, so that despite the lowering of the blue color channel value, the brightness of the pixel in the white light image 2 is essentially maintained. As a result, in particular the local brightness gradient and thus the contrast of the white light image 2 can be retained in the overlay image 1. Furthermore, unlike with alpha blending according to the Fig. 3, no global loss of brightness is observed in the overlay image 1.

[0094] The Fig. Figure 7 shows an example of a possible mapping function MF(I), with which different intensity values I(x,y) of an original spectral image 3 can be visualized using a green false color G. The mapping function thus assigns a corresponding color channel value G(I) to each intensity value on the value scale from 0..255.

[0095] The Fig. 10 illustrates how a false-color spectral image 3b can be obtained in this way by visualizing the respective original intensity I(x,y) of the original spectral image 3 using only a single green false color. In this example, it can be seen that when applying the method according to the invention, no significant color shift can be observed in the overlay image 1 in the image regions in which there is a significant intensity in the spectral image 3. However, as detail A shows, it can still happen that a certain color shift of the image signal components of the original white light image 2 can be observed in other image regions. However, this is less critical for the application, provided the operator can still recognize a valid false-color representation of the spectral image 3 in the overlay image 1.

[0096] Fig. Figure 11 illustrates a possible color space, the HSV color space, in which the false color representation can be encoded. In the example of the Fig. 10, one can imagine, for example, that the only green false color is retained in terms of its color value H, but that either the color brightness 33 (radial axis) and / or the color saturation S 32 (vertical axis) are changed in order to visualize different intensity levels with the green false color. As the thin black arrow in Fig. 11 with reference to the mapping function MF a (I) suggests, the false color representation of the Fig. 10 the H-value of the green false color is kept constant and only the color brightness 33 (in radial direction in the color space) is varied.

[0097] In the example of Fig. 6, however, different false colors from black to yellow are used to visualize the spectral image 3. With regard to the Fig. 12, which illustrates a top view of the HSV color space, this can be understood as defining a continuous path in the color space, with different color values H being assigned depending on the intensity value. Accordingly, the mapping function MF(I) can then comprise several components, for example, R(I), B(I) and G(I). In the example of Fig. 6, the corresponding mapping function would initially increase the blue color channel value B at low intensity values, but lower it again at higher intensity values. Since the color yellow is visualized by red and green color channel values, the mapping function would therefore assign correspondingly high color channel values in the green and red color channels of the false-color spectral image 3b to high signal values.

[0098] In the method according to the invention, the overlay image 1 is preferably calculated using an overlay function OF(x,y). This overlay function thus determines, in a spatially resolved manner, for each pixel of the overlay image 1, the amount by which the respective local color channel value of the white light image 2 is to be raised or amplified, wherein the overlay function OF(x,y) at least indirectly takes into account the respective local intensity value I(x,y) of the original spectral image. In a mapping function, as in Fig. 7, which in a false color spectral image 3b as in Fig. 10, the superposition function OF(I(x,y)) can be written as in Fig. 8. For low intensity values, the color channel values R / G / B of the white light image 2 are kept practically unchanged, because the respective amount ΔG / ΔR / ΔB by which the respective color channel value is raised or lowered is very small for low intensities I. At a high intensity, for example with a value of 250, the amount ΔG(I) by which the green color channel value G of the white light image 2 is to be raised will be correspondingly high, while the two remaining color channel values R and B, which are then complementary to the green false color to be visualized, are just lowered, which can be seen from the large negative values of the factors ΔR and ΔB for high intensity values I in the diagram of the Fig. 8 can understand.

[0099] The Fig. 9 shows a superposition function for the case of the application example of the Fig. 6, where the spectral image 3 is visualized using the continuous false color scale shown, from dark blue to yellow. At low intensity values, a blue false color is visualized, while at high intensity values, a yellow one is visualized. Accordingly, the overlay function of the Fig. 9 at low intensity values of I < 100, the blue color channel value B of the white light image 2 is increased accordingly (ΔB > 0). At high intensity values, however, a yellow false color should be visualized according to the false color scale. Accordingly, the complementary blue color channel value B of the white light image 2 is lowered locally, namely depending on the local intensity value I(x,y) of the original spectral image 3, as can be seen in Fig. 9 by the negative value of the quantity ΔB for high intensity values I. The remaining color channel values G and R of the white light image 2, however, are increased at high intensity values; the values for ΔG and ΔR in the diagram of the Fig. 9 out.

[0100] A possible implementation of a mapping from the original intensity distribution I(x,y) of the original spectral image 3a to a false color representation FF(x,y) in a false color spectral image 3b, as in Fig. 13, can be seen from the Fig. 4: At the bottom left of the image, one can see how the original spectral image 3a is converted into a false color spectral image 3b, visualized by three color channel values RGB, using the mapping function MF(I(x, y)). The method according to the invention can now be implemented according to Fig. 4 can be realized at the signal processing level by determining, depending on the respective local color channel value (R FF / G FF / B FF ) of the false color spectral image 3b, which all depend directly on the original corresponding local intensity value I(x,y) of the original spectral image 3a due to the mapping function, with different weightings, namely according to the weighting factors (b i , r i and g i ) are calculated with the color channel values (R / G / B) of the white light image 2 in order to finally obtain corresponding color channel values R, G, B of the overlay image 1. This weighting / calculation is therefore carried out pixel by pixel at each point x, y of the overlay image, whereby the respective weighting at each pixel is based on the respective color signal values R FF , G FF and B FFof the false color spectral image 3b, ie in indirect dependence on the original corresponding intensity value I(x,y) of the original spectral image 3a. Such a spatial change of the weighting is shown in the three lower diagrams (c), (d) and (e) of the Fig. 13 is an example of the three color channels R / G / B of the white light image 2.

[0101] Fig. Figure 14 illustrates the application of the method according to the invention to the case of Fig. 10, i.e. the visualization of a spectral image 3 with only a single green false color G FF : Here, the false color spectral image 3b contains only a single green color value G FF , which can be done using the mapping function of the Fig. 7 can be determined locally in direct dependence on the respective local intensity value I(x,y). For this green false color, the green color channel value G of the white light image 2 is non-complementary, while the red color channel R and the blue color channel B of the white light image 2 are exactly complementary. The increase or decrease of the respective complementary color component in the RGB color space corresponds to a change in the saturation of the overlay color or a color shift towards the saturated overlay color, which is used to display the fluorescence signal in the form of an image overlay.

[0102] Accordingly, one recognizes in Fig. 14, that with the help of the weighting factors g1 <0, g2 >0 and g3 < 0 the two complementary color channel values R and B of the white light image 2 are spatially resolved and selectively reduced compared to the non-complementary color channel value G (but only if a significant color value G FFis to be visualized at this point), while the non-complementary color channel value G of the white light image 2 is accentuated if a green false color is to be visualized at this point in the overlay image 1. This measure can therefore ensure that the green false color is visualized in the overlay image 1 without any relevant color shift, even for pixels of the white light image 2 that would actually illustrate a complementary mixed color (for example, an orange tone or a violet tone).

[0103] How to Fig. 14 also recognizes, here to generate the overlay image 1, a respective ratio is determined pixel by pixel, with which the respective local color channel value R / G / B of the white light image 2 is compared with the corresponding local signal value G FFof the spectral image 3b is to be calculated, is chosen location-dependent, namely in indirect dependence on the original local intensity value I(x,y) of the respective pixel (x,y) of the original spectral image 3a. This allows the respective weighting between the local color channel values R / G / B of the white light image 2 and the respective local signal value G FF of the spectral image 3b can be changed location-dependently, depending on the local original intensity value of the spectral image 3a.

[0104] Fig. Figure 15 shows a possible implementation of the method according to the invention at the signal processing level for the case of Fig. 6, i.e., the visualization of a spectral image 3 using a continuous false color scale from dark blue to yellow: Here, the false color spectral image 3b, which can be obtained from the original spectral image 3a using a mapping function, comprises three different color channel values R, G, B, whose individual values vary depending on the local intensity value I (x,y). Using the weighting factors a i First, a certain color transformation can be applied to the white light image 2. The same applies to the false color spectral image 3b, which is weighted using the weighting factors s i can be color transformed. Using the additional weighting factors b i , r i and g i The color channel values R / G / B of the false color spectral image 3b can now be calculated with the color channel values R / G / B of the white light image 2 in order to calculate the overlay image 1.

[0105] Here again, it can be seen that, for example, if the false color spectral image 3b is to locally visualize a high signal value in a red false color R, this will have a corresponding effect on the three color channel values R / G / B of the white light image 2 via the weighting factors r1 > 0, r2 < 0 and r3 < 0. As a result, the red color channel value R of the white light image 2 is accentuated accordingly, while the remaining color channel values G and B of the white light image 2 are correspondingly attenuated. Fig. 15, an overlay of the two images 2, 3 can be achieved, in which the original false color representation of the spectral image 3b can be reproduced almost unchanged in the overlay image 1, which ensures that the operator viewing the overlay image 1 can reliably infer the original intensity I(x,y) of the original spectral image 3a based on the colors of the overlay image 1.

[0106] The totality of Fig. 15 illustrated weighting factors r i , b i , g iin conjunction with the intensity-dependent color channel values R(I) / G(I) / B(I) of the false color spectral image 3b can thus be understood as an overlay function OF(x,y) in the sense of the invention, because the totality of the weighting factors specifies the amount by which the respective local color channel value R(x,y), G(x,y), B(x,y) of the white light image 2 is to be raised or lowered when calculating the overlay image 1, namely depending on the sign of the respective weighting factor and depending on the color channel values R(I) / G(I) / B(I) of the false color spectral image 3b. In the example of Fig. 15 thus define the weighting factors r i , b i , g iin each case, it is determined which color channel values of the white light image 2 are to be attenuated as complementary color channel values compared to a specific false color (defined by the three color channel values R, G, B of the false color spectral image 3b) and which of these color channel values of the white light image 2 are to be amplified as non-complementary color channel values.

[0107] In summary, a method and an associated image recording device 5 are proposed for generating realistic and color-true overlay images 1 with at least one false color FF(x,y) used to visualize a spectral image 3a, 3b. These methods make it possible to avoid undesired color shifts of such a false color in the overlay image 1 and, despite the overlay, to maintain a high image brightness of a white light image 2 illustrated as the image background as well as the original image contrast of the white light image 2 as far as possible in the overlay image 1. For this purpose, the invention provides that at least one color channel value of the white light image 2, which is complementary to the respective locally displayed false color, is locally specifically lowered, and the higher the intensity I(x,y) of the spectral image 3a, which is also to be visualized in the overlay image 1, is at this point (x,y) (cf. Fig.6). List of reference symbols 1 overlay image 2 White light image (WLI) 3 spectral image (spectral image = SI) 4 Fluorescence image (fluorescence image = FI) 5 Image acquisition device 6 image sensor 7 Image signal processing unit 8 Visualization system 9 original clipping image areas (in 2 - “original clipping”) 10 additional clipping image area (in 1 - “additional clipping”) 11 Illumination light source 12 Excitation light source 13 Display unit 14 image signals (out of 6) 15 video camera 16 beam splitters 17 Imaging optics 18 objects 19 Illumination light 20 fluorescent light 21 False color 22 reduced image brightness / loss of image contrast 23 optical reflection (on wet fabric) 24 False color calculation unit 25 color channel values (e.g. in 8-bit resolution with value range from 0 to 255; e.g. from 2 / 3 / 1) 26 Displayable / usable value range (for 25) 27 Superposition / Overlay (of 2 and 3) 28 (False) color shift (color drift) 29 False color scale 30 Value range for intensity I(x,y) (of 3) 31 Color value (Hue, H) 32 Color saturation (saturation, S) 33 Color brightness (value, V) 34 Displayable color space 35 False color mapping

Claims

[1] Method for generating an overlay image (1) - wherein the overlay image (1) jointly visualizes image signal components of a white light image (2) (WLI) and image signal components of a spectral image (3a), in particular a fluorescence image (4), - wherein local intensity values I(x,y) of the spectral image (3a) are visualized in at least one, in particular in respective, local false color(s) FF(x,y) in the overlay image (1), - wherein the white light image (2) is visualized by means of grayscale or by means of true colors in the overlay image (1), and - whereby, to generate the overlay image (1), local colour channel values (R / G / B) of the white light image (2) are offset against associated local signal values (R / G / B or GW) of the spectral image (3a, 3b), characterized by , (i) that at least one complementary color channel value (B) of the white light image (2) which is complementary to the at least one, in particular complementary to a respective, local false color FF(x i ,y i ) is locally and selectively reduced compared to the remaining local color channel values (R,G) of the white light image (2), not globally, but in direct or at least indirect dependence on an associated local intensity value I(x,y) of the spectral image (3a) and / or (ii) that, to generate the overlay image (1), preferably pixel by pixel, a respective ratio with which respective local color channel values (R / G / B) of the white light image (2) are offset against associated respective local signal values (R / G / B) of the spectral image (3a, 3b) is selected location-dependently and in direct or indirect dependence on a respective original local intensity value I(x,y) of a respective pixel (x,y) of the spectral image (3a), in particular such that a respective weighting between the local color channel values (R / G / B) of the white light image (2) and the local signal values of the spectral image (3a, 3b) changes location-dependently, depending on the local original intensity value I(x,y) of the spectral image (3a), in particular pixel by pixel, and / or (iii) that to generate the overlay image (1) a respective ratio is selected depending on the location, in particular pixel by pixel, with which local color channel values (R / G / B) of the white light image (2) are offset against associated local signal values (R / G / B) of the spectral image (3a, 3b) and that the respective ratio is calculated with an overlay function OF(I)=f(I(x,y)) (overlay function) which, in direct or indirect dependence on a respective local intensity value I(x,y) of the original spectral image (3a), specifies the amount by which the respective local color channel value of the white light image (2) is raised / amplified / accentuated or lowered / attenuated / reduced during the offset to generate the overlay image (1). [2] Method according to claim 1, - wherein the remaining color channel values (R,G) of the white light image (2) are locally and selectively maintained or even increased in direct or indirect dependence on the local intensity value I(x,y) of the spectral image (3a) and / or - wherein the at least one complementary color channel value (B) of the white light image (2) is reduced more strongly depending on the location, the higher the respective local intensity value I(x,y) of the spectral image (3a) is and / or - wherein the at least one complementary color channel value (B) of the white light image (2) is reduced to different degrees, in particular not at all, at different locations (x,y) of the overlay image (1), depending on the associated local intensity value I(x,y) of the spectral image (3a). [3] Method according to claim 1, - where the superposition function OF(I) defines - which color channel values (R / G / B) of the white light image (2) are compared to a specific local false color FF(x i yi) as complementary color channel values (B) attenuated and / or - which of these color channel values (R / G / B) of the white light image (2) are amplified as non-complementary color channel values (R,G), - preferably in order to also display the local false color FF(x i ,y i ) without significant color shift. [4] Method according to one of the preceding claims 1 to 3, wherein, in particular as an intermediate step in the calculation of the overlay image (1), a false color representation FF(x,y) of the spectral image (3a) in the form of a false color spectral image (3b) is generated from the original spectral image (3a) by means of a mapping function MF(I), which is at least partially visualized in the overlay image (1), - preferably wherein the mapping function MF(I) assigns a specific local false color FF(H,S,V) to a respective local intensity value I(x,y) of the original spectral image (3a) and / or - in particular, wherein the mapping function MF(I) maps into a false color subspace, in particular the - only a single false color value H, but for example different color saturation values S and / or different color brightness values H or - comprises several different false color values H, preferably spanning a continuous path within the false color subspace. [5] Method according to claim 4, wherein the false color representation FF(x,y) visualizes the false colors blue, turquoise, green and yellow and / or - where the mapping function MF(x,y) describes a continuous false color gradient in the false color subspace from blue to turquoise to green to yellow, - particularly preferably, high intensity values I(x,y) of the spectral image (3) are assigned to a yellow false color and low intensity values I(x,y) of the spectral image (3) are assigned to a blue false color, - in particular, the false color spectral image FF(x,y) also includes black as a false color. [6] Method according to claim 4, wherein the false color representation FF(x,y) visualizes the false color green and preferably also the false color black and / or - wherein the mapping function MF(x,y) describes a continuous brightness and / or saturation gradient in the false color subspace for at least one false color, in particular for a false color gradient from black to green, - preferably wherein high intensity values I(x,y) of the spectral image (3) are assigned to a green false color and low intensity values I(x,y) of the spectral image (3) are assigned to a black false color. [7] Method according to one of the preceding claims, - wherein, in particular in the aforementioned location-dependent calculation, local color channel values (R,G,B) of the white light image (2) are calculated with respective false color signal values (R,G,B) of the false color spectral image (3b) in each case with different local weighting, depending on the local original intensity value I(x,y) of the spectral image (3a). [8] Method according to one of claims 4 to 7, - where a respective local false color FF(x i ,y i ) of the false colour spectral image (3b) defines a respective complementary colour KF(x,y) and wherein at least one colour channel value of the white light image (2) corresponding to the complementary colour KF(x,y) is lowered in each case in a location-dependent manner, in particular pixel-by-pixel, - in particular in order to avoid a color shift (28) of the respective local false color FF(x i ,y i) in the resulting overlay image (1). [9] Method according to one of the preceding claims, wherein the overlay image (1) is based on a, - especially predefined, - an overlay function OF(x,y) is calculated, which is preferably customizable by a user and which, in direct or at least indirect dependence on a respective local intensity value I(x,y) of the original spectral image (3a), specifies by what amount the respective local color channel value (R / G / B) of the white light image (2) is raised / amplified / accentuated or lowered / attenuated / reduced, - preferably wherein a local color channel value of the white light image (2) is raised / amplified / accentuated if it corresponds to an associated local false color FF(xy) which is to be / is visualized as an image signal component of the spectral image (3a) at this point (x,y) in the overlay image (1) and / or - preferably wherein a local color channel value of the white light image (2) is lowered / attenuated / reduced if it is complementary to an associated local false color FF(xy) which is to be / is visualized as an image signal component of the spectral image (3a) at this point (x,y) in the overlay image (1). [10] Method according to one of the preceding claims, wherein the white light image (2) and the spectral image (3) / fluorescence image (4) are both sensor-captured with a single image recording device (5), in particular wherein the image recording device (5) - in the form of an endoscope, an exoscope or a microscope and / or - is designed according to claim 17. [11] Method according to one of the preceding claims, wherein the white light image (2) and the spectral image (3) / the fluorescence image (4) i) are spatially separated from each other by sensors, in particular - by means of at least two separate image sensors (6a, 6b) and / or - by means of different colour filters at the pixel level of an image sensor, in particular an RGBX sensor or a hyperspectral sensor, or ii) are recorded by sensors at different times, in particular - by means of time-varying lighting and / or - wherein the two images (2, 3 / 4) are captured alternately in time with an image sensor (6), or iii) are separated from each other only by intelligent signal processing, in particular whereby both images (2, 3 / 4) are recorded simultaneously and / or by only a single image sensor. [12] Method according to one of the preceding claims, wherein portions of the spectral image (3) in the overlay image (1) are visualized in a false color representation FF(x,y) and / or - wherein portions of the white light image (3) in the overlay image (1) are - a colored white light background image, preferably in true color, or as - a monochrome grayscale background image can be visualized. [13] Method according to one of the preceding claims, wherein the white light image (2) is visualized as a grayscale image or - wherein the white light image (2) comprises color information from at least two different color channels and / or - wherein the original spectral image (3) visualizes sensor-detected intensity values I(x,y) of a spectral light signal, in particular a fluorescence signal. [14] Method according to one of the preceding claims, wherein, during the calculation of the overlay image (1), a check is carried out in a spatially resolved manner, in particular pixel-by-pixel, as to whether a value range (26) available and / or maximally representable in the overlay image (1) has been exceeded by superimposing the image signal components of the white light image (2) with the image signal components of the spectral image (3a), and wherein, if this is the case, the image signal components of the white light image (2), in particular the at least one complementary color channel value (B) of the white light image (2), is / are reduced to such an extent that the value range (25) is locally maintained, - in particular so that additional clipping (10) in the overlay image (1) is avoided, - preferably wherein original clipping image areas (9) of the white light image (2), which already exceed the displayable value range, in particular in the form of white image spots, continue to be displayed in the overlay image (1), but in particular in the correct false color, provided that the spectral image (3) has a significant intensity I(x,y) corresponding to the respective clipping image area (9). [15] Method according to one of the preceding claims, wherein, during the calculation of the overlay image (1), the different colour channel values (R / G / B) of the white light image (2) are each raised and / or lowered in a spatially resolved manner, in particular pixel-by-pixel, in direct or indirect dependence on the local intensity value I(x,y) of the spectral image (3a) in such a way that a local overall brightness, - in particular which is formed by the sum of the colour channel values (R / G / B) of all colour channels of the white light image (2), is retained at least approximately, in particular with a relative brightness loss of less than 25%, preferably of less than 10%, - in particular such that no significant image brightness of the white light image (2) is lost in the overlay image (1) and / or that a brightness contrast of the white light image (2) can be largely maintained locally despite the overlay in the overlay image (1). [16] Method according to one of the preceding claims, wherein a local intensity threshold I0 is defined and a reduction of at least one color channel value (B) of the white light image (2) is only carried out locally if the intensity threshold I0 in the spectral image (3) is exceeded at this point (x,y), that is to say if: I(x i, y i ) > I0. [17] Image recording device (5), in particular in the form of a medical visualization system (8), for example designed as an endoscope, exoscope or microscope, with - at least one image sensor (6) which is designed to sensorically detect a white light image (2) and a spectral image (3), in particular a fluorescence image (4), and with - an image signal processing unit (7) which is designed to calculate an overlay image (1) from a white light image (2) recorded with the at least one image sensor (6) and a spectral image (3), in particular a fluorescence image (4), recorded with the at least one image sensor (6), characterized by , - that the image signal processing unit (7) is configured to implement a method according to one of the preceding claims 1 to 16 when the image signal processing unit (7) calculates the overlay image (1).

Images