Image processor and computer-implemented method for medical viewing device using location-dependent color transfer function

Abstract

An image processor (170) and a computer-implemented method for a medical viewing device (100), such as a microscope or endoscope, are described. To improve color accuracy, an image processor (170) includes a color conversion function (140). Further, the image processor (170) is configured to retrieve input pixels (150d) of the digital input color image (200) and locations (x, y) of the input pixels in the input color image (200), and to apply a color conversion function to the input pixels to generate output pixels (150c) in the digital output color image (160). The color conversion function (140) depends on the position of the input pixel. Using this solution it is possible to compensate for non-uniformities in the spatial color distribution in the digital input color image, introduced by the optical element between the imaging object (106) and the image sensor and / or by the illumination system.

Description

Technical Field

[0001] The present invention relates to an image processor and a computer implemented method for a medical viewing device, such as a microscope or an endoscope. Background Art

[0002] Currently, medical viewing devices use digital color cameras and digital color images obtained therefrom to present information to a user, such as a surgeon or scientist. In order to present accurate visual information to the user, it is important that the color image represents the object as realistically as possible. Therefore, digital color images are processed in an image processing pipeline using several processing steps in order to improve the visualization of the image content.

[0003] There is a need to improve the accuracy of the representation of objects in recorded color images. Summary of the invention

[0004] This need is met by an image processor for a medical viewing device such as a microscope or an endoscope, wherein the image processor includes a color conversion function; wherein the image processor is configured to retrieve an input pixel of a digital input color image and a position of the input pixel in the input color image, and apply the color conversion function to the input pixel to generate an output pixel in a digital output color image; wherein the color conversion function depends on the position of the input pixel.

[0005] This need is also met by a computer-implemented method for a medical viewing device such as a microscope or an endoscope, wherein the computer-implemented image processing method comprises the steps of: retrieving an input pixel of a digital input color image; retrieving a position of the input pixel in the digital input color image; and applying a color conversion function to the input pixel to generate an output pixel in a digital output color image; wherein the color conversion function depends on the position of the input pixel.

[0006] Typically, color correction is used for color mapping, i.e., the colors of a digital input color image are mapped to different colors in a digital output color image. For example, colors recorded in the digital input color image can be mapped to colors that appear more natural in the digital output color image. Furthermore, it is known to adjust the colors of an image to compensate for the characteristics of a display device, which is done, for example, in gamma correction. These known color corrections are applied globally. Thus, the same color correction is applied to each pixel of the entire digital input color image.

[0007] However, defects in the quality and / or arrangement of the optical components of the medical viewing device between the object and the color camera sensor may produce a spatially non-uniform color distribution in the digital input color image. Thus, even if the imaged object is a constant color, the color recorded in the digital input image may vary in the image.

[0008] The uneven spatial color distribution in the recorded color image can additionally or alternatively be caused by the illumination system illuminating the object. Imperfections in the optical elements directing the illumination light onto the object and / or imperfections in the light generation itself can also lead to a spatially uneven color distribution in the digital input color image.

[0009] By having a color transfer function that depends on the position of the input pixel, i.e. a function of position, a spatially non-uniform color distribution can be compensated and made more uniform: the color of the pixel is adjusted according to its position. Thus, the color of the imaged object is represented more accurately.

[0010] In the following, additional features are described that further improve the invention. Each of the following features is advantageous in itself and can be independently combined with any of the other features below. Furthermore, each of the following features can be used independently to improve the image processor and / or the computer-implemented method, even if the corresponding feature is only mentioned in the context of the image processor or only in the context of the computer-implemented method. More specifically, the image processor can be configured to perform each process step described below, even if the image processor is not specifically mentioned in conjunction with that process step.

[0011] For example, the color conversion function may be part of an imaging processing pipeline comprising at least one processing step of the group consisting of demosaicing, registration, contrast enhancement, brightness homogenization, color calibration and / or color mapping. Contrast enhancement, brightness homogenization, color calibration and / or color mapping may use a position-independent color correction function. Preferably, the position-dependent color conversion function is applied after registration and before applying the position-independent color correction function.

[0012] In particular, the color conversion function may be a spatial color distribution homogenization function. After applying the color conversion function, the color gradient over a specific region of the digital input color image may be greater than the color gradient over the specific region in the digital output color image, depending on the position of the input pixel. The specific region may extend over a plurality of pixels, for example, spanning a quarter, a half, or the entirety of the digital input color image.

[0013] In one embodiment, the digital color input image may represent an object of spatially constant color, which preferably has been illuminated using a standard light source. Color gradients may be determined in such a digital input color image and compared with the color gradients of a digital output color image obtained after applying the color conversion function to the digital input color image.

[0014] In one embodiment, for each color space coordinate of an input pixel, the color conversion function may simply represent the inverse of a color transfer function that generates a spatial color distribution in the digital output image for this color space coordinate. More specifically, the color transfer function may map the color space coordinate at the location of the input pixel to the color space coordinate at the location of the output pixel. The color transfer function may represent an imperfection in the optical system that introduces spatial non-uniformities that are corrected by the color conversion function. In one embodiment, by taking the inverse of the color transfer function, the non-uniformities may be compensated. Thus, the color conversion function may be considered an inverse spatial color filter, where the color filtering properties depend on the location.

[0015] In addition to being dependent on the position of the input pixel, the color conversion function may also be dependent on at least one optical parameter of a group comprising an optical parameter representing a working distance at which the digital input color image is recorded; an optical parameter representing an aperture at which the digital input color image is recorded; and an optical parameter representing a magnification at which the digital input color image is recorded. Any of these optical parameters may be changed, for example, during operation of the medical viewing device or by a user. Thus, if any of these optical parameters are changed during operation of the medical viewing device, the uniformity of the spatial color distribution in the digital input color image may change. Making the color conversion function dependent on these changing optical parameters allows compensating for any color inhomogeneities introduced during operation of the medical viewing device.

[0016] Furthermore, the set of optical parameters on which the color transfer function depends may also comprise optical parameters representing the spatial color distribution of the illumination system in the object plane.The illumination system may be part of a medical viewing device.

[0017] The illumination system may include a first illumination mode and a second illumination mode, wherein in the first illumination mode, at least one of an illumination spectrum and an intensity may vary in an object plane. The illumination system may be configured to generate a first illumination spectrum in the first illumination mode and a second illumination spectrum in the second illumination mode, the first illumination spectrum being different from the second illumination spectrum.

[0018] The image processor is preferentially configured to retrieve at least one optical parameter, for example by retrieving at least one optical parameter from a controller of the medical viewing device.

[0019] The spatial color distribution homogenization function, or color transfer function used synonymously herein, can be determined by experimental calibration. For example, various samples, for example uniformly illuminated by a standard light source, can be recorded. Each of the various samples can be a different predetermined, spatially constant color. Such samples can be, for example, a color card or an illuminated and calibrated screen.

[0020] By comparing the predetermined constant color with the recorded color, the spatial color distribution caused by the optics between the object and the image sensor can be determined. For example, by recording images of a gray card under various lighting modes provided by a color-adjustable lighting system, the spatial color distribution caused by non-uniform lighting can be determined.

[0021] The input pixel may include color space coordinates, which represent the color of the input pixel in the input color space. Such an input color space may be, for example, a non-uniform color space, such as an RGB color space, a YCbCr (Y represents a brightness signal, Cb represents a blue chromaticity component, and Cr represents a red chromaticity component) color space, a YUV (Y represents brightness, and U and V represent chromaticity) color space; a three-color color space, such as a CIE 1931XYZ (one of the color spaces created by the International Commission on Illumination (CIE) in 1931 to describe colors by three numerical values ​​X, Y, and Z) color space, a CIEUVW color space (a color space based on chromaticity U, V and lightness index W proposed by the International Commission on Illumination (CIE); or a color space such as a CIELUV (a color space proposed by the International Commission on Illumination (CIE) to describe colors by L * 、u * 、v * three coordinates to represent the brightness and chromaticity of a color), CIELab (a color space proposed by the International Commission on Illumination (CIE) using L * 、a * 、b * The color space that uses three coordinates to represent the brightness and chromaticity of a color) and the uniform color space of HSLuv.

[0022] In addition, the output pixel may include color space coordinates in an output color space. Preferably, the output color space is the same as the input color space. However, this is not necessary, and the output color space may be different from the input color space. In this case, the color space conversion may be included in the color conversion function.

[0023] In one embodiment, a digital output color image is created in real time from a digital input color image. To this end, it may be helpful if the color conversion function may be a linear conversion function, the color conversion function in particular consisting of or including a color conversion matrix. The color conversion matrix may contain at least one matrix element, at least one matrix element being dependent on the position of the input pixel and / or at least one optical parameter as described above. The use of a matrix makes the color conversion computationally fast and efficient.

[0024] The first dimension of the color conversion matrix may correspond to the number of color space coordinates of the input color space, and the second dimension of the color conversion matrix may correspond to the number of color space coordinates of the output color space.Preferably, the color conversion matrix is ​​square.

[0025] In one embodiment, at least one matrix element depending on the position of the input pixel and / or at least one optical parameter may include at least one of a polynomial function, a spline function, and a multivariate interpolation function. The polynomial function, the spline function, and the multivariate interpolation function may be stored in a memory of the image processor. The polynomial function, the spline function, or the multivariate interpolation function may be determined by a calibration process. The input of the polynomial, the spline, or the multivariate interpolation function is the position of the input pixel and / or at least one optical parameter.

[0026] The positions of the output pixels within the digital output color image are preferably the same as the positions of the input pixels within the digital input color image.Thus, according to a preferred embodiment, the digital input color image and the digital output color image may have the same aspect ratio and the same number of pixels.

[0027] In one embodiment, the input color image may be generated by a combination of two color images, such as a digital white light color image and a digital fluorescent color image. The input pixel may result in a combination of a first pixel from the digital white light color image and a second pixel from the digital fluorescent color image. The combination of the first pixel and the second pixel is preferably the union of their color space coordinates: if the color space coordinates of the first pixel and the second pixel are each considered as a set of coordinates, the combination may simply come from and / or correspond to the union of the two sets.

[0028] For example, the image processor can be configured to retrieve a digital white light color image of an object, the digital white light color image being recorded in a first imaging spectrum, the digital white light color image also comprising a plurality of first pixels, each first pixel comprising a first set of color space coordinates in a color space; to retrieve a digital fluorescent color image of an object being recorded in a second imaging spectrum, the digital fluorescent color image comprising a plurality of second pixels, each second pixel comprising a second set of color space coordinates in a color space; to generate input pixels of a digital input color image from one of the first pixels and one of the second pixels, the input pixels comprising a third set of color space coordinates in a color space; and to generate the third set as a union of the first set and the second set. This allows the spatial color distribution in the digital white light color image and the digital fluorescent color image to be jointly homogenized, and thus allows the color conversion to be a real-time process.

[0029] In particular, if the medical viewing device is a stereoscopic device, each stereoscopic channel may have a digital white light color image and a digital fluorescent color image.

[0030] The use of fluorescence color images is particularly useful in medical applications, where fluorophores are used to mark specific areas of interest. For example, some fluorophores accumulate in tumors, so that the fluorescence of such fluorophores indicates the presence of tumor tissue. Other fluorophores can bind to blood cells and can therefore be used to highlight blood vessels. Accurate color reproduction of fluorescence is particularly important, since the intensity of fluorescence is very low and the fluorescence itself can indicate the state of the tissue containing the fluorescence. Any color variations caused by spatially inhomogeneous optical imaging may lead to misinterpretation of the recorded image.

[0031] The digital white light color image and the digital fluorescent color image are preferably registered images such that the first pixel and the second pixel of the digital fluorescent color image are corresponding pixels. Since in the registered images the image patterns in the registered images have the same size, orientation and position, the corresponding pixels are located at the same image position in their respective images and represent the same area of ​​the object.

[0032] By calculating the input pixel or the entire digital input color image from the digital white light color image and the digital fluorescent color image or from the first pixel and the second pixel, respectively, the input pixel or the digital input color image can be physically formed in the memory of the image processing device, respectively. In an alternative embodiment, the input pixel or the digital input color image can be formed separately by processing the first pixel and the second pixel or the entire digital white light color image and the entire digital fluorescent color image together. In this case, the first pixel and the second pixel are stored separately. Therefore, the input pixel can only exist virtually.

[0033] According to a further improvement, the first imaging spectrum and the second imaging spectrum may be complementary to each other and / or even more preferably non-overlapping. Therefore, the digital white light color image and the digital fluorescent color image contain separate spectral information about the object. This facilitates their common treatment as a union of color space coordinates, because there is no crosstalk or the crosstalk between the color space coordinates is negligible.

[0034] Both the first and second imaging spectra may overlap the visible spectrum, but may also include electromagnetic wavelengths outside the visible spectrum, such as infrared (IR) or near infrared (NIR) wavelengths.

[0035] If the digital white light color image and the digital fluorescent color image are recorded in the RGB color space, for example, each first pixel and each second pixel contains three color space coordinates. The union of the color space coordinates of the first pixel and the second pixel contains six color space coordinates. If the color conversion matrix is ​​used for homogenization of the spatial color distribution, the color conversion matrix has a dimension of 6×6 and the output pixel contains 6 color space coordinates.

[0036] If a color space other than RGB is used, the number of color space coordinates in the first pixel and / or the second pixel and, respectively, the number of color space coordinates in the output pixel may vary.

[0037] Any number of color images can be input to the color conversion function. Any additional images will increase the dimensionality of the matrix and the number of color space coordinates in the input and output pixels. For example, if three RGB color images are input to the color conversion function, the union of the first, second, and third pixel color coordinates is a 9-tuple. Therefore, the color conversion matrix in this case is a 9×9 matrix.

[0038] In one embodiment, a medical viewing device such as a microscope or an endoscope may include an image processor configured to perform any of the processing steps described above. The medical viewing device may further include at least one color camera and may be configured to generate a digital input color image from at least one color image generated by the at least one camera. In another embodiment, the medical viewing device may include at least two color cameras, such as a white light color camera for recording a digital white light color image and a fluorescence color camera for recording a digital fluorescence color image.

[0039] Furthermore, the medical observation device may include an additional color camera that records additional color images in an additional imaging spectrum that may be complementary to the first imaging spectrum and the second imaging spectrum as well as other imaging spectrums. For example, an additional fluorescence color camera may be configured to record the fluorescence spectrum of a fluorophore in a third imaging spectrum that is different from the first imaging spectrum and the second imaging spectrum. The third imaging spectrum may overlap the fluorescence emission spectrum of this fluorophore, which does not overlap the fluorescence emission spectrum covered by the second imaging spectrum.

[0040] The method for operating the medical observation device of any of the above described embodiments may include the computer-implemented method as described above. In addition, the method for operating the medical observation device may include the following steps: recording a digital fluorescent color image in the second imaging spectrum using a fluorescent color camera; recording a digital white light color image in the first imaging spectrum using a white light color camera; forming a digital input color image from a combination of the digital white light color image and the digital fluorescent color image.

[0041] In one embodiment, the digital fluorescence color image may be recorded as a reflectance image of the object, ie a second reflectance image in addition to the digital white light color image. In another embodiment, the digital fluorescence color image may represent the fluorescence emission of at least one fluorophore.

[0042] Furthermore, the digital white light color image may be recorded as a reflectance image of the object. In another embodiment, the digital white light color image may further include the fluorescence emission of at least one fluorophore.

[0043] The claimed subject matter also relates to a computer-readable medium and a computer program comprising instructions for causing a computer to perform the computer-implemented method in any of the above-described embodiments.

[0044] The image processor may be a data processing device.

[0045] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0046] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0047] In the following, the present invention is exemplarily described with reference to the embodiments and the accompanying drawings. The feature combinations shown in the embodiments should not be considered as restrictive. For example, features of the embodiments with the technical effects explained above that are not required in a specific application can be omitted. On the contrary, if the technical effects associated with this specific feature are required in a specific application, the above-mentioned features that are not part of the following embodiments can be added.

[0048] Throughout the description and the drawings, the same reference numerals are used for elements that correspond to each other in terms of function and / or structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In the accompanying drawings,

[0050] Figure 1 A schematic diagram showing an embodiment of a medical observation device;

[0051] Figure 2 A schematic diagram showing a method for spatial color harmony using a color conversion function;

[0052] Figure 3 shows the use of a color conversion matrix Figure 2 A more detailed schematic diagram of the spatial color harmony;

[0053] Figure 4 A schematic diagram showing an example of a color image pipeline with spatial color harmony;

[0054] Figure 5 A schematic diagram of a color adjustable lighting system is shown;

[0055] Figure 6 A schematic diagram showing a general system for performing spatial color harmony; and

[0056] Figure 7 A schematic example of the effect of a color conversion function is shown. DETAILED DESCRIPTION

[0057] First, refer to Figure 1 Exemplary embodiments of the present invention are described.

[0058] Figure 1 A medical observation device 100 is schematically shown. The medical observation device 100 may be a fluorescence microscope or a fluorescence endoscope, the difference between a microscope and an endoscope being that in an endoscope (not shown) the object 106 is observed via an optical fiber which is brought into the vicinity of the object 106 to be investigated, for example by being inserted into the body, whereas in a microscope the objective lens 174 is directed directly onto the object 106. Although Figure 1 The medical observation device 100 is shown as a microscope, but the following description is also applicable to an endoscope. The medical observation device 100 may be, for example, a medical observation device used in a surgical operation. The medical observation device 100 may also be a medical observation device used in a laboratory, such as a laboratory microscope. The object 106 to be studied may consist of or include a biological tissue 107, but may also include or consist of an inorganic substance.

[0059] The object 106, or more specifically the object plane, is located at a working distance 162 from an objective lens 174. The depth of field may be determined by the aperture 164. The magnification 166 of the object 106 may be determined by the focal length of the objective lens 174, which in one embodiment may be a zoom objective lens. The working distance 162, the magnification 166 and / or the aperture 164 are optical parameters that may be changed by a user during operation of the medical viewing device 100. If the optical parameters 162, 164, 166 are changed, the imaging properties of the medical imaging device 100 may change. One of the imaging characteristics that may change is the spatial color distribution, i.e., the variation of colors on the image plane.

[0060] The object 106 may include one or more fluorophores 116, 118. At least one fluorophore 116 may be a fluorophore naturally contained in the object. For example, bones and blood contain fluorophores. At least one fluorophore may also be added to the object 106, for example by applying it to the object 106. For example, if the object includes biological tissue 107, at least one fluorophore 116, 118 may be injected into the object 106. Examples of fluorophores that may be added to the object 106 are indocyanine green (ICG), fluorescein and / or 5-aminolevulinic acid (5-ALA). 5-ALA is synthesized as protoporphyrin IX (pPIX) in cells.

[0061] The medical viewing device 100 may be configured to record the fluorescence of one or more fluorophores 116, 118. This means that the medical viewing device 100 may be configured to observe, record and preferably also excite the fluorescence of one or more fluorophores 116, 118.

[0062] The medical observation device 100 may be Figure 1 101R for each of the two stereo channels. Since the two subassemblies 101L, 101R are identical in function and structure, the following description focuses on the right subassembly 101R, but is equally applicable to the left stereo channel 101L. Each subassembly 101L, 101R may be monoscopic.

[0063] The medical observation apparatus 100 may also be a monoscopic device. In this case, only one of the two subassemblies 101L, 101R may be present. Therefore, for the monoscopic medical observation apparatus 100, the following description is also applicable.

[0064] In one embodiment, the medical viewing device 100 can be used to generate one or more digital white light color images 114. The digital white light color image 114 can specifically represent a reflection image of the object 106 in the visible light range. The visible light range or visible spectrum includes wavelengths from about 310 nanometers to about 1100 nanometers, or from about 380 nanometers to 750 nanometers, or from about 450 nanometers to about 700 nanometers.

[0065] The fluorescence spectrum, or if more than one fluorophore is used, the fluorescence spectrum of the fluorophores 116, 118 is preferably omitted from the spectrum recorded in the digital white light color image 114. This ensures that if fluorescence is present, only reflected light is included in the digital white light color image 114. For example, in one embodiment, if 5-ALA / pPIX is used as the fluorophore, the fluorescence spectrum from about 625 nanometers to about 650 nanometers is not recorded in the digital white light color image 114.

[0066] If specific wavelengths of light are used to excite fluorescence, spectra consisting of or containing these wavelengths are preferably not recorded or represented in the digital white light color image 114. For example, if 5-ALA / pPIX is used as the fluorophore, fluorescence can be excited by illuminating the object 106 with a wavelength between about 380 nanometers and about 450 nanometers. These wavelengths are preferably not combined in the digital white light color image 114. For fluorescein and ICG, as well as other fluorophores, different but known spectral ranges for excitation and emission apply to 5-ALA / pPIX.

[0067] When the fluorescence excitation spectrum and the fluorescence emission spectrum are not recorded, as they would be seen by a human observer, the digital white light color image 114 represents the reflectance of the object 106, i.e., a white light image of the object 106. In such a configuration, the digital white light color image 114 may be considered a natural or true color image.

[0068] The digital imaging system 102 may further be used to record one or more digital fluorescence color images 112 of the object 106. The digital fluorescence color image 112 may represent or include fluorescence emissions of one or more fluorophores 116, 118. The digital fluorescence color image 112 may be recorded simultaneously with the white light color image 114.

[0069] The digital fluorescence color image 112 preferably does not record the emission spectrum of one or more fluorophores or any wavelengths outside of the emission spectrum. The digital fluorescence color image 112 may also be a reflectance image of the object 106 or include both fluorescence colors and reflected light from the object 106.

[0070] The digital white light color image 114 and the digital fluorescent color image 112 can be recorded using at least three color bands or equivalently the primary colors of the color space. For example, both the digital white light color image 114 and the digital fluorescent color image 112 can be recorded in the RGB color space using the three primary colors or color bands R (red), G (green), B (blue). Alternatively, the digital white light color image 114 and the digital fluorescent color image 112 can both be recorded in different color spaces. Each of them can represent a multi-spectral or hyper-spectral color image. The digital white light color image 114 and the digital fluorescent image 112 do not need to be recorded in the same color space, although this is preferred.

[0071] At least one digital white light color image 114 and at least one digital fluorescent color image 112 each include pixels 150a, 150b, respectively. Each pixel 150a, 150b includes color space coordinates. In a color space such as an RGB color space, each color of the pixels 150a, 150b is represented by a color space coordinate forming a triple of three integers. Each integer indicates the intensity of one of the primary colors R, G, B. For example, the strongest red can be indicated by a triple {255, 0, 0}. The strongest green can be indicated, for example, by {0, 255, 0}, and the strongest blue can be indicated, for example, by {0, 0, 255}. Therefore, the RGB color space is a three-dimensional space. The CMYK (cyan, magenta, yellow, black) color space will be a four-dimensional space. Colors can be considered as points in a color space with color space coordinates such as {0, 0, 255}. In a more general way, a multispectral or hyperspectral color space with n color bands will correspondingly result in an n-dimensional color space, and each color will be represented by an n-tuple of values.

[0072] Examples of RGB color spaces include RGB, sRGB (standard RGB color space), AdobeRGB (a color space developed by Adobe), Adobe White Gamut RGB, REC.2100 (a high dynamic range video coding standard recommended by the International Telecommunication Union (ITU), DCI-P3 (a color space standard of the Digital Cinema Initiative (DCI)), Rec.3020 (a color space standard for ultra-high definition television (UHDTV)), Rec.709 (a color space standard for high definition television (HDTV)), Rec.601 (a color space standard for standard definition television (SDTV), and ACES (Color Exchange System). However, the images 112, 114 may be recorded, stored, and / or processed in any color space.

[0073] The spectrum recorded and represented in the digital white light color image 114, i.e. the first imaging spectrum, and the spectrum recorded in the digital fluorescent color image 112, i.e. the second imaging spectrum, are preferably complementary to each other. In particular, they do not overlap except for unavoidable filter leakage. Preferably, they together represent the complete spectrum of visible light, or at least a major part of the visible light range. The first and / or second image may also contain wavelengths outside the visible spectrum, such as the near infrared spectrum (NIR).

[0074] More specifically, the medical viewing apparatus 100 may include a digital imaging system 102 for generating a digital fluorescent color image 112 and a digital white light color image 114. The digital imaging system 102 may include a white light color camera 110 and a fluorescent color camera 111. The digital imaging system 102 may include an additional color camera (not shown) for each stereo channel 101L, 101R.

[0075] The white light color camera 110 is configured to record a digital white light color image 114. In particular, the white light color camera 110 may be configured to generate a stream of digital white light color images 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images in the entire visible spectrum in the wavelengths described above. The white light color camera 110 may be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) or a multispectral or hyperspectral camera.

[0076] The fluorescence color camera 111 is configured to record a digital fluorescence color image 112. In particular, the fluorescence color camera 111 may be configured to generate a stream of digital fluorescence color images 112 in the form of a digital video stream. The fluorescence color camera 111 may be configured to record the digital fluorescence color image 112 only in the fluorescence spectrum or fluorescence spectrum of at least one fluorophore 116, 118. The fluorescence color camera 111 may be configured to record the digital fluorescence color image 112 only in one or more narrow bands of light. These narrow bands should overlap with the fluorescence spectrum or spectrum of the one or more fluorophores 116, 118 whose fluorescence is to be recorded. Preferably, the fluorescence spectra of the fluorophore 116 and the second fluorophore 118 do not overlap, so that the fluorescence color camera 111 can record light in two separate spectral fluorescence bands that are spaced apart from each other.

[0077] Fluorescence color camera 111 may be a CCD, CMOS or multi-spectral or hyper-spectral camera, or any other type of suitable camera. Preferably, white light color camera 110 and fluorescence color camera 111 are of the same type, although this is not required. Since fluorescence usually has very low intensity, fluorescence color camera 111 may have a higher integration time than white light color camera 110.

[0078] The respective fields of view 184 of the cameras 110, 111 are preferably aligned or even coincident and coaxial. Therefore, it is preferred that the cameras 110, 111 and any additional color cameras provide the same field of view 184 with the same viewing angle and focal length. This results in the same representation of the object 106 in the images 112, 114 generated by the different cameras 110, 111. The cameras 110, 111 can all use the same objective lens 174. In such a configuration, the images 112, 114 can be optically registered, which makes registration by image processing unnecessary or much easier.

[0079] If a match of the field of view and angle of view cannot be generated optically, a matching or registration image processing routine may be applied to the digital images 112, 114, as explained further below. Applying a computer-implemented registration procedure may even be desirable if the cameras 110, 111 have the same angle of view and field of view.

[0080] Preferably, the two cameras 110 , 111 are operated synchronously. Specifically, the exposure times of the cameras 110 , 111 may be synchronized. Therefore, the medical observation apparatus 100 may be configured to generate the digital white light color image 114 and the digital fluorescent color image 112 simultaneously.

[0081] Preferably, the gain of the two cameras 110, 111 is synchronized, i.e. it is adjusted simultaneously in both cameras 110, 111. Furthermore, the ratio of the gain applied in camera 110 to the gain applied in camera 111 may be constant, even if the gain is changed. Gamma correction and color adjustment or white balance may be turned off or left unchanged.

[0082] Any of the above measures may facilitate comparison, co-processing, and / or combination of the two images 112 , 114 .

[0083] In order to separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescent color image 112, that is, to separate the reflection spectrum from the fluorescent spectrum, the medical viewing device 100 may include an optical color separation component 176. The color separation component 176 may include an optical element, such as a beam splitter 192, which may be dichroic. The color separation component 176 may further or alternatively include an optical white light filter 188 and / or an optical fluorescent color filter 190.

[0084] The fluorescent color filter 190 is preferably configured to transmit light in the fluorescent spectrum or frequency spectrum of the one or more fluorophores 116, 118 and to block light outside of the fluorescent spectrum or frequency spectrum.

[0085] The fluorescence color filter 190 can be configured as a bandpass filter including one or more passbands. Each passband should overlap with the fluorescence emission spectrum of the corresponding fluorophore 116, 118 whose fluorescence is to be recorded. Since the fluorescence color filter 190 is located in the light path between the beam splitter 192 and the fluorescence color camera 111, only wavelengths in the passband of the fluorescence color filter 190 are transmitted to the fluorescence color camera 111.

[0086] The white light filter 188 is preferably configured to block light in the fluorescence spectrum of one or more fluorophores 116, 118. The white light filter 188 may also be configured to block light in the fluorescence excitation spectrum.

[0087] The white light filter 188 is preferably configured as a band stop filter, whose stop band corresponds to or at least includes the pass band of the fluorescence color filter 190. The white light filter 188 is located in the light path between the beam splitter 192 and the white light camera 110. Therefore, the white light camera 110 only records wavelengths outside the stop band of the white light filter 188 and, therefore, also wavelengths outside the pass band of the fluorescence color filter 190.

[0088] Either of the white light filter 188 and the fluorescent color filter 190 may be a tunable filter.

[0089] The digital white light color image 114 and the fluorescent color image 112 preferably have the same aspect ratio and the same number of pixels. In other variations, the aspect ratio and / or number of pixels of the images 114, 112 may be different.

[0090] If the beam splitter 192 is a dichroic beam splitter, at least one of the filters 188, 190 can be omitted, since in this case the optical spectral filtering is already integrated in the dichroic beam splitter. The above description of passband and stopband should apply mutatis mutandis to the dichroic beam splitter 192.

[0091] Thus, white light color camera 110 records digital white light color image 114 in a first imaging spectrum, which may be a reflectance spectrum. The first imaging spectrum may be different from a second imaging spectrum, which is recorded by fluorescence color camera 111 and may include one or more fluorescence emission spectra. The wavelengths included in the first and second imaging spectra are determined by the filter settings of color separation component 176.

[0092] The medical viewing device 100 may also include an illumination assembly 178 configured to illuminate the object 106, preferably through the same objective lens 174, through which the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 may be configured to selectively generate white light, i.e., light uniformly distributed over the entire visible spectrum, and fluorescence excitation light, which contains only light in wavelengths that excite fluorescence of at least one fluorophore 116, 118. The illumination light generated by the illumination assembly 178 may be fed into the objective lens 174 using an illumination beam splitter 180.

[0093] Depending on the fluorophore and its fluorescence specific excitation spectrum, an illumination filter 179 may be provided. For example, if 5-aminolevulinic acid (5-ALA) / protoporphyrin IX (PpIX) is used as the fluorophore, the illumination filter 179 may have a transmittance of 90% to 98% for wavelengths up to 425 nanometers, a transmittance of 0.5% to 0.7% for wavelengths between 450 nanometers and 460 nanometers, a transmittance of no more than 0.1% for wavelengths between 460 nanometers and 535 nanometers, and a transmittance of almost zero for wavelengths above 535 nanometers.

[0094] Instead of or in addition to the illumination filter 179, the illumination assembly 178 may include an adjustable light source including, for example, a plurality of differently colored diodes (LEDs) or organic light emitting diodes (OLEDs).

[0095] The illumination assembly 178 may include a plurality of different illumination modes, wherein each different illumination mode may include a different illumination spectrum and / or a different light intensity distribution for illuminating the object 106. For example, the illumination assembly 178 may be configured to generate fluorescence excitation wavelengths for different fluorophores 116, 118 or different combinations of fluorophores in different illumination modes at intensities different from white light illumination of the object for recording a reflectance image. In an alternative or cumulative example, the illumination assembly 178 may be configured to generate an illumination spectrum having a spectral power distribution corresponding to a standard illuminant (e.g., a set of standard illuminants established by the International Commission on Illumination (CIE)).

[0096] One example of an illumination assembly 178 that may be used with the medical viewing device 100 is described and shown in EP 3878355 A1, which is incorporated herein by reference in its entirety.

[0097] The medical observation device 100 may further include an image processor 170. The image processor 170 may be a hardware module, such as a microprocessor or a software module. The image processor 170 may also be a combination of a hardware module and a software module, such as by using a software module configured to run on a specific processor, such as a vector processor, a floating point graphics processor, a parallel processor, and / or a plurality of processors. The image processor 170 may be part of a general purpose computer 186, such as a personal computer (PC).

[0098] The image processor 170 is configured to retrieve the digital white light color image 114 and the digital fluorescent color image 112. For example, the image processor 170 may be configured to retrieve the digital white light color image 114 and the digital fluorescent color image 112 from the memory 194 and / or directly from the cameras 110, 111. The memory 194 may be part of the image processor 170, or located elsewhere in the medical viewing device 100.

[0099] Image processor 170 is further configured to calculate digital output color image 160 from digital white light color image 114 and digital fluorescent color image 112. Digital output color image 160 is a color image represented in a color space that may be different from the color space of any of digital white light color image 114 and digital fluorescent color image 112. However, preferably, the color space of digital output color image 160 is the same as the color space of digital white light color image 114 and digital fluorescent color image 112. In order to convert coordinates from one color space to another, a normalization function exists or can be determined experimentally or analytically.

[0100] In one embodiment, the digital white light color image 114 can be used as a background color image, and the fluorescent color image 112 is overlaid as a second image on the background color image. In another embodiment, the digital fluorescent color image 112 can be used as a background image, but the second pixel 150b corresponds to a pixel from the digital white light color image 114.

[0101] Image processor 170 may be configured to apply color conversion function 140 to digital white light color image 114 and digital fluorescent color image 112 to generate digital output color image 160. Generation of output color image 160 may occur on a pixel-by-pixel basis, where a first pixel 150a of digital white light color image 114 is combined with a second pixel 150b from digital fluorescent color image 112. Color conversion function 140 is then applied to this combination to form output pixel 150c of digital output color image 160. This process may then be repeated for the remaining pixels.

[0102] The color conversion function 140 is configured to compensate for spatial non-uniformity in the distribution of colors in the images 112, 114. The color conversion function 140 preferably consists of or includes a color conversion matrix 142, which will be referred to below. Figure 3 Describe in more detail.

[0103] The first pixel 150a and the second pixel 150b are preferably corresponding pixels. Therefore, in the registered images 112, 114, the first pixel 150a and the second pixel 150b may be located at the same position within the respective images 112, 114.

[0104] The number of pixels in the images 112, 114 does not necessarily need to be the same. In this case, a pixel in one image 112, 114 may correspond to a number of pixels in the other image 114, 122.

[0105] The image processor 170 may be configured to output the digital output color image 160 to any kind of peripheral device. Output in this case includes transmitting and / or allowing access to the digital output color image 160, for example by allowing access to a memory (e.g., memory 194), where the digital output color image 160 may be stored, or output on an output interface 172 of the medical viewing device 100.

[0106] The digital output color image 160 may be displayed on a display 132 that is integrated into the medical viewing device 100. For example, the display 132 may be integrated into a viewing mirror or eyepiece 104 of the medical viewing device 100. The display 132 may also display a graphical user interface for operating the medical viewing device 100.

[0107] The medical viewing device 100 may include a direct light path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In this case, the display may be an integrated display 132, such as a semi-transparent display located in the direct light path 134 or a display projected into the direct light path 134. A beam splitter 136 may be provided to split the light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed to the eyepiece 104.

[0108] In another embodiment, the medical viewing device 100 need not have a direct optical path 134, but may simply display images using the integrated display 132. As a further alternative, the medical viewing device 100 need not have any integrated display at all.

[0109] The medical viewing device 100 may include an output interface 172, to which one or more (external) displays 182 and / or any type of unidirectional or bidirectional wired or wireless data connection may be connected. To this end, the output interface 172 may include standardized connectors and data transmission protocols, such as wireless local area network (WLAN), transmission control protocol / internet protocol (TCP / IP), Ethernet, universal serial bus (USB), high-definition multimedia interface (HDMI), interactive digital video system (DVI), display port, Bluetooth and / or others. The external display may be a monitor, 3D goggles, viewing mirrors, etc. Any combination of external displays may be connected to the output interface 172. Any display 182 may display a graphical user interface for operating the medical viewing device 100.

[0110] The computer 186 or the image processor 170 can be connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission line 196 can be wired or wireless, or partially wired and partially wireless. The computer 186 and / or the image processor 170 do not need to be integrated as a whole in the medical observation device 100, but are physically away from the digital imaging system 102. To this end, the digital imaging system 102 and the computer 186 and / or the image processor 170 can be connected to a network, such as a local area network (LAN), a wireless local area network (WLAN) or a wide area network (WAN), and at least one display 182 can also be connected to the network. The network connection can be provided by the output interface 172.

[0111] According to a modification, the medical observation device 100 can be stereoscopic, but only includes two cameras, one in each stereo channel. In one stereo channel, a fluorescence color camera 111 is used and is configured to also selectively record white light reflections, while in another stereo channel, a white light color camera 110 is used. If fluorescence is not used, this arrangement provides a stereoscopic white light color image, and if fluorescence is used, a monoscopic white light color image and a monoscopic fluorescence color image are provided. The above and following descriptions also apply to this configuration.

[0112] According to another configuration, each stereo channel may include more than two cameras. For example, for each stereo channel, there may be two fluorescence color cameras 111 and one white light color camera 110. Additionally or alternatively, for each stereo channel, there may be more than one white light color camera 110, wherein preferably, the imaging spectra of these white light color cameras 110 are different or even complementary.

[0113] exist Figure 2 , a color conversion function 140, more specifically, a spatial color distribution homogenization function 140, is shown being applied to a digital input color image 200. The digital input color image 200 is the result of combining the digital white light color image 114 and the digital fluorescent color light image 112.

[0114] In this combination, an input pixel 150d of the digital input color image 200 is formed by a (first) pixel 150a of the digital white light color image 114 and a (second) pixel 150b of the digital fluorescent color image 112. If the digital white light color image 114 and the digital fluorescent color image 112 are registered images, the pixels 150a and 150b are corresponding pixels. This means that the position x, y of the first pixel 150a within the digital white light color image 114 corresponds to the position x, y of the second pixel 150b within the digital fluorescent image 112. The color conversion function 140 is then applied to each input pixel 150d, forming an output pixel 150c of the digital output color image 160. Since the color conversion function 140 depends on the position x, y of the input pixel 150d, non-uniformities in the color distribution within the images 112, 114 can be evened out.

[0115] exist Figure 3 In FIG. 1 , an example is given of how the images 112 , 114 or the first pixel 150 a and the second pixel 150 b are combined.

[0116] The first imaging spectrum 302 may be recorded by the digital white light camera 110 and / or represented in the digital white light color image 114, respectively. The intensity I over the wavelength / color λ of the first imaging spectrum 302 is shown as normalized. The first imaging spectrum 302 preferably extends at least across the visible spectrum 312.

[0117] Merely as an example, the color space in which the first imaging spectrum 302 is recorded may be an RGB color space having three primary colors or color bands 304, 306, and 308, which are only quantitatively represented. In the RGB color space, one primary color 304 is blue, another primary color 306 is green, and the third primary color 308 is red. The sensitivity of the sensor of the white light color camera 110 in the different primary colors 304, 306, 308 is tuned to produce a combined sensitivity that is as constant as possible over the visible spectrum 312.

[0118] If a color space other than RGB is used, the number, position and / or width of the color bands may be different.

[0119] Furthermore, an example of a second imaging spectrum 322 is shown as it can be recorded by the fluorescence color camera 111 and / or represented in the digital fluorescence color image 112, respectively. Merely as an example, the color space in which the second imaging spectrum 322 is recorded is also the RGB color space. The sensitivity of the sensor of the fluorescence color camera 111 in the different primary colors 304, 306, 308 is tuned to produce a sensitivity that is as constant as possible over the visible spectrum 312.

[0120] The spectra 302, 322 need not be recorded in the same color space, although this is preferred.

[0121] In the example shown, the first imaging spectrum 302 does not include the fluorescence excitation light and the fluorescence emission spectrum 326 of at least one fluorophore 116, 118. Therefore, the first imaging spectrum 302 may include at least one stop band 310, which is consistent with the fluorescence emission spectrum 326 of at least one fluorophore 116, 118, and the fluorescence of the fluorophore is to be recorded by the fluorescence color camera 111. The stop band 310 is generated, for example, by the white light filter 188. The number, width and / or position of the stop bands 310 depends on the number and type of fluorophores to be observed in the object 106.

[0122] The second imaging spectrum 322 may include one or more passbands 324. The number, position and / or width of the passbands depends on the number and type of fluorophores used. At least one passband 324 preferably corresponds to at least one stopband 310. At least one passband 324 may be generated, for example, by a fluorescent color filter 190.

[0123] The first imaging spectrum 302 and the second imaging spectrum 322 are preferably complementary to each other. They preferably complement each other to cover the entire or most of the visible spectrum 312.

[0124] It should be noted that the above description relates only to exemplary arrangements. In other embodiments or configurations of the medical viewing device 100, the spectra 302, 322 need not be complementary and / or the stopband 310 and / or passband 324 are not required, or the number, range, and position of the bands 310, 324 are different. In addition, in alternative embodiments, each of the images 112, 114 can be a reflectance image or a white light color image.

[0125] In the color conversion function 140, the individual spectra 302 and 322 are processed into a single combined multispectral spectrum having a number of color bands corresponding to the sum of the color bands in the first imaging spectrum 302 and the second imaging spectrum 322. In other words, the digital fluorescent color image 112 and the digital white light color image 114 and their pixels 150a, 150b are processed together into a single (virtual) multispectral image.

[0126] The RGB digital white light color image 114 may include color space coordinates R1, G1, and B1 at each pixel 150a. Each color space coordinate represents the intensity I of light in a corresponding color band.

[0127] The digital fluorescent color image 112 includes color space coordinates R2, G2, B2 in each color band at each pixel 150b. Since the first imaging spectrum 302 and the second imaging spectrum 322 do not overlap, each color space coordinate R1, R2, G1, G2, B1, B2 includes different spectral information, just like in a multi-spectral image. Due to the complementary stop band 310 and pass band 324, each (or at least some) color bands are separated between the images 112, 114.

[0128] In the illustrated embodiment, color band 304 is divided into two non-overlapping portions, which are represented by color space coordinates R1 and R2. In addition, color band 306 is divided into two non-overlapping portions, represented by color space coordinates G1 and G2, respectively, and color band 308 is also divided into two non-overlapping portions, represented by color space coordinates B1 and B2, respectively.

[0129] In order to process the two images 112, 114 together as a multispectral image, a union or synonymous union 330 of color space coordinates is formed for each pixel 150a and the corresponding pixel 150b. The first pixel 150a and the second pixel 150b together form an input pixel 150d, which can be physically present because the union 330 of the two pixels 150a, 150b is only virtually or logically stored in the memory because the first pixel 150a and the second pixel 150b are kept separate but are processed together by using pointers to their respective storage locations. For example, the union 330 of the color space coordinates {R1, G1, B1} of the pixel 150a and the color space coordinates {R2, G2, B2} of the pixel 150b is the set {R1, R2, G1, G2, B1, B2}. Thus, input pixels 150d may form a digital input color image 200 that is physically present in memory or logically created in the manner in which pixels 150a, 150b are processed.

[0130] Union 330 represents a multispectral image of object 106 that includes more color bands than the individual color spaces of first imaging spectrum 302 and second imaging spectrum 322 .

[0131] In one embodiment, the color conversion function 140 only uniformizes the spatial distribution of colors without changing the number of color bands. In this case, the color conversion matrix can be a square matrix with m dimensions, where m corresponds to the sum of the number of color bands in the digital white light color image 114 and the number of color bands in the fluorescent color image 112.

[0132] Thus, if both the digital output color image 160 and the digital fluorescent color image 112 are represented in the RGB color space, the color conversion matrix 142 may be a 6×6 matrix. The color conversion matrix 142 is applied to the union {R1, R2, G1, G2, B1, B2} of the color space coordinates of each first pixel 150a and the color space coordinates of its corresponding second pixel 150b to obtain the output color space coordinates R1 * , R2 * 、G1 * , G2 * 、B1 * 、B2 * , they are Figure 3 It is given the reference number 340.

[0133] The matrix coefficients C11, C12, ..., C63 can be determined experimentally. At least one matrix element is a function of the position x, y. For example, the matrix element C53 can depend on the position: C53 = C53 (x, y). In general, any matrix element Cmn (m = 1 ... M, n = 1 ... N, where Figure 3 Where M=N=6) can be a function of position x, aperture a, focal length f, and / or working distance d, or a function of a particular lighting setting, which can be represented by a variable s, such as for input to a lookup table Cmn=Cmn(x, y, a, f, d, s). The variable s can be, for example, an integer value representing the lighting mode of the lighting system 178.

[0134] The color conversion matrix 142 may operate on one input pixel 150d to generate an output pixel 150c. The output pixel 150c is located at the same position in the digital output color image 160 as at least one pixel 150a, 150b in their respective images.

[0135] Here, the color conversion function 140 is a spatial color distribution uniformization function, which uniformizes the initially non-uniform spatial color distribution.

[0136] Any optical element of the group including any element of the objective 174, the cameras 110, 111 and the color separation component 176 may introduce an uneven color distribution in the images 112, 114 recorded by the respective cameras 110, 111. The uneven color distribution results in a color gradient on the respective images.

[0137] The color gradient may depend on optical parameters such as magnification 166 , aperture 164 , and / or working distance 162 .

[0138] For example, there may be a gradient in color tone between different parts of an image. If an image of known, constant color is recorded, the color gradient can be determined. This is referred to below Figure 7 Provide explanation.

[0139] exist Figure 7 In FIG. 1 , a digital color image of a predetermined color is shown at a), the predetermined color being constant in the image, the digital color image being taken using the medical observation apparatus 100 without applying a color conversion function 140 for spatially homogenizing the color distribution.

[0140] exist Figure 7 In b), for the entire image, the deviation of the recorded color from the predetermined color is shown. For example, the upper half of the image may have a red-pink deviation, while the lower half of the image may have a red-orange deviation, resulting in a color gradient 700 across the image. Other deviations may occur in other directions.

[0141] After applying the color conversion function 140, i.e. after homogenizing the spatial color distribution, the Figure 7 The image shown in c). Figure 7 As can be seen at d) in FIG. 5 , the deviation from the intended color is reduced and the color gradient is smaller in any direction. The hue is uniform across the image.

[0142] The color gradient may also depend on the configuration of the lighting assembly 178, which may illuminate the object 106 with a spatially non-uniform spectrum. Figure 5 17 is exemplarily shown in FIG. 17 , which shows an example of an illumination system 178 including a plurality of illumination elements 502 (500), which may be light emitting diodes (LEDS). For example, the plurality (illumination elements) 500 may include an LED array. The plurality 500 is configured to illuminate the object 106 using an adjustable spectrum. In one embodiment, each light emitting element may be configured to produce a different color, as in the case of using a multi-color LED, which may be operated to produce different colors. In another embodiment, each light emitting element may produce the same color. In the latter case, different combinations of illumination elements are activated to change the illumination spectrum.

[0143] like Figure 5 An illumination system as shown or as described in EP3878355A1 may introduce spatial inhomogeneities, since even if the illumination system is arranged to produce a constant color, the color distribution in the object plane may not be constant over the object. This may be due to the illumination elements not having exactly the same single illumination spectrum and / or not being precisely aligned.

[0144] The quantitative curve 504 quantitatively represents the color distribution along random directions of the constant color object 106, which is uniformly illuminated by standard illumination. The spatial color distribution homogenization function 140 is applied to even out the effect of the irregular illumination in terms of hue. A separate color conversion function 140 can be created for each different setting of the illumination system and accessed via a lookup table. The lookup table can be indexed by assigning a value representing this setting to each different setting.

[0145] The color conversion function 140 for the optical parameters can be determined experimentally by using an illumination sphere in front of the medical viewing assembly and recording images for different constant, predetermined colors and settings of different optical parameters. The analytical function of any matrix element can be obtained using splines, polynomials or multivariate interpolation.

[0146] Once the calibration of the inhomogeneities introduced by the optical elements has been completed, the calibration of the spatial color inhomogeneities introduced by the illumination system can be performed. For example, the illumination system 178 can illuminate a gray card as the object 106 using different colors.

[0147] As a result of this calibration, a color conversion function 140 is obtained which depends on the position x, y and / or the optical parameters of the input pixel 150d so as to homogenize the color gradient.

[0148] exist Figure 4 An example of an image processing pipeline is provided in . The color conversion function 140 may be part of such an image processing pipeline 400. Figure 4 As shown, in addition to the images 112, 114, a further color camera 402 may provide a further digital color image 404. As explained above, there may be any number of additional color cameras and digital color images.

[0149] In a first optional step 410 of the image processing pipeline 400 , de-layering or demosaicing 410 may be performed on the respective image 112 , 114 , 404 .

[0150] As a further additional step, a registration 420 of the images 112 , 114 and optionally of the image 404 may be performed.

[0151] Next, the images 112, 114 and optionally the image 404 are processed together by a color conversion function 140 to homogenize the spatial color distribution. If a color conversion matrix 142 is used, its size depends on the number of input images 112, 114, 404 and the number of color space coordinates. For example, Figure 3As described above, if the images 112, 114 are RGB images, the color conversion matrix 142 is a 6×6 matrix. If the three RGB color images 112, 114, 404 are input to the color conversion function 140, the color conversion matrix 142 is a 9×9 matrix. As a result of applying the color conversion function 140, a digital output color image 160 is obtained. The output color image 160 has the same number of color space coordinates as the input to the color conversion function 140.

[0152] As a next optional step 430, input constraints may be imposed on the digital output color image to avoid saturation. As a further optional step, color calibration 440 may be applied. Color calibration 440 may include a reduction of color space coordinates. For example, during color calibration 440, the number of color coordinates in the digital output color image 160 may be reduced. For example, after color calibration 440, the digital output color image 160 may be an RGB image, having only color space coordinates. For example, color calibration 440 may include a color calibration matrix, one dimension of which corresponds to the number of color space coordinates in the union 330, and the other dimensions correspond to the number of color space coordinates in the output color space, such as RGB. In particular, if the digital input color image 200 is generated by two images, the color calibration matrix may be a 6×3 matrix, and if the digital input color image 200 is generated by three images 112, 114, 404, the color calibration matrix may be a 9×3 matrix. Color calibration 440 may map the colors of the digital output color image 160 to more natural colors.

[0153] A next optional step may include intensity or brightness homogenization 450 to compensate for brightness drops towards the edges of the digital output color image 160 .

[0154] In a further optional step, contrast enhancement 460 may be applied to the digital output color image 160 .

[0155] Before the output color image 160 is displayed on the display 132 , 182 , further post-processing steps 470 may be performed, such as gamma correction and / or conversion to the color space used by the display 132 , 182 , such as sRGB.

[0156] Figure 6A schematic diagram of a system 600 configured to perform the methods described herein is shown. System 600 includes a microscope 610 and a computer system 620. Microscope 610 is configured to capture images and is connected to computer system 620. Computer system 620 is configured to perform at least part of the methods described herein. Computer system 620 can be configured to execute a machine learning algorithm. Computer system 620 and microscope 610 can be separate entities, but can also be integrated together in a common housing. Computer system 620 can be part of a central processing system of microscope 610, and / or computer system 620 can be part of a subcomponent of microscope 610, such as a sensor of microscope 610, an actor in reinforcement learning, a camera or a lighting unit, etc.

[0157] The computer system 620 may be a local computer device (e.g., a personal computer, a laptop computer, a tablet computer, or a mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system distributed in different locations, such as one or more processors and one or more storage devices distributed in a local client and / or one or more remote server farms and / or data centers). The computer system 620 may include any circuit or combination of circuits. In one embodiment, the computer system 620 may include one or more processors, which may be of any type. As used herein, a processor may refer to any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA) such as a microscope or a microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 620 may be custom circuits, application specific integrated circuits (ASICs), or the like, such as one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 620 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as a main memory in the form of a random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 620 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 620.

[0158] Some method steps or all method steps can be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers or electronic circuits. In some embodiments, some one or more of the most important method steps can be performed by such devices.

[0159] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation may be performed using a non-transitory storage medium, such as a digital storage medium, such as a floppy disk, a digital video disk (DVD), a Blu-ray, a compact disk (CD), a read-only memory (ROM), a programmable read-only memory (PROM), and an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory, on which electronically readable control signals are stored, which cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer readable.

[0160] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0161] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is operable to perform one of the methods.For example, the program code can be stored on a machine-readable carrier.

[0162] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0163] In other words, therefore, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0164] Therefore, a further embodiment of the present invention is a storage medium (or data carrier, or computer-readable medium) comprising a computer program stored thereon, which, when executed by a processor, is used to perform one of the methods described herein. The data carrier, digital storage medium or recorded medium is generally tangible and / or non-transient. A further embodiment of the present invention is a device as described herein, comprising a processor and a storage medium.

[0165] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection, for example via the Internet.

[0166] Further embodiments comprise a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0167] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0168] Another embodiment according to the invention comprises a device or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a storage device, etc. The device or system may, for example, comprise a file server for transmitting the computer program to the receiver.

[0169] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the method described herein. In some embodiments, the field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, the method is preferably performed by any hardware device.

[0170] Reference numerals:

[0171] 100 Medical observation device

[0172] 101L Left channel stereo component

[0173] 101R Right Channel Stereo Component

[0174] 102 Digital Imaging System

[0175] 104 Eyepiece

[0176] 106 Objects to be studied

[0177] 107 Biological tissues

[0178] 110 White light color camera

[0179] 111 Fluorescent Color Camera

[0180] 112 Fluorescent color images

[0181] 114 White light / background color image

[0182] 116 Fluorophore

[0183] 118 Second Fluorophore

[0184] 132 Overall / internal image display device

[0185] 134 Direct light path

[0186] 136 beam splitter

[0187] 140 Color conversion function / spatial color distribution uniformity function

[0188] 142 color conversion matrix

[0189] 150a The first pixel in the background color image

[0190] 150b The second pixel in the second image

[0191] 150c output pixel in the output color image

[0192] 160 Output color image

[0193] 162 working distance

[0194] 164 aperture

[0195] 166 magnification

[0196] 170 Image Processor

[0197] 172 output interface

[0198] 174 objective lens

[0199] 176 Color Separation Components

[0200] 178 Lighting components

[0201] 179 Lighting Filters

[0202] 180 illumination beam splitter

[0203] 182 Image display device

[0204] 184 Field of View

[0205] 186 Computer

[0206] 188 White Light Filter

[0207] 190 Fluorescent Color Filter

[0208] 192 Dichroic Beam Splitter

[0209] 194 Memory

[0210] 196 data transmission lines

[0211] 200 digital input color images

[0212] 302 First Imaging Spectrum

[0213] 304, 306, 308 ribbon

[0214] 310 Stopband

[0215] 312 The visible spectrum or range of light

[0216] 322 Second Imaging Spectrum

[0217] 324 passband

[0218] Fluorescence emission spectra of 326 fluorophores

[0219] 330 Union / Union

[0220] 340 Output color space coordinates

[0221] 400 Image Processing Pipeline

[0222] 402 Additional color camera

[0223] 404 Additional digital color image

[0224] 410 Demosaic

[0225] 420 Registration

[0226] 430 Input Restriction

[0227] 440 Color Calibration

[0228] 450 Brightness uniformity

[0229] 460 Contrast Enhancement

[0230] 470 Additional post-processing steps 500 Multiple / array of lighting elements 502 Lighting elements / LEDs

[0231] 504 color distribution curve

[0232] 600 System

[0233] 610 Microscope

[0234] 620 Computer Systems

[0235] 700 color gradients

[0236] R, G, B, R1, G1, B1, R2, G2, B2, R * , B * , G *

[0237] Color space coordinates

[0238] C matrix coefficients

[0239] I Strength

[0240] x,y pixel position

[0241] λ wavelength

[0242] a-Aperture

[0243] fFocal length / magnification

[0244] d Working distance

[0245] s Variables for lighting settings

Claims

1. An image processor (170) for a medical observation device (100), the medical observation device (100) being, for example, a microscope or an endoscope, wherein the image processor (170) includes a color conversion function (140); Wherein the image processor (170) is configured to: Retrieving an input pixel (150d) of a digital input color image (200) and a position (x, y) of said input pixel in said input color image (200); applying a color conversion function to an input pixel to generate an output pixel (150c) in a digital output color image (160); and wherein the color conversion function (140) depends on the position of the input pixel.

2. The image processor (170) of claim 1, wherein the image processor is configured to retrieve at least one optical parameter (162, 164, 166) from a group consisting of: an optical parameter representing a working distance (162) at which the digital input color image is recorded; representing optical parameters of an aperture (164) in which the digital input color image is recorded; an optical parameter representing a magnification (166) at which the digital input color image is recorded; Optical parameters that represent the spatial color distribution of an illumination system; And wherein the color conversion function (140) also depends on the at least one optical parameter.

3. An image processor (170) according to claim 1 or 2, wherein the color conversion function (140) comprises a color conversion matrix (142) having a first dimension and a second dimension; wherein each of said input pixels (150d) comprises color space coordinates (R1, R2, G1, G2, B1, B2) in an input color space (RGB); Each of the output pixels (150c) comprises a color space coordinate (R1 * , R2 * 、G1 * , G2 * 、B1 * 、B2 * ); The first dimension of the color conversion matrix (142) corresponds to the number of color space coordinates (R, G, B) in the input color space (RGB); The second dimension of the color conversion matrix (142) corresponds to the number of color space coordinates (R, G, B) in the output color space (RGB).

4. An image processor (170) according to claim 3, wherein the color conversion matrix (142) includes at least one matrix element (C11, ..., C66), and the at least one matrix element (C11, ..., C66) depends on at least one of the positions (x, y) of the input pixel (150d).

5. An image processor (170) according to claim 3 or 4 and according to claim 2, wherein the color conversion matrix (142) includes at least one matrix element (C11, ..., C66), and the at least one matrix element (C11, ..., C66) depends on the at least one optical parameter.

6. An image processor (170) according to any one of claims 3 to 5, wherein the at least one matrix element (C11, ..., C66) comprises (C11, ..., C66) at least one of a polynomial function, a spline function and a multivariate interpolation function.

7. An image processor (170) according to any one of claims 1 to 6, wherein the color conversion function (140) is configured to homogenize the spatial color distribution, wherein the color gradient (700) on the digital input color image (200) is greater than the color gradient (700) on the digital output color image (160).

8. The image processor (170) according to any one of claims 1 to 7, wherein: The image processor is configured to: retrieving a digital white light color image (114) of the object (106) recorded in the first imaging spectrum (302), the digital white light color image (114) comprising a plurality of first pixels (150a), each first pixel (150a) comprising a first set ({R1, B1, G1}) of color space coordinates ({R1, B1, G1}) in a color space (RGB); retrieving a digital fluorescent color image (112) of the object (106) recorded in the second imaging spectrum (322), the digital fluorescent color image (112) comprising a plurality of second pixels (150b), each second pixel (150b) comprising a second set ({R2, B2, G2}) of color space coordinates (R2, G2, B2) in a color space (RGB); generating an input pixel (150d) of a digital input color image (200) from one of the first pixels (150a) and one of the second pixels (150b), the input pixel comprising a third set ({R1, R2, G1, G2, B1, B2}) of color space coordinates (R1, R2, G1, G2, B1, B2) in a color space (RGB); to generate a third set as the union of the first set and the second set.

9. The image processor (170) of claim 8, wherein the image processor is configured to register the digital fluorescent color image (112) and the digital white light color image (114) prior to retrieving the input pixels.

10. A medical observation device (100), such as a microscope or an endoscope, comprising: An image processor (170) according to claims 1 to 9; and at least one color camera (110, 111); The medical viewing device is configured to generate the digital input color image (200) from at least one color image (112, 114) generated by the at least one camera.

11. The medical observation device (100) according to claim 10, wherein: The medical observation device also includes at least: a white light color camera (110) for recording a digital white light color image (114); and A fluorescence color camera (111) is provided for recording a digital fluorescence color image (112).

12. A computer-implemented method for a medical observation device (100), such as a microscope or an endoscope, the computer-implemented image processing method comprising the steps of: Retrieving input pixels (150d) of a digital input color image (200); retrieving the position (x, y) of an input pixel in the digital input color image (200); applying a color conversion function (140) to the input pixel to generate an output pixel (150c) in a digital output color image (160); The color conversion function (140) depends on the position of the input pixel.

13. A computer program product or a computer readable medium comprising instructions which, when the program is executed by a computer, cause the computer to perform the method of claim 12.

14. A method for operating a medical viewing device (100), such as a microscope or an endoscope, comprising the computer-implemented method according to claim 12 and further comprising the steps of: Recording the digital fluorescence color image (112) in the second imaging spectrum (322) using a fluorescence color camera (111); recording the digital white light color image (114) in the first imaging spectrum (302) using a white light color camera (110); The digital input color image (200) is formed from a combination of the digital white light color image and the digital fluorescent color image.

15. The method according to claim 14, further comprising the steps of: recording the digital fluorescent color image (112) as a reflected image of the object (106); The digital white light color image (114) is recorded as a reflected image of the object (106).

Images