Image processor and computer-implemented method for a medical observation device using a position-dependent color conversion function

Abstract

An image processor (170) and a computer-implemented method for a medical observation device (100) such as a microscope or an endoscope are described. To improve color accuracy, the image processor (170) includes a color conversion function (140). Further, the image processor (170) is configured to obtain an input pixel (150d) of a digital input color image (200) and a position (x, y) of the input pixel within the input color image (200), and apply the color conversion function to the input pixel to form an output pixel (150c) in a digital output color image (160). The color conversion function (140) depends on the position of the input pixel. By using such a solution, it is possible to compensate for non-uniformities in the spatial color distribution within the digital input color image introduced by an optical element between the object (106) to be imaged and the image sensor and / or by an illumination system.

Description

Technical Field

[0001] The present invention relates to an image processor for a medical observation device such as a microscope or an endoscope, and a computer-implemented method.

Background Art

[0002] Today, medical observation devices use digital color cameras and the digital color images derived therefrom to present information to users such as surgeons or scientists. To present accurate visual information to the user, it is important that the color image represents the object as faithfully as possible. Therefore, digital color images are processed in an image processing pipeline that uses several processing steps to improve the visualization of the image content.

Summary of the Invention

Problems to be Solved by the Invention

[0003] There is a requirement to improve the accuracy of the representation of objects in the recorded color image.

Means for Solving the Problems

[0004] The above requirement is satisfied by an image processor for a medical observation device such as a microscope or an endoscope, including a color conversion function, acquiring an input pixel of a digital input color image and the position of the input pixel in the input color image, and applying the color conversion function to the input pixel to form an output pixel in a digital output color image, wherein the color conversion function depends on the position of the input pixel.

[0005] The above requirement is further satisfied by a computer-implemented method for a medical observation device such as a microscope or an endoscope, the computer-implemented image processing method including the steps of obtaining input pixels of a digital input color image, obtaining the positions of the input pixels within the digital input color image, and applying a color conversion function to the input pixels to form output pixels within a digital output color image, where the color conversion function depends on the position of the input pixels.

[0006] Typically, color correction is used for color mapping, i.e., mapping the colors of a digital input color image to different colors of a digital output color image. For example, the colors recorded in a digital input color image can be mapped to colors that appear more natural in a digital output color image. Further, it is known to adjust the colors of an image for compensating for the characteristics of a display device, such as in gamma correction. Such known color corrections are applied globally. Thus, the same color correction is applied to all pixels of the entire digital input color image.

[0007] However, due to defects in the quality and / or arrangement of the optical components of a medical observation device disposed between an object and a color camera sensor, a non-uniform spatial color distribution may occur in the digital input color image. For this reason, even if the object being imaged has a certain color, the colors recorded in the digital input image may vary across the image.

[0008] The non-uniform spatial color distribution in the recorded color image may additionally or alternatively be caused by an illumination system illuminating the object. Also, due to defects in the optical elements that direct the illumination light onto the object and / or defects in the light generation itself, a non-uniform spatial color distribution may occur in the digital input color image.

[0009] By having a color conversion function that depends on the position of the input pixel, i.e., a color conversion function that is a function of position, it is possible to compensate for non-uniform spatial color distribution and make it even more uniform. The color of the pixel is adjusted depending on the location where the pixel is located. As a result, the color of the imaged object can be expressed more accurately.

[0010] In the following, additional features for further improving the present invention will be described. Each of the following features is advantageous in itself and can be combined independently with any of the other features below. Further, each of the following features can be used independently to improve an image processor and / or a computer-implemented method even if each of these features is mentioned only in the context of an image processor or only in the context of a computer-implemented method. More specifically, an image processor can be configured to execute the process steps described below even if not specifically mentioned in relation to each process step.

[0011] For example, the color conversion function can be part of an image processing pipeline that includes at least one processing step from a group including demosaicing, registration, contrast enhancement, luminance equalization, color calibration, and / or color mapping. Contrast enhancement, luminance equalization, color calibration, and / or color mapping can use a color correction function that does not depend on position. Preferably, the position-dependent color conversion function is applied after registration and before the position-independent color correction function is applied.

[0012] In particular, the color conversion function can be a spatial color distribution equalization function. The color gradient across a specific region of the digital input color image can be greater than the color gradient across the corresponding specific region in the digital output color image after application of the color conversion function that depends on the position of the input pixel. The specific region can extend over a plurality of pixels, for example over 1 / 4 or 1 / 2 or the whole of the digital input color image.

[0013] In one embodiment, the digital input color image can represent an object having a spatially uniform color, preferably illuminated using a standard illuminant. The color gradient can be determined in the digital input color image and can be equivalent to the color gradient of the digital output color image obtained after applying a color conversion function to the digital input color image.

[0014] In one embodiment, the color conversion function can simply represent the inverse function of the color transfer function that forms the spatial color distribution of the digital output image in the respective color space coordinates for each color space coordinate of the input pixel. More specifically, the color transfer function can map the color space coordinates at the position of the input pixel to the color space coordinates at the position of the output pixel. The color transfer function can represent the defects of the optical system that introduce spatial non-uniformities corrected by the color conversion function. In one embodiment, the non-uniformities can be compensated by taking the inverse function of the color transfer function. Thus, the color conversion function here can be regarded as an inverse spatial color filter whose color filtering characteristics depend on the position.

[0015] In addition to depending on the position of the input pixel, the color conversion function can also depend on at least one optical parameter from a group including an optical parameter representing the working distance when the digital input color image is recorded, an optical parameter representing the aperture when the digital input color image is recorded, and an optical parameter representing the magnification when the digital input color image is recorded. Any of these optical parameters can change, for example, during the operation of a medical observation device, or can be changed by the user. Here, if any of these optical parameters changes during the operation of the medical observation device, the uniformity of the spatial color distribution in the digital input color image can also change. By making the color conversion function depend on these variable optical parameters, any color non-uniformities introduced during the operation of the medical observation device can be compensated.

[0016] Furthermore, the group of optical parameters on which the color conversion function may depend can also include optical parameters representing the spatial color distribution of the illumination system on the object surface. The illumination system may be part of a medical observation device.

[0017] The illumination system can include a first illumination mode and a second illumination mode, where in the first illumination mode, at least one of the illumination spectrum and intensity can vary across the object surface. The illumination system can be configured to form a first illumination spectrum in the first illumination mode and a second illumination spectrum different from the first illumination spectrum in the second illumination mode.

[0018] The image processor is preferably configured to obtain at least one optical parameter, for example, by obtaining at least one optical parameter from a controller of a medical observation device.

[0019] The spatial color distribution normalization function or the color conversion function used synonymously herein can be determined by experimental calibration. For example, various samples uniformly illuminated by a standard illuminant can be recorded. Each of the various samples can have a different spatially uniform color determined in advance. The samples here can be, for example, color cards or illuminated and calibrated screens.

[0020] By comparing the predetermined constant color with the recorded color, the spatial color distribution obtained from the optical system between the object and the image sensor can be determined. The spatial color distribution obtained from non-uniform illumination can be determined, for example, by recording gray card images in various illumination modes provided by a color-adjustable illumination system.

[0021] The input pixel can have color space coordinates representing the color of the input pixel in the input color space. Such input color spaces can be, for example, non-uniform color spaces such as the RGB color space, YCbCr color space, YUV color space, tristimulus color spaces such as the CIE1931 XYZ color space, CIEUVW color space, or uniform color spaces such as CIELUV, CIELab, and HSLuv.

[0022] Furthermore, the output pixel can include color space coordinates in the output color space. Preferably, the output color space is the same as the input color space. However, this is not essential, and the output color space may be different from the input color space. In this case, the color space conversion can be included in the color conversion function.

[0023] In one embodiment, a digital output color image is created in real time from a digital input color image. For this purpose, it is useful if the color conversion function is a linear conversion function, in particular one consisting of or including a color conversion matrix. The color conversion matrix can include at least one matrix element that depends on the position of the input pixel and / or at least one of the optical parameters described above. By using matrices, the color conversion calculations are speeded up and become efficient.

[0024] The first dimension of the color conversion matrix can correspond to the number of color space coordinates of the input color space, while the second dimension of the color conversion matrix can correspond to the number of color space coordinates of the output color space. Preferably, the color conversion matrix is square.

[0025] In one embodiment, at least one matrix element that may depend on the position of the input pixel and / or at least one optical parameter can include at least one of a polynomial function, a spline function, and a multivariate interpolation function. The polynomial function, the spline function, and the multivariate interpolation function can be stored in the memory of the image processor. The polynomial function, the spline function, or the multivariate interpolation function can also be determined by a calibration process. The input to the polynomial function, the spline function, or the multivariate interpolation function is the position of the input pixel and / or at least one optical parameter.

[0026] The position of the output pixel in the digital output color image is preferably the same as the position of the input pixel in the digital input color image. Thus, according to a preferred embodiment, the digital input color image and the digital output color image can have the same aspect ratio and the same number of pixels.

[0027] In one embodiment, the input color image can consist of a combination of two color images, for example, a combination of a digital white light color image and a digital fluorescence color image. The input pixel can produce a combination of a first pixel derived from the digital white light color image and a second pixel derived from the digital fluorescence color image. The combination of the first pixel and the second pixel is preferably the union of their respective color space coordinates. That is, if the color space coordinates of the first pixel and the color space coordinates of the second pixel are regarded as sets of their respective coordinates, the combination here can simply be obtained from and / or correspond to the union of the two sets.

[0028] For example, an image processor may be configured to obtain a digital white light color image of an object recorded in a first imaging spectrum, the digital white light color image further including a plurality of first pixels, each first pixel including a set of first color space coordinates in a color space, obtain a digital fluorescent color image of the object recorded in a second imaging spectrum, the digital fluorescent color image including a plurality of second pixels, each second pixel including a set of second color space coordinates in the color space, form an input pixel of a digital input color image from one of the first pixels and one of the second pixels, the input pixel including a third set of color space coordinates in the color space, and further form the third set as a union of the first set and the second set. Thereby, the spatial color distributions in the digital white light color image and the digital fluorescent color image can be homogenized together, and thus color conversion can be made a real-time process.

[0029] In particular, when the medical observation device is a stereomicroscope device, digital white light color images and digital fluorescent color images can exist for each stereo channel.

[0030] The use of fluorescent color images is particularly useful in medical applications where phosphors are used to mark special regions of interest. For example, some phosphors accumulate in tumors, so the fluorescence of this phosphor indicates the presence of tumor tissue. Other phosphors can be combined with blood cells and thus can be used for highlighting blood vessels. Although the intensity of fluorescence is extremely low, since the phosphor itself can indicate the state of the tissue containing the fluorescence, accurate color reproduction of the fluorescence is particularly important. Any color changes due to spatially non-uniform optical imaging may lead to an incorrect interpretation of the recorded image.

[0031] The digital white light color image and the digital fluorescence color image are preferably images registered to make the first pixel and the second pixel of the digital fluorescence color image corresponding pixels. The corresponding pixels are located at the same image position in each image, and in the registered image, represent the same object region as the image pattern in the registered image having the same size, the same orientation, and the same position.

[0032] The input pixel or the digital input color image can be physically formed in the memory of the image processing apparatus by calculating the input pixel or the entire digital input color image from the digital white light color image and the digital fluorescence color image respectively, or from the first pixel and the second pixel. In an alternative embodiment, the input pixel or the digital input color image can be formed by processing the first pixel and the second pixel or the entire digital white light color image and the entire digital fluorescence color image together respectively. In this case, the first pixel and the second pixel are held separately. Therefore, the input pixel can exist only virtually.

[0033] According to a further improvement, the first imaging spectrum and the second imaging spectrum may be complementary to each other and / or even more preferably do not overlap. Therefore, the digital white light color image and the digital fluorescence color image contain separate spectral information about the object. As a result, there is little or only negligible crosstalk between the color space coordinates, facilitating integration processing as the union of the color space coordinates.

[0034] Both the first imaging spectrum and the second imaging spectrum may overlap with the visible spectrum, but they may also include electromagnetic wavelengths beyond the visible spectrum, such as IR wavelengths or NIR wavelengths.

[0035] When a digital white light color image and a digital fluorescence color image are recorded in, for example, the RGB color space, each of the first pixel and the second pixel includes three color space coordinates. At this time, the union of the color space coordinates of the first pixel and the color space coordinates of the second pixel includes six color space coordinates. When a color conversion matrix is used for uniformizing the spatial color distribution, the color conversion matrix has a 6×6 dimension, and the output pixel includes six color space coordinates.

[0036] When a color space other than RGB is used, the number of color space coordinates within the first pixel and / or within the second pixel, and thus the number of color space coordinates within the output pixel, can be changed.

[0037] A color conversion function can take any number of color images as input. Any additional image increases the dimensions of the matrix and the number of color space coordinates in the input and output pixels. For example, when three RGB color images are input to the color conversion function, the union of the color coordinates of the first pixel, the color coordinates of the second pixel, and the color coordinates of the third pixel is a 9-tuple. Thus, the color conversion matrix in this case is a 9×9 matrix.

[0038] In one embodiment, a medical observation device such as a microscope or an endoscope can include an image processor configured to execute any of the above-described processing steps. The medical observation device can further include at least one color camera and can be configured to form a digital input color image from at least one color image formed by the at least one camera. In another embodiment, the medical observation device can include at least two color cameras, for example, a white light color camera that records a digital white light color image and a fluorescence color camera that records a digital fluorescence color image.

[0039] Furthermore, the medical observation device can include an additional color camera that records additional color images in a different imaging spectrum, and the additional color camera can be complementary to the first imaging spectrum, the second imaging spectrum, and other imaging spectra. For example, another fluorescent color camera can be configured to record the fluorescence spectrum of a phosphor in a third imaging spectrum that is different from the first imaging spectrum and the second imaging spectrum. The third imaging spectrum can overlap with a fluorescence emission spectrum of the phosphor that does not overlap with the fluorescence emission spectrum covered by the second imaging spectrum.

[0040] A method of operating a medical observation device according to any of the above embodiments can include the computer-implemented method described above. Furthermore, a method of operating a medical observation device can include recording a digital fluorescent color image in a second imaging spectrum using a fluorescent color camera, recording a digital white light color image in a first imaging spectrum using a white light color camera, and forming a digital input color image from a combination of the digital white light color image and the digital fluorescent color image.

[0041] In one embodiment, in addition to the digital white light color image, the digital fluorescent color image can be recorded as a reflected image of the object, i.e., as a second reflected image. In another embodiment, the digital fluorescent color image can represent the fluorescence emission of at least one phosphor.

[0042] Furthermore, the digital white light color image can be recorded as a reflected image of the object. In another embodiment, the digital white light color image can also include the fluorescence emission of at least one phosphor.

[0043] The subject matter recited in the claims also relates to a computer-readable medium and a computer program including instructions for causing a computer to execute the computer-implemented method according to any of the above embodiments.

[0044] The image processor may be a data processing device.

[0045] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0046] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent descriptions of corresponding methods, where a block or apparatus corresponds to a step or a feature of a step. Similarly, aspects described in the context of a step also represent descriptions of corresponding blocks or items or features of a corresponding apparatus.

[0047] The present invention will be illustratively described below with reference to embodiments and drawings. The combinations of features shown in these embodiments should not be construed as limiting. For example, among the embodiments having the technical effects described above as examples, features that are not required in a specific application can be omitted. Conversely, among the features described above that are not part of the embodiments described later, if the technical effects related to the specific features are required in a specific application, they can also be added.

[0048] Throughout this specification and the drawings, the same reference numerals are used for elements that correspond to each other with respect to function and / or structure. The drawings show the following.

Brief Description of the Drawings

[0049]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0050] Referring to FIG. 1, first, an exemplary embodiment of the present invention will be described.

[0051] In FIG. 1, a medical observation device 100 is schematically shown. The medical observation device 100 may be a fluorescence microscope or a fluorescence endoscope. The main difference between a microscope and an endoscope is that in an endoscope (not shown), for example, an object 106 to be examined is observed by an optical fiber drawn near the object 106 to be examined by insertion into the body, while in a microscope, the objective lens 174 is oriented towards the object 106. Although the medical observation device in FIG. 1 is shown as a microscope, the following description is also applicable to an endoscope. The medical observation device 100 may be, for example, a medical observation device used in a surgical operation. Also, the medical observation device 100 may be a medical observation device used in a laboratory, such as a laboratory microscope. The object 106 to be examined may consist of a biological tissue 107 or include the biological tissue 107, but may also contain an inorganic substance or consist of an inorganic substance.

[0052] The object 106, or more specifically the object surface, is located at the position of the working distance 162 from the objective lens 174. The depth of field can be determined by the aperture 164. The magnification 166 of the object 106 can be determined by the focal length of the objective lens 174, which can be a zoom objective lens in one embodiment. The working distance 162, the magnification 166, and / or the aperture 164 are optical parameters that can be changed by the user during the operation of the medical observation device 100. The imaging characteristics of the medical imaging device 100 can change when the optical parameters 162, 164, 166 change. One of the imaging characteristics that can change is the spatial color distribution, i.e., the change in color across the image plane.

[0053] The object 106 can include one or more phosphors 116, 118. At least one phosphor 116 can be a phosphor that is inherently contained in the object. For example, bone and blood contain phosphors. Also, at least one phosphor can be one that can be added to the object 106, for example, by applying it to the object 106. For example, when the object includes biological tissue 107, at least one phosphor 116, 118 can be injected into the object 106. Examples of phosphors that can be added to the object 106 include ICG, fluorescein, and / or 5-ALA. 5-ALA is synthesized into pPIX in cells.

[0054] The medical observation device 100 can be configured to record the fluorescence of one or more phosphors 116, 118. This means that the medical observation device 100 can be configured to observe, record, and preferably also excite the fluorescence of one or more phosphors 116, 118.

[0055] The medical observation device 100 can be a stereomicroscope device illustratively shown in FIG. 1. Therefore, the medical observation device 100 can have two identical sub-assemblies 101L and 101R for each of the two stereo channels. Since the two sub-assemblies 101L and 101R are identical in function and structure, the following description focuses on the right sub-assembly 101R, but is equally applicable to the left stereo channel 101L. Each of the sub-assemblies 101L and 101R may be visible in plan view.

[0056] Alternatively, the medical observation device 100 may be a plan view device. In this case, only one of the two sub-assemblies 101L and 101R may be present. Therefore, the following description is equally applicable to the medical observation device 100 that is visible in plan view.

[0057] In one embodiment, the medical observation device 100 can be used to form one or more digital white light color images 114. The digital white light color image 114 can particularly represent a reflected image of the object 106 over the visible light region. The visible light region or visible spectrum includes wavelengths from about 310 nm to about 1100 nm, or wavelengths from about 380 nm to 750 nm, or wavelengths from about 450 nm to about 700 nm.

[0058] The fluorescence spectrum, or when two or more phosphors are used, the fluorescence spectra of the phosphors 116 and 118 are preferably excluded from the spectrum recorded in the digital white light color image 114. This ensures that only the reflected light is included in the digital white light color image 114 when fluorescence is present. For example, when 5-ALA / pPIX is used as the phosphor, in one embodiment, the fluorescence spectrum from about 625 nm to about 650 nm is not recorded in the digital white light color image 114.

[0059] When light of a specific wavelength is used to excite fluorescence, the spectrum consisting of or including that wavelength is preferably not recorded or represented in the digital white light color image 114. For example, when 5-ALA / pPIX is used as a phosphor, fluorescence can be excited by illuminating the object 106 with wavelengths from about 380 nm to about 450 nm. These wavelengths are preferably not combined within the digital white light color image 114. For fluorescein, ICG, and other phosphors, known ranges different from those for 5-ALA / pPIX apply to the excitation and emission spectra.

[0060] When the fluorescence excitation spectrum and the fluorescence emission spectrum are not recorded, the digital white light color image 114 represents the reflection on the object 106, i.e., the white light image of the object 106 observed by a human observer. In such a configuration, the digital white light color image 114 can be regarded as a natural color or pure color image.

[0061] The digital imaging system 102 can further be used to record one or more digital fluorescence color images 112 of the object 106. The digital fluorescence color images 112 can represent or include the fluorescence emission of one or more phosphors 116, 118. The digital fluorescence color images 112 can be recorded simultaneously with the white light color image 114.

[0062] The digital fluorescence color images 112 preferably do not record wavelengths other than the emission spectra of one or more phosphors. The digital fluorescence color images 112 can be reflection images of the object 106, or can include the fluorescence color and reflected light from the object 106.

[0063] The digital white light color image 114 and the digital fluorescent color image 112 can be recorded using at least three color bands or the primary colors of a color space as an equivalent. For example, both the digital white light color image 114 and the digital fluorescent color image 112 can be recorded in the RGB color space using the three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescent color image 112 can be recorded in different color spaces respectively. Each of these can represent a multi-spectral color image or a hyper-spectral color image. Although it is preferred that the digital white light color image 114 and the digital fluorescent image 112 be recorded in the same color space, this is not essential.

[0064] At least one digital white light color image 114 and at least one digital fluorescent color image 112 each include pixels 150a, 150b. Each pixel 150a, 150b has color space coordinates. In a color space, such as the RGB color space, each color of the pixels 150a, 150b is represented by color space coordinates that form a triplet of three integers. Each integer indicates the intensity of one of the primary colors R, G, B. For example, the strongest red can be indicated by the triplet {255, 0, 0}. The strongest green can be indicated by, for example, {0, 255, 0}, and the strongest blue can be indicated by, for example, {0, 0, 255}. Thus, the RGB color space is a three-dimensional space. The CMYK color space is a four-dimensional space. Color can be regarded as a point within a color space having color space coordinates, such as {0, 0, 255}. More generally, an n-dimensional color space is obtained from a multi-spectral or hyper-spectral color space having n corresponding color bands, and each color is represented by a tuple of n values.

[0065] Examples of the RGB color space include RGB, sRGB, AdobeRGB, AdobeWhiteGamutRGB, REC.2100, DCI-P3, Rec.3020, Rec.709, Rec.601, and ACES. Note that the images 112, 114 can be recorded, stored, and / or processed in any color space.

[0066] The spectrum recorded and represented in the digital white light color image 114, i.e., the first imaging spectrum, and the spectrum recorded in the digital fluorescence color image 112, i.e., the second imaging spectrum, are preferably complementary to each other. In particular, these spectra do not overlap except for the inevitable filter cage. Preferably, these spectra together represent the complete visible light spectrum or at least most of the visible light range. The first image and / or the second image may include wavelengths outside the visible spectrum, such as NIR.

[0067] More specifically, the medical observation device 100 can include a digital imaging system 102 for forming the digital fluorescence color image 112 and the digital white light color image 114. The digital imaging system 102 can include a white light color camera 110 and a fluorescence color camera 111. The digital imaging system 102 can include a separate color camera (not shown) for each of the stereo channels 101L, 101R.

[0068] The white light color camera 110 is configured to record the digital white light color image 114. In particular, the white light color camera 110 can be configured to form a stream of the digital white light color image 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images over the visible light spectrum of the wavelengths shown above. The white light color camera 110 can be a CCD, CMOS, or multi-spectral camera or a hyperspectral camera.

[0069] The fluorescence color camera 111 is configured to record a digital fluorescence color image 112. In particular, the fluorescence color camera 111 can be configured to form a stream of digital fluorescence color images 112 in the form of a digital video stream. The fluorescence color camera 111 can be configured to record the digital fluorescence color image 112 only in one or more fluorescence spectra of at least one phosphor 116, 118. The fluorescence color camera 111 can be configured to record the digital fluorescence color image 112 with only one or more narrow bands of light. Here, the narrow band is made to overlap with one or more fluorescence spectra of one or more phosphors 116, 118 for which fluorescence is to be recorded. Preferably, the fluorescence spectrum of the phosphor 116 and the fluorescence spectrum of the second phosphor 118 do not overlap, whereby the fluorescence color camera 111 can record light in two separate spectral fluorescence bands spaced apart from each other.

[0070] The fluorescence color camera 111 can be a CCD camera, a CMOS camera, or a multispectral camera, or a hyperspectral camera, or any other suitable type of camera. Preferably, the white light color camera 110 and the fluorescence color camera 111 are of the same type, but this is not essential. Since fluorescence usually has a very low intensity, the fluorescence color camera 111 can have a higher integration time than the white light color camera 110.

[0071] The respective fields of view 184 of the cameras 110, 111 are preferably aligned, or even more preferably, coincide and are coaxial. Thus, preferably, the cameras 110, 111 and any other color camera provide the same field of view 184 with the same viewpoint and focal length. As a result, the same object 106 can be represented in the images 112, 114 formed by the different cameras 110, 111. The two cameras 110, 111 can use the same objective lens 174. In such a configuration, the images 112, 114 can be optically registered, whereby registration by image processing is not required or is significantly simplified.

[0072] As will be further described below, if the coincidence of viewpoints and fields of view cannot be optically formed, image processing routines for matching or registration can be applied to the digital images 112, 114. If the cameras 110, 111 have the same viewpoints and fields of view, the application of a computer-implemented registration routine may be recommended.

[0073] It is preferable that the two cameras 110, 111 operate synchronously. In particular, the exposure times of the cameras 110, 111 can be synchronized. Thus, the medical observation device 100 can be configured to simultaneously form a digital white light color image 114 and a digital fluorescence color image 112.

[0074] Preferably, the gains of the two cameras 110, 111 are synchronized, that is, adjusted simultaneously in the two cameras 110, 111. Furthermore, the ratio of the gain applied to camera 110 to the gain applied to camera 111 can be made constant even when the gains vary. Gamma correction and color adjustment or white balance may be switched off or may be kept constant.

[0075] Any of the above measures facilitates the comparison, integration processing and / or combination of the two images 112, 114.

[0076] To separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescence color image 112, that is, to separate the reflection spectrum from the fluorescence spectrum, the medical observation device 100 can include an optical color separation assembly 176. The color separation assembly 176 can include optical elements, such as a beam splitter 192 which can be dichroic. The color separation assembly 176 can additionally or alternatively include an optical white light filter 188 and / or an optical fluorescence color filter 190.

[0077] The fluorescent color filter 190 is preferably configured to transmit light of one or more fluorescence spectra of one or more phosphors 116, 118 and block light other than the one or more fluorescence spectra.

[0078] The fluorescent color filter 190 can be configured as a band - pass filter including one or more pass - bands. Each pass - band is made to overlap with the fluorescence emission spectrum of each phosphor 116, 118 for which fluorescence is to be recorded. Since the fluorescent color filter 190 is present in the optical path between the beam splitter 192 and the fluorescent color camera 111, only the light of the wavelengths of the pass - bands of the fluorescent color filter 190 is transmitted to the fluorescent color camera 111.

[0079] The white - light filter 188 is preferably configured to block light of one or more fluorescence spectra of one or more phosphors 116, 118. Also, the white - light filter 188 can be configured to block light in the fluorescence excitation spectrum.

[0080] The white - light filter 188 is preferably configured as a band - stop filter having a stop - band corresponding to or at least including the pass - band of the fluorescent color filter 190. The white - light filter 188 is arranged in the optical path between the beam splitter 192 and the white - light camera 110. Therefore, the white - light camera 110 records only wavelengths that are outside the stop - band of the white - light filter 188 and thus also outside the pass - band of the fluorescent color filter 190.

[0081] Either one of the white - light filter 188 and the fluorescent color filter 190 may be an adjustable filter.

[0082] The digital white - light color image 114 and the fluorescent color image 112 preferably have the same aspect ratio and the same number of pixels. In other variations, the aspect ratio and / or the number of pixels of the images 114, 112 may be different.

[0083] When the beam splitter 192 is a dichroic beam splitter, in this example, since optical spectral filtering is already incorporated in the dichroic beam splitter, at least one of the filters 188 and 190 can be omitted. In this case, the above description regarding the passband and stopband shall apply mutatis mutandis to the dichroic beam splitter 192 with necessary modifications.

[0084] Accordingly, the white light color camera 110 records the digital white light color image 114 in a first imaging spectrum which may be a reflection spectrum. The first imaging spectrum may be different from the second imaging spectrum recorded by the fluorescence color camera 111 and may include one or more fluorescence emission spectra. The wavelengths included in the first imaging spectrum and the second imaging spectrum are determined by the filter settings of the color separation assembly 176.

[0085] The medical observation device 100 can further include an illumination assembly 178 which is preferably configured to illuminate the object 106 through the same objective lens 174 through which the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 can be configured to selectively generate white light, i.e., light uniformly dispersed over the entire visible spectrum, and fluorescence excitation light including only light of wavelengths that stimulate the fluorescence of at least one phosphor 116, 118. The illumination light generated by the illumination assembly 178 can be supplied to the objective lens 174 using the illumination beam splitter 180.

[0086] Depending on the phosphor and the excitation spectrum specific to its fluorescence, the illumination filter 179 can be provided. For example, when 5-ALA / pPIX is used as the phosphor, the illumination filter 179 can have a transmittance of 90% - 98% up to a wavelength of 425 nm, a transmittance of 0.5% - 0.7% at wavelengths of 450 nm - 460 nm, a transmittance of 0.1% or less at wavelengths of 460 nm - 535 nm, and a transmittance of substantially zero at wavelengths exceeding 535 nm.

[0087] Instead of or in addition to the illumination filter 179, the illumination assembly 178 can include an adjustable light source, for example, a plurality of LEDs or OLEDs colored differently from each other.

[0088] The illumination assembly 178 can have a plurality of different illumination modes, where each of the plurality of different illumination modes can include a different illumination spectrum and / or a different light intensity distribution for illuminating the object 106. For example, the illumination assembly 178 can be configured to generate fluorescence excitation wavelengths for different phosphors 116, 118 or different combinations of phosphors in different illumination modes at intensities different from the white light illumination of the object for recording a reflected image. In alternative or additional examples, the illumination assembly 178 can be configured to generate an illumination spectrum having a spectral power distribution corresponding to a standard emitter such as a CIE emitter.

[0089] An example of an illumination assembly 178 that can be used in the medical observation device 100 is described and illustrated in European Patent Application Publication No. 3878355, the entire content of which is hereby incorporated by reference into this application.

[0090] The medical observation device 100 can further include an image processor 170. The image processor 170 can be a hardware module, such as a microprocessor, or a software module. The image processor 170 can also be a combination of both a hardware module and a software module, for example, by using a software module configured to be executed on a specific processor, such as a vector processor, a floating-point graphic processor, a parallel processor, and / or a multiprocessor. The image processor 170 can be part of a general-purpose computer 186, such as a PC.

[0091] The image processor 170 is configured to acquire a digital white light color image 114 and a digital fluorescence color image 112. For example, the image processor 170 can be configured to acquire the digital white light color image 114 and the digital fluorescence color image 112 directly from the memory 194 and / or from the cameras 110, 111. The memory 194 can be part of the image processor 170 or can be provided elsewhere in the medical observation device 100.

[0092] The image processor 170 is further configured to calculate a digital output color image 160 from the digital white light color image 114 and the digital fluorescence color image 112. The digital output color image 160 is a color image represented in a color space that can be different from the color space of either the digital white light color image 114 or the digital fluorescence color image 112. However, preferably, the color space of the digital output color image 160 is the same as the color space of the digital white light color image 114 and the digital fluorescence color image 112. A standardized function can be generated or determined by experiment or analysis for the conversion of coordinates from one color space to another.

[0093] In one embodiment, the digital white light color image 114 can be used as a background color image over which the fluorescent color image 112 is overlaid as a second image. In another embodiment, the digital fluorescent color image 112 can be used as the background image, and in this case, the second pixel 150b corresponds to a pixel from the digital white light color image 114.

[0094] The image processor 170 can be configured to apply the color conversion function 140 to the digital white light color image 114 and the digital fluorescent color image 112 to form the digital output color image 160. The formation of the output color image 160 can be performed on a pixel-by-pixel basis when the first pixel 150a of the digital white light color image 114 is combined with the second pixel 150b from the digital fluorescent color image 112. In this case, the color conversion function 140 is applied to the combination to form the output pixel 150c of the digital output color image 160. Then, this can be repeated for the remaining pixels.

[0095] The color conversion function 140 is configured to compensate for the non-uniformity of the spatial color distribution in the images 112, 114. The color conversion function 140 preferably consists of or includes a color conversion matrix 142, which will be described in more detail below with reference to FIG. 3.

[0096] The first pixel 150a and the second pixel 150b are preferably corresponding pixels. Thus, in the registered images 112, 114, the first pixel 150a and the second pixel 150b can be located at the same position within their respective images 112, 114.

[0097] The number of pixels in the images 112, 114 does not necessarily have to be the same. In this case, the pixels in one of the images 112, 114 can correspond to a plurality of pixels in the other image 114, 112.

[0098] The image processor 170 can be configured to output the digital output color image 160 to any type of peripheral device. In this context, outputting includes, for example, transmitting the digital output color image 160 and / or enabling access to the digital output color image 160 by permitting access to a memory, such as memory 194, that can store the digital output color image 160, or output to the output interface 172 of the medical observation device 100.

[0099] The digital output color image 160 can be displayed on the display device 132 integrated with the medical observation device 100. For example, the display device 132 can be incorporated into the eyepiece or viewfinder 104 of the medical observation device 100. The display device 132 can also display a graphic user interface for operating the medical observation device 100.

[0100] The medical observation device 100 can include a direct optical path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In this case, the display device can be an integrated display device 132, for example, a semi-transparent display device disposed within the direct optical path 134, or a display device projected into the direct optical path 134. A beam splitter 136 can be provided to split light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light can be directed to the eyepiece 104.

[0101] In another embodiment, the medical observation device 100 does not necessarily have to have a direct optical path 134 and can simply use the integrated display device 132 to display an image. As another alternative, the medical observation device 100 does not necessarily have to have an integrated display device.

[0102] The medical observation device 100 can include an output interface 172, and one or more (external) display devices 182 and / or any type of unidirectional or bidirectional wired or wireless data connection can be formed with this output interface 172. For this purpose, the output interface 172 can include a standardized connector and data transmission protocol, such as WLAN, TCP / IP, Ethernet, USB, HDMI, DVI, DisplayPort, Bluetooth and / or others. The external display device can be a monitor, 3D goggles, glasses, etc. Any combination of external display devices can be connected to the output interface 172. Any of the display devices 182 can display a graphic user interface for operating the medical observation device 100.

[0103] The computer 186 or the image processor 170 can be connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission lines 196 can be wired or wireless, or can be partially wired or partially wireless. The computer 186 and / or the image processor 170 do not have to be physically incorporated into the medical observation device 100, but need to be located at a position physically separated from the digital image system 102. For this purpose, the digital imaging system 102 and the computer 186 and / or the image processor 170 can be connected to a network, such as a LAN, WLAN or WAN, and at least one more display device 182 can be connected to this network. The network connection can be provided by the output interface 172.

[0104] According to one variant, the medical observation device 100 is a stereomicroscope device, but may include only two cameras, one for each stereoscopic channel. In one stereoscopic channel, a fluorescence color camera 111 is used and is further configured to selectively record white light reflection as well. In the other stereoscopic channel, a white light color camera 110 is used. With such an arrangement, when fluorescence is not used, a white light color image for stereoscopic viewing is provided, and when fluorescence is used, a white light color image for planar viewing and a fluorescence color image for planar viewing are provided. The above and following descriptions are equally applicable to this configuration.

[0105] According to another configuration, each stereoscopic channel may include three or more cameras. For example, two fluorescence color cameras 111 and one white light color camera 110 may be provided for each stereoscopic channel. Additionally or alternatively, there may be two or more white light color cameras 110 for each stereoscopic channel, where preferably the imaging spectra of these white light color cameras 110 are different from each other or even complementary to each other.

[0106] FIG. 2 shows that a color conversion function 140, more particularly a spatial color distribution equalization function 140, is applied to the digital input color image 200. The digital input color image 200 is the result of a combination of the digital white light color image 114 and the digital fluorescence color light image 112.

[0107] In such a combination, the input pixel 150d of the digital input color image 200 is formed from the (first) pixel 150a of the digital white light color image 114 and the (second) pixel 150b of the digital fluorescence color image 112. When the digital white light color image 114 and the digital fluorescence color image 112 are registered images, the pixel 150a and the pixel 150b are corresponding pixels. That is, the positions x, y of the first pixel 150a in the digital white light color image 114 correspond to the positions x, y of the second pixel 150b in the digital fluorescence image 112. In this case, the color conversion function 140 is applied to each input pixel 150d, and as a result, the output pixel 150c of the digital output color image 160 is obtained. Since the color conversion function 140 depends on the position x, y of the input pixel 150d, it is possible to equalize the non-uniformity of the color distribution in the images 112, 114.

[0108] FIG. 3 shows an example of how the images 112, 114 or the first pixel 150a and the second pixel 150b can be combined.

[0109] The first imaging spectrum 302 can be recorded by the digital white light camera 110 and / or can be represented in the digital white light color image 114, respectively. The intensity I over the wavelength / color λ of the first imaging spectrum 302 is shown normalized. The first imaging spectrum 302 preferably extends at least over the visible spectrum 312.

[0110] By way of example only, the color space in which the first imaging spectrum 302 is recorded can be an RGB color space having three primary colors or color bands 304, 306, 308 shown only quantitatively. In the RGB color space, one primary color 304 is blue, another primary color 306 is green, and the third primary color 308 is red. The sensitivities of the sensors of the white light color camera 110 in the respective different primary colors 304, 306, 308 are adjusted so that, as a result, a combined sensitivity over the visible spectrum 312 as constant as possible is obtained.

[0111] If a color space other than RGB is used, the number, position, and / or width of the color bands may be different.

[0112] Furthermore, as an example of the second imaging spectrum 322, it has been shown that each can be recorded by the fluorescence color camera 111 and / or can be represented in the digital fluorescence color image 112. By way of example only, the color space in which the second imaging spectrum 322 is recorded is also the RGB color space. The sensitivity of the sensors of the fluorescence color camera 111 at different primary colors 304, 306, 308 is adjusted, so that as constant a sensitivity as possible is obtained over the visible spectrum 312.

[0113] Preferably, the spectra 302, 322 are recorded in the same color space, but this is not essential.

[0114] In the illustrated example, the first imaging spectrum 302 does not include the fluorescence excitation light and the fluorescence emission spectrum 326 of at least one phosphor 116, 118. Thus, the first imaging spectrum 302 can include at least one blocking band 310 that coincides with the fluorescence emission spectrum 326 of at least one phosphor 116, 118 whose fluorescence is to be recorded by the fluorescence color camera 111. The blocking band 310 is formed, for example, by the white light filter 188. The number, width, and / or position of the blocking band 310 depend on the number and type of phosphors observed in the object 106.

[0115] The second imaging spectrum 322 can include one or more pass bands 324. The number, position, and / or width of the pass bands depend on the number and type of phosphors used. At least one pass band 324 preferably corresponds to at least one blocking band 310. At least one pass band 324 can be formed, for example, by the fluorescence color filter 190.

[0116] The first imaging spectrum 302 and the second imaging spectrum 322 are preferably complementary to each other. These imaging spectra are preferably completed with each other to cover the whole or most of the visible spectrum 312.

[0117] It should be noted that the above description is related to the setting only as an example. In other embodiments or other configurations of the medical observation device 100, the spectra 302, 322 do not have to be complementary, and / or do not have to be the stop band 310 and / or the pass band 324, or the number, spread and position of the bands 310, 324 are different. Further, each of the images 112, 114 may be a reflection image or a white light color image in alternative embodiments.

[0118] In the color conversion function 140, the individual spectra 302 and 322 are processed as a single combined multi - spectral spectrum having a number of color bands corresponding to the sum of the color bands in the first imaging spectrum 302 and the second imaging spectrum 322. In other words, the digital fluorescence color image 112 and the digital white light color image 114 and their respective pixels 150a, 150b are processed together as a single (virtual) multi - spectral image.

[0119] The RGB digital white light color image 114 may include the color space coordinates R1, G1, and B1 at each pixel 150a. Each color space coordinate represents the light intensity I in its respective color band.

[0120] The digital fluorescence color image 112 includes the color space coordinates R2, G2, B2 in each color band at each pixel 150b. Since the first imaging spectrum 302 and the second imaging spectrum 322 do not overlap, the color space coordinates R1, R2, G1, G2, B1, B2 include different spectral information, similar to the case of a multi - spectral image. Due to the complementary stop band 310 and pass band 324, each (or at least some) color band is divided between the images 112, 114.

[0121] In the illustrated embodiment, the color band 304 is divided into two non-overlapping parts represented by color space coordinates R1 and R2. Also, the color band 306 is divided into two non-overlapping parts represented by color space coordinates G1 and G2 respectively, and the color band 308 is also divided into two non-overlapping parts represented by color space coordinates B1 and B2 respectively.

[0122] To integrally process the two images 112, 114 as multi-spectral images, for each pixel 150a and the corresponding pixel 150b, a union 330 of color space coordinates is formed as a merger or equivalent of the color space coordinates. Both the first pixel 150a and the second pixel 150b are such that the union 330 of the two pixels 150a, 150b is stored in the memory only virtually or logically, and the first pixel 150a and the second pixel 150b are maintained separately, but by using pointers to their respective memory locations, an input pixel 150d that can physically exist by being integrally processed is formed. For example, the union 330 of the set of color space coordinates {R1, G1, B1} of pixel 150a and the set of color space coordinates {R2, G2, B2} of pixel 150b is the set {R1, R2, G1, G2, B1, B2}. Thus, the input pixel 150d can form a digital input color image 200 that physically exists in the memory or is logically created in a form in which the pixels 150a, 150b are processed.

[0123] The union 330 represents a multi-spectral image of the object 106, and this multi-spectral image individually contains more color bands than the color spaces of the first imaging spectrum 302 and the second imaging spectrum 322.

[0124] In one embodiment, the color conversion function 140 only equalizes the spatial color distribution and does not change the number of color bands. In this case, the color conversion matrix is a square matrix having m dimensions, where m can be made to correspond to the sum of the number of color bands in the digital white light color image 114 and the number of color bands in the fluorescent color image 112.

[0125] Thus, when both the digital output color image 160 and the digital fluorescence color image 112 are represented in the RGB color space, the color conversion matrix 142 can be a 6×6 matrix. The color conversion matrix 142 is applied to the union set {R1, R2, G1, G2, B1, B2} of the color space coordinates of each first pixel 150a and the color space coordinates of the corresponding second pixel 150b, whereby the output color space coordinates R1 to which reference numeral 340 is assigned in FIG. 3 * , R2 * , G1 * , G2 * , B1 * , B2 * are obtained.

[0126] The matrix coefficients C11, C12, …, C63 can be determined experimentally. At least one matrix element is a function of the positions x, y. For example, the matrix element C53 may depend on the position C53 = C53(x, y). In general, any matrix element Cmn (m = 1…M, n = 1…N, where M = N = 6 in FIG. 3) can be a function of the position x, the aperture a, the focal length f and / or the working distance d, or a function of a specific illumination setting that can be represented by a variable s for input to, for example, a look-up table Cmn = Cmn(x, y, a, f, d, s). The variable s can be, for example, an integer value representing the illumination mode of the illumination system 178.

[0127] The color conversion matrix 142 can be calculated for one input pixel 150d to form one output pixel 150c. The output pixel 150c is arranged at the same position as at least one of the pixels 150a, 150b in each image within the digital output color image 160.

[0128] Here, the color conversion function 140 is a spatial color distribution equalization function. Thereby, first, the non-uniform spatial color distribution is equalized.

[0129] Any optical element of a group including any element of the objective lens 174, cameras 110, 111, and color separation assembly 176 can introduce a non-uniform color distribution into the images 112, 114 recorded by the respective cameras 110, 111. Such non-uniform color distributions result in color gradients across the respective images.

[0130] The color gradients can depend on optical parameters such as magnification 166, aperture 164, and / or working distance 162.

[0131] For example, a hue gradient can occur between different parts of an image. The color gradient can be determined when an image of a known constant color is recorded. This will be described below with reference to FIG. 7.

[0132] In FIG. 7, in a), a digital color image having a certain predetermined color across the image, captured using the medical observation device 100, is shown without applying the color conversion function 140 that spatially equalizes the color distribution.

[0133] In FIG. 7 b), the deviation between the recorded color and the predetermined color is shown for the entire image. The upper half of the image can have, for example, a red-pink deviation, while the lower half of the image can have a red-orange deviation, resulting in a color gradient 700 across the image. Other deviations can occur in other directions.

[0134] After the color conversion function 140 is applied, i.e., after the spatial color distribution is equalized, the image shown in FIG. 7 c) is obtained. As seen in FIG. 7 d), the deviation from the predetermined color is reduced and the color gradient is small in any direction. The hue is uniform across the image.

[0135] Also, the color gradient may also depend on the settings of the illumination assembly 178 that may illuminate the object 106 with a spatially non-uniform spectrum. This is exemplarily shown in FIG. 5, which shows an example of an illumination system 178 including a plurality of lighting units 500 composed of a plurality of lighting elements 502 that may be LEDs. For example, the plurality of lighting units 500 can include an LED array. The plurality of lighting units 500 are configured to illuminate the object 106 with an adjustable spectrum. In one embodiment, each lighting element can be configured to form a different color, similar to the case of a multicolor LED operable to form various colors. In another embodiment, each lighting element can form the same color. In the latter case, different combinations of lighting elements are activated to vary the illumination spectrum.

[0136] The illumination system shown in FIG. 5 or the illumination system described in European Patent Application Publication No. 3878355 may introduce spatial non-uniformity because the color distribution on the object surface may not be constant across the object even when set to form a certain color. This can be caused by lighting elements with individually non-exactly identical illumination spectra and / or lighting elements that are not exactly aligned.

[0137] The quantitative curve 504 quantitatively shows the color distribution along a random direction across the object 106 of a certain color uniformly illuminated by a standard illuminant. By applying the spatial color distribution equalization function 140, the influence of irregular illumination on the hue can be equalized. Separate color conversion functions 140 can be created for different settings of the illumination system and accessed via a look-up table. In the look-up table, indexing can be achieved by assigning a value representing the setting for each different setting.

[0138] The color conversion function 140 of the optical parameters can be determined experimentally by using the illumination sphere in front of the medical observation assembly and recording images with different predetermined constant colors and different settings of the optical parameters respectively. An analytical function for any matrix element can be obtained by using spline, polynomial or multivariable interpolation.

[0139] Calibration of the spatial color non-uniformity introduced by the illumination system can be performed after the calibration of the non-uniformity introduced by the optical elements. For example, a gray card can be illuminated as the object 106 by an illumination system 178 using various colors.

[0140] As a result of such calibration, a color conversion function 140 that depends on the positions x, y of the input pixels 150d and / or the optical parameters is obtained to equalize the color gradients.

[0141] FIG. 4 presents an example of an image processing pipeline. The color conversion function 140 can be part of such an image processing pipeline 400. As shown in FIG. 4, another digital color image 404 can be formed by another color camera 402 in addition to the images 112, 114. As described above, any number of additional color cameras and digital color images may exist.

[0142] In the first optional step 410 of the image processing pipeline 400, a debayer or demosaic 410 can be performed on each of the images 112, 114, 404.

[0143] As a further additional step, registration 420 of the images 112, 114 and optionally the image 404 can also be performed.

[0144] Next, images 112, 114 and optionally image 404 are integrally processed by color conversion function 140, thereby equalizing the spatial color distribution. When color conversion matrix 142 is used, its dimension depends on the number of input images 112, 114, 404 and the number of color space coordinates. As described with reference to FIG. 3, when images 112, 114 are RGB images, color conversion matrix 142 is a 6×6 matrix. When three RGB color images 112, 114, 404 are input to color conversion function 140, color conversion matrix 142 is a 9×9 matrix. As a result of applying color conversion function 140, digital output color image 160 is obtained. Output color image 160 has the same number of color space coordinates as the input to color conversion function 140.

[0145] As the next optional step 430, input restrictions can be imposed on the digital output color image to avoid saturation. As a further optional step, color calibration 440 can also be applied. Color calibration 440 can include reduction of color space coordinates. For example, the number of color coordinates in digital output color image 160 can be reduced during color calibration 440. For example, digital output color image 160 can be one RGB image after color calibration 440 having only color space coordinates. For example, color calibration 440 can include a color calibration matrix where one dimension corresponds to the number of color space coordinates in union 330 and the other dimension corresponds to the number of color space coordinates in an output color space such as RGB. In particular, the color calibration matrix can be a 6×3 matrix when digital input color image 200 is obtained from two images, and can be a 9×3 matrix when digital input color image 200 is obtained from three images 112, 114, 404. Color calibration 440 can map the colors of digital output color image 160 to more natural colors.

[0146] The next optional step can include intensity or luminance equalization 450 to compensate for the falloff of luminance towards the edges of digital output color image 160.

[0147] In a further optional step, contrast enhancement 460 can be applied to digital output color image 160.

[0148] Before the output color image 160 is displayed on the display devices 132, 182, further post-processing steps 470, such as gamma correction and / or conversion to the color space used by the display devices 132, 182, such as sRGB, can also be performed.

[0149] FIG. 6 shows a schematic diagram of a system 600 configured to implement the method described herein. The system 600 includes a microscope 610 and a computer system 620. The microscope 610 is configured to image and is connected to the computer system 620. The computer system 620 is configured to implement at least a portion of the method described herein. The computer system 620 may be configured to execute a machine learning algorithm. The computer system 620 and the microscope 610 may be separate entities or may be integrated within a single common housing. The computer system 620 may be part of the central processing system of the microscope 610 and / or the computer system 620 may be part of a sub-component of the microscope 610, such as a sensor, actuator, camera, or illumination unit of the microscope 610.

[0150] The computer system 620 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or a distributed computer system (e.g., a cloud computing system distributed across various locations such as local clients and / or one or more remote server farms and / or data centers, comprising one or more processors and one or more storage devices). The computer system 620 may include any circuit or combination of circuits. In one embodiment, the computer system 620 may include one or more processors, which can be of any kind. As used herein, a processor may contemplate any kind of computing circuit, such as, for example, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other kind of processor or processing circuit, but is not limited thereto. Other kinds of circuits that may be included in the computer system 620 may be, for example, custom circuits, application specific integrated circuits (ASICs), etc., such as, for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 620 may include one or more storage devices that may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives for handling removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 620 may include a display device, one or more speakers and a keyboard and / or a mouse, a trackball, a touch screen, a controller that may include a voice recognition device, or any other device that enables a user of the system to input information into the computer system 620 and to receive information from the computer system 620.

[0151] Some or all of the steps may be performed by, or using, a hardware device such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the critically important steps may be performed by such a device.

[0152] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is executable by a non-transitory recording medium, which is a digital recording medium, such as an electronic-readable control signal stored therein that cooperates (or is capable of cooperating) with a programmable computer system to implement each method, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory. Thus, the digital recording medium may be computer-readable.

[0153] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system such that any of the methods described herein are implemented.

[0154] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code, which operates to implement any of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.

[0155] Another embodiment includes a computer program stored on a machine-readable carrier for implementing any of the methods described herein.

[0156] Thus, in other words, an embodiment of the present invention is a computer program having program code for implementing any of the methods described herein when the computer program is executed on a computer.

[0157] Thus, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for implementing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium to be recorded is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein comprising a processor and a recording medium.

[0158] Thus, another embodiment of the present invention is a data stream or signal sequence representing a computer program for implementing any of the methods described herein. The data stream or signal sequence may be configured to be transferred via a data communication connection, for example the Internet.

[0159] Another embodiment includes processing means, for example a computer or programmable logic device configured or adapted to implement any of the methods described herein.

[0160] Another embodiment includes a computer having an installed computer program for implementing any of the methods described herein.

[0161] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0162] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to execute some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to implement any of the methods described herein. Generally, advantageously, the methods are implemented by any hardware device.

Description of Reference Numerals

[0163] 100 Medical observation device 101L Stereo sub-assembly for left channel 101R Stereo sub-assembly for right channel 102 Digital imaging system 104 Eyepiece 106 Object to be inspected 107 Biological tissue 110 White light color camera 111 Fluorescent color camera 112 Fluorescent color image 114 White light color image / background color image 116 Phosphor 118 Second phosphor 132 Integrated image display device / internal image display device 134 Direct optical path 136 Beam splitter 140 Color conversion function / spatial color distribution uniformization function 142 Color conversion matrix 150a First pixel in background color image 150b The second pixel in the second image 150c The output pixel in the output color image 160 The output color image 162 The operating distance 164 The aperture 166 The magnification 170 The image processor 172 The output interface 174 The objective lens 176 The color separation assembly 178 The illumination assembly 179 The illumination filter 180 The illumination beam splitter 182 The image display device 184 The field of view 186 The computer 188 The white light filter 190 The fluorescent color filter 192 The dichroic beam splitter 194 The memory 196 The data transmission line 200 The digital input color image 302 The first imaging spectrum 304, 306, 308 The color bands 310 The stop band 312 The visible spectrum or light range 322 The second imaging spectrum 324 The pass band 326 The fluorescence emission spectrum of the phosphor 330 The merge / union 340 The output color space coordinates 400 The image processing pipeline 402 Another color camera 404 Another digital color image 410 The demosaicing 420 The registration 430 The input limitation 440 The color calibration 450 The luminance uniformity 460 The contrast enhancement 470 Further post - processing steps 500 Multiple lighting elements / lighting element arrays 502 Lighting element / LED 504 Color distribution curve 600 System 610 Microscope 620 Computer system 700 Color gradient R, G, B, R1, G1, B1, R2, G2, B2, R * , B * , G * Color space coordinates C… Matrix coefficients I Intensity x, y Pixel positions λ Wavelength a Aperture f Focal length / magnification d Working distance s Variable for lighting settings

Claims

1. An image processor (170) for a medical observation device (100) such as a microscope or an endoscope, wherein the image processor (170) includes a color conversion function (140), the image processor (170) is configured to: acquire an input pixel (150d) of a digital input color image (200) and a position (x, y) of the input pixel within the digital input color image (200), apply the color conversion function to the input pixel to form an output pixel (150c) in a digital output color image (160), wherein the color conversion function (140) depends on the position of the input pixel, Image processor (170).

2. The image processor is configured to: acquire at least one optical parameter (162, 164, 166) from a group including an optical parameter representing an operating distance (162) when the digital input color image is recorded, an optical parameter representing an aperture (164) when the digital input color image is recorded, an optical parameter representing a magnification (166) when the digital input color image is recorded, and an optical parameter representing a spatial color distribution of an illumination system, wherein the color conversion function (140) also depends on the at least one optical parameter, The image processor (170) according to claim 1.

3. The color conversion function (140) includes a color conversion matrix (142) having a first dimension and a second dimension, each of the input pixels (150d) includes color space coordinates (R1, R2, G1, G2, B1, B2) in an input color space (RGB), the first dimension of the color conversion matrix (142) corresponds to the number of color space coordinates (R, G, B) in the input color space (RGB), the second dimension of the color conversion matrix (142) corresponds to the number of color space coordinates (R, G, B) in the output color space (RGB), The image processor (170) according to claim 1 or 2.

4. Each of the output pixels (150c) has color space coordinates (R1 ＊ , R2 ＊ , G1 ＊ , G2 ＊ , B1 ＊ , B2 ＊ ) within an output color space (RGB), The color conversion matrix (142) includes at least one matrix element (C11,..., C66) that depends on at least one of the positions (x, y) of the input pixels (150d), The image processor (170) according to claim 3.

5. The color conversion matrix (142) includes at least one matrix element (C11,..., C66) that depends on the at least one optical parameter, ​ ​ ​ ​ The image processor (170) according to claim 3 or 4, which cites claim 2.

6. The at least one matrix element (C11,..., C66) includes at least one of a polynomial function, a spline function, and a multivariate interpolation function. The image processor (170) according to any one of claims 3 to 5.

7. The color conversion function (140) is configured to equalize the spatial color distribution, and the color gradient (700) across the digital input color image (200) is greater than the color gradient (700) across the digital output color image (160). The image processor (170) according to any one of claims 1 to 6.

8. The image processor is configured to acquire a digital white light color image (114) of the object (106) recorded in the first imaging spectrum (302), the digital white light color image (114) includes a plurality of first pixels (150a), and each first pixel (150a) includes a first set ({R1, B1, G1}) of color space coordinates ({R1, B1, G1}) within the color space (RGB). The image processor is configured to acquire a digital fluorescent color image (112) of the object (106) recorded in the second imaging spectrum (322), the digital fluorescent color image (112) includes a plurality of second pixels (150b), and each second pixel (150b) includes a second set ({R2, B2, G2}) of color space coordinates (R2, G2, B2) within the color space (RGB). The image processor is configured to form an input pixel (150d) of the digital input color image (200) from one of the first pixels (150a) and one of the second pixels (150b), and the input pixel includes a third set ({R1, R2, G1, G2, B1, B2}) of color space coordinates (R1, R2, G1, G2, B1, B2) within the color space (RGB). The image processor is configured to form the third set as the union of the first set and the second set. The image processor (170) according to any one of claims 1 to 7.

9. The image processor is configured to register the digital fluorescent color image (112) and the digital white light color image (114) before acquiring the input pixel. The image processor (170) according to claim 8.

10. A medical observation device (100) such as a microscope or an endoscope, wherein the medical observation device (100) The image processor (170) according to any one of claims 1 to 9; At least one color camera (110, 111); Including The medical observation device is configured to form a digital input color image (200) from at least one color image (112, 114) formed by the at least one color camera. Medical observation device (100).

11. The medical observation device includes at least A white light color camera (110) for recording the digital white light color image (114); A fluorescence color camera (111) for recording the digital fluorescence color image (112); Further including The medical observation device (100) according to claim 10.

12. A computer-implemented method for a medical observation device (100) such as a microscope or an endoscope, the computer-implemented method comprising: Obtaining input pixels (150d) of a digital input color image (200); Obtaining the positions (x, y) of the input pixels in the digital input color image (200); Applying a color conversion function (140) to the input pixels to form output pixels (150c) in a digital output color image (160); Including The color conversion function (140) depends on the position of the input pixels. Computer-implemented method.

13. A computer program product or computer-readable medium including instructions that cause a computer to execute the method according to claim 12 when the program is executed by the computer.

14. A method of operating a medical observation device (100) such as a microscope or an endoscope, the method including the computer-implemented method according to claim 12, the method further including: - Using a fluorescence color camera (111) to record a digital fluorescence color image (112) in a second imaging spectrum (322); - Using a white light color camera (110) to record a digital white light color image (114) in a first imaging spectrum (302); - Forming a digital input color image (200) from a combination of the digital white light color image and the digital fluorescence color image. A method further including.

15. The method includes - a step of recording the digital fluorescence color image (112) as a reflected image of the object (106); - a step of recording the digital white light color image (114) as a reflected image of the object (106); further comprising; The method according to claim 14.

Images