Data processing apparatus and computer-implemented method for combining two images and an overlay color using a uniform color space
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- LEICA INSTRUMENTS (SINGAPORE) PTE LTD
- Filing Date
- 2023-05-15
- Publication Date
- 2026-05-22
AI Technical Summary
Existing methods struggle to effectively mix background and secondary images while maintaining information integrity and avoiding artifacts, particularly when dealing with significantly different intensity levels like in fluorescence imaging.
A data processing apparatus and method that calculate the output color of an output pixel by mixing background and overlay colors based on light intensity values, using a uniform color space to ensure natural color transitions and prevent over-saturation.
The solution preserves information from both images, ensures natural color transitions, prevents over-saturation, and avoids non-linear dependencies, resulting in a more accurate and visually appealing combined image.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a data processing apparatus and a computer-implemented method for calculating an output color of an output pixel in an output color image by mixing a background color and an overlay color of a first pixel in a background color image depending on a light intensity value of a second pixel in a second image.
[0002] In some applications, it is desirable to mix two images and, in the resulting image, highlight information contained in one of the two images with a predefined overlay color that can be, for example, a false color or a pseudo color.
Background Art
[0003] One such use is, for example, fluorescence imaging used in medical observation devices such as microscopes and endoscopes. Here, a background color image can be recorded to provide a reflected image of the object. The reflected image can be used, for example, to show the anatomical structure of the area where the surgery is being performed. The object can include one or more phosphors that emit fluorescence in response to excitation, or one or more phosphors that emit fluorescence in response to excitation can be artificially added to the object. These phosphors can be used to highlight the region of interest. For example, some phosphors can be accumulated within the tumor so that the tumor can be more easily detected during fluorescence-guided surgery. Other phosphors can be used to highlight blood vessels. A fluorescence image can be recorded as a second image using only the wavelengths included in the fluorescence emission spectrum of at least one phosphor. In such a configuration, the background information and the fluorescence information are included in two different images. To combine (or "mix") the background image and the fluorescence image in order to bring such information together into a single image. This enables a surgeon or other medical personnel to identify the fluorescent regions within the background image. It is desirable for the region of interest emitting fluorescence to be immediately recognizable. Therefore, an overlay color that can correspond to the natural color or pseudo-color of the fluorescence is assigned to the fluorescence.
[0004] Mixing the background color image with the second image and the overlay color so that all information is visually maintained and no artifacts are introduced is a difficult process. Since the intensity of fluorescence is usually much lower than that of reflected light, special care needs to be taken to ensure that as much information as possible from the two images is retained in the combination of the fluorescence image and the background image.
[0005] Note that the present invention is not limited to a combination of images where the background image is a reflection image and the second image is a fluorescence image. There are situations where two reflection images of the same object recorded using different spectra or different wavelengths must be mixed. For example, in other applications, it may be required to combine a thermal image, a microwave image, or a radar image with the background image.
Summary of the Invention
Problems to be Solved by the Invention
[0006] In all of these situations, it is required to preserve as much information as possible by mixing the background image with the second image using the overlay color.
Means for Solving the Problems
[0007] To meet the above requirements, the data processing device acquires the background color coordinates of a background color having a predetermined maximum luminance value and a predetermined minimum luminance value in a uniform color space, acquires the overlay color coordinates of an overlay color in the uniform color space, acquires a light intensity value within a range extending from the predetermined maximum light intensity value including the predetermined maximum light intensity value to the predetermined maximum light intensity value including the predetermined maximum light intensity value, calculates a luminance line extending from the overlay color coordinates to an end point having a luminance lower than the luminance at the overlay color coordinates along a point having the chroma of the overlay color coordinates in the uniform color space, determines a first point on the luminance line, where the ratio of the difference between the predetermined maximum luminance value and the luminance of the background color coordinates to the difference between the luminance of the background color coordinates and the predetermined minimum luminance value corresponds to the ratio of the length of the luminance line from the first point to the overlay color coordinates to the length of the luminance line from the end point to the first point, calculates an overlay color line extending linearly from the background color coordinates to the first point, determines a second point on the overlay color line, where the ratio of the difference between the predetermined maximum light intensity value and the light intensity value to the difference between the light intensity value and the predetermined minimum light intensity value corresponds to the ratio of the length of the overlay color line from the second point to the first point to the length of the overlay color line from the background color coordinates to the second point, and outputs the color coordinates of the second point in the uniform color space as the output color of the output pixel.
[0008] The above requirements also include steps of obtaining background color coordinates of a background color within a uniform color space having a predetermined maximum luminance value and a predetermined minimum luminance value, obtaining overlay color coordinates of an overlay color within the uniform color space, obtaining luminance values within a range extending from a minimum luminance value including the minimum light intensity value to a maximum luminance value including the maximum light intensity value, calculating a luminance line extending from the overlay color coordinates to an end point having a luminance lower than the luminance at the overlay color coordinates along a point having the chroma of the overlay color coordinates in the uniform color space, determining a first point on the luminance line, where the ratio of the difference between the predetermined maximum luminance value and the luminance of the background color coordinates to the difference between the luminance of the background color coordinates and the predetermined minimum luminance value corresponds to the ratio of the length of the luminance line from the first point to the overlay color coordinates to the length of the luminance line from the end point to the first point, calculating an overlay color line linearly extending from the background color coordinates to the first point, determining a second point on the overlay color line, where the ratio of the difference between the maximum light intensity value and the light intensity value to the difference between the light intensity value and the minimum light intensity value corresponds to the ratio of the length of the overlay color line from the second point to the first point to the length of the overlay color line from the background color coordinates to the second point, and outputting the color coordinates of the second point in the uniform color space as the output color of the output pixel. This can also be addressed by a computer-implemented method.
[0009] By using the above combinations for each pixel of the background color image, several problems associated with conventional blending processes are solved by the second image and the overlay color. First, the color transitions better conform to the physical characteristics of human vision and appear more natural. Second, over-saturation is prevented. Third, non-linear dependencies that could lead to unexpected results are avoided.
[0010] A perceptually uniform color space is a color space in which a geometric distance corresponds to a perceived color difference. The uniform color space can be a color space from a group including tristimulus color spaces such as the CIE1931 color space, the CIELUV color space, the CIELAB color space, the HSLuv color space, the IPT color space, and color spaces derived from OKLab.
[0011] The predetermined maximum luminance value and the predetermined minimum luminance value may depend on the saturation of the background color. The maximum luminance value and the minimum luminance value for a given saturation are determined by the color space or, when considering the display device, by the color gamut of the display device. In this specification, the terms color and color coordinates are used synonymously. Color is represented by color space coordinates. Color space coordinates for the same color are different in different color spaces.
[0012] The maximum light intensity value and the minimum light intensity value can be determined by the predetermined color space or color gamut of the display device, especially in the case of neutral colors. For example, the maximum light intensity value can correspond to white, and the minimum light intensity value can correspond to black within the uniform color space or color gamut. Also, the maximum light intensity value and the minimum light intensity value can be determined by the camera that recorded the second image.
[0013] The term "output" includes both transmitting the output data and permitting access to the output data, for example, by permitting access to the memory in which the color coordinates of the second point are stored.
[0014] The terms "data processing device", "image processor", or "processor" are used synonymously in this specification.
[0015] Further features of the present invention are described below. Each of the following features is advantageous in itself and can be combined independently with any of the other of the following features.
[0016] Each of the following features can be used independently to improve a data processing apparatus or a computer-implemented method, even if each feature is only mentioned in the context of a data processing apparatus or only in the context of a computer-implemented method. More specifically, the data processing apparatus can be configured to execute each of the method steps even if not specifically mentioned in relation to the method steps described below.
[0017] According to one aspect, for a plurality or all of the pixels in the background image, in the second image and / or in the output image, the ratio of the luminance at the endpoint of the luminance line to the luminance of the overlay color may be constant. For example, the ratio can be set to a value between 0.1 and 0.4. That is, in this case, the luminance at the endpoint is 10% to 40% of the luminance of the overlay color. The ratio can be adjusted by the user and / or stored in the memory of the data processing apparatus.
[0018] The ratio of the luminance of the overlay color coordinates to the luminance at the endpoint of the luminance line may not depend, in particular, on the background color coordinates of the first pixel and / or the light intensity value of the second pixel. Preferably, the ratio is constant for all pixels in the background color image and the second image.
[0019] In one embodiment, the endpoint of the luminance line can be located within a predefined color gamut of a display device. Such a configuration ensures that the endpoint is properly displayed on a predefined display device. The predefined display device can be a VR device, an external monitor such as an LED screen or an OLED screen or a projector, or a monitor incorporated into an eyepiece of a medical observation device.
[0020] For example, if the endpoint calculated using a predefined ratio of the luminance of the overlay color coordinates is outside the color gamut, this endpoint can be automatically shifted along the luminance line to the position where the luminance line intersects the limit of the color gamut.
[0021] According to another embodiment, the luminance line can be a straight or curved line in a uniform color space. When one dimension of the uniform color space is luminance, the luminance line can extend parallel to this dimension.
[0022] When the overlay color coordinates are provided in a non-uniform color space, they must be converted to a uniform color space before or during acquisition. The data processing device can be configured to perform such a color space conversion. The function for color space conversion can be standardized or determined experimentally.
[0023] The non-uniform color space can be, for example, an RGB color space such as RGB, sRGB, AdobeRGB, Adobe White Gamut RGB, Rec.2100, DCI-P3, Rec.2020, Rec.709, Rec.601 or ACES. Other non-uniform color spaces are, for example, HSV, HSL, YUV and YCbCr and those derived from these.
[0024] When the background color coordinates are in a non-uniform color space, it is preferable to convert them to a uniform color space before or during acquisition. The data processing device can be configured to perform such a conversion.
[0025] The color coordinates of the second point on the overlay color line can be converted from a uniform color space to a non-uniform color space before output. This can be preferable when the display device cannot process the color coordinates of the uniform color space.
[0026] In one embodiment, the background color image and the second image can be registered images. In registered images, the pattern orientations, shapes, sizes, and positions match. Registered images are congruent and thus include corresponding pixels. Corresponding pixels, i.e., pixels representing the same area of the pattern, have the same position in both the background color image and the second image when these two are registered. This does not necessarily imply that the two images have the same pixel resolution. One pixel in one image may correspond to multiple pixels in the other image. For example, if one of the images has a higher resolution, a first pixel may correspond to multiple second pixels, or a second pixel may correspond to multiple first pixels. However, preferably, the background color image and the second image have the same format and the same amount of pixels. For example, the two images can be 1k, 2k, 4k, or 8k images.
[0027] Registration can be performed by a computer-implemented method and / or a data processing device.
[0028] Furthermore, it is preferable that the position of the output pixel corresponds to the position of at least one of the first pixel and the second pixel. The output color image can have the same format and the same number of pixels as either the background color image or the second image.
[0029] The acquisition of the background color coordinates can include the acquisition of a first pixel including the background color coordinates. Furthermore, the acquisition of the first pixel can include the acquisition of the background color image. For example, the background color image can be loaded into the memory of the data processing device, either in its entirety or pixel by pixel.
[0030] The acquisition of the light intensity value can include the acquisition of a second pixel including the light intensity value. Further, the acquisition of the second pixel can include the acquisition of a second image. For example, the second image can be loaded into the memory of the data processing device, either in its entirety or pixel by pixel.
[0031] In one embodiment, the second image may be one of a grayscale image and a color image. When the second image is a grayscale image, the grayscale value may already correspond to the light intensity value. In this case, the value can be used to determine the position of the second point on the overlay color line. The second image can be recorded by a grayscale camera or a color camera.
[0032] When the second image is a color image, the acquisition of the light intensity value can include the calculation of the light intensity value from the color coordinates of the second pixel. The calculation may be part of the acquisition of the light intensity value. The calculation of the light intensity value here can be performed in any color space, i.e., a non-uniform color space or a uniform color space, using a standardized function or a function determined experimentally.
[0033] The above-described calculations of the luminance line and the overlay color line are preferably repeated for a plurality or all of the pixels in the background color image and the second image. The overlay color coordinates preferably do not depend on the first pixel and the second pixel. The overlay color coordinates can be, for example, constant values that can be changed by the user.
[0034] As initially explained, as one field of application of a data processing apparatus and / or a computer-implemented method, fluorescence imaging can be cited, where a background color image can be recorded in a first imaging spectrum and can include a plurality of first pixels. For example, the background color image can be a digital white light color image. A second image can be recorded in a second imaging spectrum and can similarly include a plurality of second pixels. The second imaging spectrum can overlap with the fluorescence emission spectrum of at least one phosphor. Thus, the fluorescence of at least one phosphor can be recorded in the second imaging spectrum. For example, the second image can be a digital fluorescence image. Further, the second imaging spectrum can be different from the first imaging spectrum. For example, the first imaging spectrum and the second imaging spectrum can be complementary. The first imaging spectrum and the second imaging spectrum can include both a plurality of stop bands and a plurality of pass bands. The stop bands of the first imaging spectrum can be made to correspond to the pass bands of the second imaging spectrum, and the pass bands of the first imaging spectrum can be made to correspond to the stop bands of the second imaging spectrum. In one embodiment, the background color image does not record wavelengths within the fluorescence emission spectrum of at least one phosphor.
[0035] Both the first imaging spectrum and the second imaging spectrum may overlap with the visible spectrum.
[0036] The above configuration can be particularly useful in a medical observation apparatus such as a microscope or an endoscope. The microscope can be a laboratory microscope or a surgical microscope.
[0037] Medical observation devices such as microscopes or endoscopes may include the data processing device in any of the above embodiments. The medical observation device may further include a white light color camera configured to record a background image, and a fluorescence camera configured to record a second image. The fluorescence camera may be a grayscale camera or a color camera. The term "color camera" includes CCD cameras, CMOS cameras, multispectral cameras, hyperspectral cameras, and all other types of cameras that record color images.
[0038] A method of operating a medical observation device includes the computer-implemented method in any of the above-described embodiments, and may further include the step of recording a background color image of an object using a white light color camera, and the step of recording a second image of the object using a fluorescence camera.
[0039] The subject matter recited in the claims also relates to a computer-readable medium and a computer program including instructions for causing a computer to perform image processing implemented by a computer in any of the above-described embodiments.
[0040] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items, and may be abbreviated as " / ".
[0041] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent descriptions of corresponding methods, where a block or apparatus corresponds to a step or a feature of a step. Similarly, aspects described in the context of a step also represent descriptions of corresponding blocks or items or features of a corresponding apparatus.
[0042] The present invention will be illustratively described below with reference to embodiments and the drawings. The combinations of features shown in these embodiments should not be construed as limiting. For example, among the embodiments having the technical effects described above as examples, features that are not required in a specific application can be omitted. Conversely, among the features described above, for those that are not part of the embodiments described later, if the technical effects related to the specific features are required in a specific application, they can also be added.
[0043] Throughout this specification and the drawings, the same reference numerals are used for elements that correspond to each other with respect to function and / or structure. The drawings show the following.
Brief Description of the Drawings
[0044]
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Modes for Carrying Out the Invention
[0045] First, with reference to FIG. 1, an exemplary embodiment of the present invention will be described.
[0046] Figure 1 schematically shows a medical observation device 100. The medical observation device 100 may be a fluorescence microscope or a fluorescence endoscope. The main difference between a microscope and an endoscope is that in an endoscope (not shown), for example, an object 106 to be inspected is observed by an optical fiber drawn close to the object 106 to be inspected, for example, by insertion into the body, while in a microscope, the objective lens 174 is oriented towards the object 106. Although the medical observation device in Figure 1 is shown as a microscope, the following description also applies to an endoscope. The medical observation device 100 may be, for example, a medical observation device used in a surgical operation. Also, the medical observation device 100 may be a medical observation device used in a laboratory, for example, a laboratory microscope. The object 106 to be inspected may consist of a biological tissue 107 or include a biological tissue 107, but may also include an inorganic substance or consist of an inorganic substance.
[0047] The object 106 can include one or more phosphors 116, 118. At least one phosphor may be a phosphor inherently contained in the object. For example, bone and blood contain phosphors. Also, at least one phosphor may be, for example, one that can be added to the object 106 by injecting it into the biological tissue 107. Examples of phosphors that can be added to the object 106 include ICG, fluorescein, and / or 5-ALA. 5-ALA is synthesized into pPIX in cells.
[0048] The medical observation device 100 can be configured to record the fluorescence of one or more phosphors 116, 118. This means that the medical observation device 100 can be configured to observe, record, and preferably also excite the fluorescence of one or more phosphors 116, 118.
[0049] The medical observation device 100 can be a stereomicroscope device exemplarily shown in FIG. 1. Accordingly, the medical observation device 100 can have two identical sub-assemblies 101L and 101R for each of the two stereo channels. Since the two sub-assemblies 101L and 101R are identical in function and structure, the following description focuses on the right sub-assembly 101R, but is equally applicable to the left stereo channel 101L.
[0050] Alternatively, the medical observation device 100 may be a planar viewing device. In this case, only one of the two sub-assemblies 101L and 101R may exist. Accordingly, the following description is equally applicable to the medical observation device 100 capable of planar viewing.
[0051] In one embodiment, the medical observation device 100 can be used to form one or more digital white light color images 114. The digital white light color image can particularly represent a reflected image of the object 106 over the visible light range. The visible light region or visible spectrum includes wavelengths from about 310 nm to about 1100 nm or wavelengths from about 380 nm to 750 nm or wavelengths from about 450 nm to about 700 nm. The fluorescence spectrum, or when two or more phosphors are used, the fluorescence spectra of the phosphors 116 and 118 are preferably excluded from the spectrum recorded in the digital white light color image 114. Thereby, when fluorescence is present, it is ensured that only the reflected light is included in the digital white light color image 114. For example, when 5-ALA / pPIX is used as the phosphor, the fluorescence spectrum from about 625 nm to about 650 nm may not be recorded in the digital white light color image 114.
[0052] When light of a specific wavelength is used to excite fluorescence, a spectrum containing or consisting of these wavelengths may not be recorded or represented in the digital white light color image 114. For example, when 5-ALA / pPIX is used as a phosphor, fluorescence can be excited by illuminating the object 106 with wavelengths from approximately 380 nm to approximately 450 nm. For fluorescein and ICG and other phosphors, known ranges different from those for 5-ALA / pPIX apply to the excitation spectrum and the emission spectrum.
[0053] When the fluorescence excitation spectrum and the fluorescence emission spectrum are not recorded, the digital white light color image 114 represents the reflection at the object 106, i.e., the white light image of the object 106 as observed by a human observer. In such a configuration, the digital white light color image 114 can be regarded as a natural color or pure color image.
[0054] The digital imaging system 102 can also be used to form one or more digital fluorescence images 112 of the object 106. The digital fluorescence images 112 can represent the fluorescence emission of one or more phosphors 116, 118. The digital fluorescence images 112 can be grayscale or color images.
[0055] The digital fluorescence images 112 preferably do not record wavelengths outside the emission spectrum or the emission spectrum of one or more phosphors. The digital fluorescence images 112 can be a reflection image of the object 106 or can include fluorescence and reflected light from the object.
[0056] In one embodiment, both the digital white light color image 114 and the digital fluorescence image 112 may be color images. The digital white light color image 114 and the digital fluorescence image 112 can be recorded using at least three color bands or the primary colors of a color space as an equivalent. For example, both the digital white light color image 114 and the digital fluorescence image 112 can be recorded in the RGB color space using the three primary colors or color bands R, G, B. Alternatively, the digital white light color image 114 and the digital fluorescence image 112 can be recorded in different color spaces, respectively, each of which can represent a multi-spectral color image or a hyper-spectral color image. Preferably, the digital white light color image 114 and the digital fluorescence image 112 are recorded in the same color space, but this is not essential.
[0057] At least one digital white light color image 114 and at least one digital fluorescence image 112 each include pixels 150a, 150b. In a color space, such as the RGB color space, each color of the pixels 150a, 150b is represented by color space coordinates that form a triplet of three integers. Each integer indicates the intensity of one of the primary colors R, G, B. For example, the strongest red can be indicated by the triplet {255, 0, 0}. The strongest green can be indicated, for example, by {0, 255, 0}, and the strongest blue can be indicated, for example, by {0, 0, 255}. Thus, the RGB color space is a three-dimensional space. The CMYK color space is a four-dimensional space. A color can be regarded as a point within a color space having color space coordinates, such as {0, 0, 255}. More generally, an n-dimensional color space is obtained from a multi-spectral color space or a hyper-spectral color space having n corresponding color bands, and each color is represented by n tuples of the color space.
[0058] Examples of the RGB color space are RGB, sRGB, Adobe RGB, Adobe White Gamut RGB, REC.2100, DCI-P3, Rec.2020, Rec.709, Rec.601, and ACES.
[0059] The spectrum recorded and represented in the digital white light color image 114, i.e., the first imaging spectrum, and the spectrum recorded in the digital fluorescence image 112, i.e., the second imaging spectrum, are preferably complementary to each other, i.e., they do not overlap except for the inevitable filter cage. Preferably, both of these spectra represent the complete visible light spectrum or at least a majority of the visible light range. The first image and / or the second image may include wavelengths outside the visible spectrum, such as NIR.
[0060] More specifically, the medical observation device 100 can include a digital imaging system 102 that forms the digital fluorescence image 112 and the digital white light color image 114. The digital imaging system 102 can include a white light color camera 110 and a fluorescence emission camera 111.
[0061] The white light camera 110 is configured to record the digital white light color image 114. In particular, the white light color camera 110 can be configured to form a stream of the digital white light color image 114 in the form of a digital video stream. The white light color camera 110 is preferably configured to record digital images over the entire visible spectrum of the wavelengths shown above. The white light color camera 110 can be a CCD, CMOS, or multi-spectrum camera or a hyperspectrum camera.
[0062] The fluorescence camera 111 is configured to record a digital fluorescence image 112. In particular, the fluorescence camera 111 can be configured to form a stream of the digital fluorescence image 112 in the form of a digital video stream. The fluorescence camera 111 can be configured to record the digital fluorescence image 112 only in one or more fluorescence spectra of at least one phosphor 116, 118. The fluorescence camera 111 can be configured to record the digital fluorescence image 112 with only one or more narrow-band lights. Here, the narrow band is made to overlap with one or more fluorescence spectra of one or more phosphors 116, 118 for which fluorescence is to be recorded. Preferably, the fluorescence spectrum of the phosphor 116 and the fluorescence spectrum of the second phosphor 118 do not overlap, whereby the fluorescence camera 111 can record lights in two separate spectral fluorescence bands spaced apart from each other.
[0063] The fluorescence camera 111 may be a CCD camera, a CMOS camera, or a multispectral camera or a hyperspectral camera or any other suitable type of camera. Preferably, the white light color camera 110 and the fluorescence camera 111 are of the same type of camera, but this is not essential. Since fluorescence usually has a very low intensity, the fluorescence camera 111 can have a larger integration time.
[0064] The respective fields of view 184 of the cameras 110, 111 are preferably aligned, or even more preferably coincide and are coaxial. Thus, preferably, the cameras 110, 111 provide the same field of view 184 with the same viewpoint and the same focal length. As a result, the same object 106 can be represented in the images 112, 114 formed by the different cameras 110, 111. The two cameras 110, 111 can use the same objective lens 174. In such a configuration, the images 112, 114 can be optically registered, whereby registration by image processing becomes unnecessary or is significantly simplified.
[0065] As will be further described below, when the coincidence of viewpoints and fields of view cannot be optically formed, the matching or registration routine can be applied to the digital images 112, 114 to form the coincidence here by image processing. When the cameras 110, 111 have the same viewpoints and fields of view, the application of a computer-implemented registration routine may be recommended.
[0066] It is preferable that the two cameras 110, 111 operate synchronously. Specifically, the exposure times can be synchronized. Accordingly, the medical observation device 100 can be configured to simultaneously form a digital white light color image 114 and a digital fluorescence image 112.
[0067] Preferably, the gains of the two cameras 110, 111 are synchronized, that is, adjusted simultaneously in the two cameras 110, 111. Further, the ratio of the gain applied to camera 110 to the gain applied to camera 111 can be made constant even when the gains vary. Gamma correction and color adjustment or white balance may be switched off or may be kept constant.
[0068] Any of the above measures facilitates the comparison, integration processing, and / or combination of the two images 112, 114.
[0069] In order to separate the spectrum recorded in the digital white light color image 114 from the spectrum recorded in the digital fluorescence image 112, that is, to separate the reflection spectrum from the fluorescence spectrum, the medical observation device 100 can include an optical color separation assembly 176. The color separation assembly 176 can include optical elements, such as a beam splitter 192 which can be dichroic. The color separation assembly 176 can additionally or alternatively include an optical white light filter 188 and / or an optical fluorescence filter 190.
[0070] The fluorescence filter 190 is preferably configured to transmit light of one or more fluorescence spectra of one or more phosphors 116, 118 and block light outside the one or more fluorescence spectra.
[0071] The fluorescence filter 190 can be configured as a band - pass filter including one or more pass - bands. Each pass - band is made to overlap with the fluorescence emission spectrum of each phosphor 116, 118 for which fluorescence is to be recorded. Since the fluorescence filter 190 is present in the optical path between the beam splitter 192 and the fluorescence camera 111, only the light of the wavelengths of the pass - bands of the fluorescence filter 190 is transmitted to the fluorescence camera 111.
[0072] The white - light filter 188 is preferably configured to block light of one or more fluorescence spectra of one or more phosphors 116, 118. Also, the white - light filter 188 can be configured to block light in the fluorescence excitation spectrum.
[0073] The white - light filter 188 is preferably configured as a band - stop filter having a stop - band corresponding to or at least including the pass - band of the fluorescence filter 190. The white - light filter 188 is arranged in the optical path between the beam splitter 192 and the white - light camera 110. Accordingly, the white - light camera 110 records only wavelengths that are outside the stop - band of the white - light filter 188 and thus also outside the pass - band of the fluorescence filter 190.
[0074] Either the white - light filter 188 or the fluorescence filter 190 may be an adjustable filter.
[0075] When the beam splitter 192 is a dichroic beam splitter, in this example, since optical spectral filtering is already incorporated in the dichroic beam splitter, at least one of the filters 188 and 190 can be omitted. In this case, the above description regarding the passband and stopband shall apply mutatis mutandis to the dichroic beam splitter 192 after making necessary modifications.
[0076] Accordingly, the white light color camera 110 records the digital white light color image 114 in the first imaging spectrum, which is different from the second imaging spectrum recorded as the fluorescence spectrum by the fluorescence camera 111. The wavelengths included in the first imaging spectrum and the second imaging spectrum are determined by the filter settings of the color separation assembly 176.
[0077] The medical observation device 100 can further include an illumination assembly 178, which is preferably configured to illuminate the object 106 through the same objective lens 174 through which the imaging system 102 records at least one digital image 112, 114. The illumination assembly 178 can be configured to selectively generate white light, i.e., light uniformly dispersed over the entire visible spectrum, and fluorescence excitation light including only light of wavelengths that stimulate the fluorescence of at least one phosphor 116, 118. The illumination light generated by the illumination assembly 178 can be supplied to the objective lens 174 using the illumination beam splitter 180.
[0078] Depending on the phosphor and its specific excitation spectrum of fluorescence, an illumination filter 179 can be provided. For example, when 5-ALA / pPIX is used as the phosphor, the illumination filter 179 can have a transmittance of 90% - 98% up to a wavelength of 425 nm, a transmittance of 0.5% - 0.7% at wavelengths of 450 nm - 460 nm, a transmittance of 0.1% or less at wavelengths of 460 nm - 535 nm, and a transmittance of substantially zero at wavelengths above 535 nm.
[0079] Instead of or in addition to the illumination filter 179, the illumination assembly 178 can include an adjustable light source, such as a plurality of LEDs or OLEDs each colored differently.
[0080] The medical observation device 100 can further include a data processing device 170. The data processing device 170 can be a hardware module or a software module, such as a microprocessor. The data processing device 170 can also be a combination of both a hardware module and a software module, for example, by using a software module configured to be executed on a specific processor, such as on a vector processor, a floating-point graphic processor, a parallel processor, and / or a multi-processor. The data processing device 170 can be a part of a general-purpose computer 186, such as a PC.
[0081] The data processing device 170 is configured to acquire a digital white light color image 114 and a digital fluorescence image 112. For example, the data processing device 170 can be configured to acquire the digital white light color image 114 and the digital fluorescence image 112 directly from the memory 194 and / or from the cameras 110, 111. The memory 194 can be a part of the data processing device 170 or can be located elsewhere in the medical observation device 100.
[0082] The data processing device 170 is further configured to calculate a digital output color image 160 from the digital white light color image 114 and the digital fluorescence image 112. The digital output color image 160 is a color image represented in a color space that may be different from the color space of either the digital white light color image 114 or the digital fluorescence image 112. Preferably, the color space of the digital output color image 160 is the same color space as the color spaces of the digital white light color image 114 and the digital fluorescence image 112. A standardized function can be provided or determined by experiment or analysis for the conversion of coordinates from one color space to another. In the case of fluorescence, the fluorescence intensity, i.e., luminance, can be used. If the fluorescence image is a grayscale image, the intensity value or luminance value is directly recorded in the fluorescence image. If the fluorescence image is a color image, the intensity or luminance can be calculated using a function suitable for the color space.
[0083] The data processing device 170 is preferably configured to form the digital output color image 160 from the digital fluorescence image 112 and the digital white light color image 114.
[0084] In one embodiment, the digital white light color image 114 can be used as a background color image overlaid with the fluorescence image 112 as a second image. In another embodiment, the digital fluorescence image 112 can be used as the background image, and in this case, the second pixel 150b corresponds to a pixel from the digital white light color image 114.
[0085] The data processing device 170 can be configured to combine the second image 112 and the background color image 114 and use the color conversion function 140 to form the digital output color image 160. The formation of the output color image 160 can be performed pixel by pixel, where the first pixel 150a of the background color image 114 is combined with the second pixel 150b from the second image 112 using the color conversion function 140, thereby forming the output pixel 150c.
[0086] The first pixel 150a and the second pixel 150b are preferably corresponding pixels. Thus, in the registered images 112, 114, the first pixel 150a and the second pixel 150b can be located at the same position within their respective images 112, 114.
[0087] The number of pixels within the images 112, 114 does not necessarily have to be the same. For example, pixels within one of the images 112, 114 having a high resolution can be made to correspond to a plurality of pixels within the other image 114, 122 having a low resolution. Alternatively, two pixels located at the same position within the same pattern identified by a pattern recognition algorithm in both images 112, 114 can also be regarded as corresponding pixels.
[0088] In the context of a surgical operation, the background color image 114 can represent an anatomical background image with fluorescent information overlaid thereon. When attempting to represent the fluorescence recorded in the second image 112 by a predetermined overlay color 152, for example a pseudo-color, the color of the output pixel 150c can be calculated using the color of the first pixel 150a, the luminance of the second pixel 150b, and the predetermined overlay color.
[0089] The data processing device 170 can be configured to output the digital output color image 160 to any type of peripheral device. In this context, outputting includes, for example, transmitting the digital output color image 160 and / or enabling access to the digital output color image 160 by permitting access to a memory, such as memory 194, capable of storing the digital output color image 160, or output to the output interface 172 of the medical observation device 100.
[0090] The digital output color image 160 can be displayed on the display device 132 integrated with the medical observation device 100. For example, the display device 132 can be incorporated into the eyepiece or viewfinder 104 of the medical observation device 100. The display device 132 can also display a graphic user interface for operating the medical observation device 100.
[0091] The overlay color 152 can be stored in the data processing device 170 and / or the computer 186 and can be changed according to the user's preferences.
[0092] The medical observation device 100 can include a direct optical path 134 that extends from the object 106 through the objective lens 174 to the eyepiece 104. In this case, the display device can be a semi-transparent display device disposed within the direct optical path 134, or a display device capable of projecting into the direct optical path 134. A beam splitter 136 can be provided to split light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light can be directed to the eyepiece 104.
[0093] In another embodiment, the medical observation device 100 does not necessarily have a direct optical path 134 and may only display an image using the integrated display device 132. As a further alternative, the medical observation device 100 may not need to have any integrated display device.
[0094] The medical observation device 100 can include an output interface 172 to which one or more (external) display devices 182 and / or any type of unidirectional or bidirectional wired or wireless data connection can be connected. For this purpose, the output interface 172 can include a standardized connector and data transmission protocol, such as WLAN, TCP / IP, Ethernet, USB, HDMI, DVI, DisplayPort, Bluetooth and / or others. The external display device can be a monitor, 3D goggles, glasses, etc. Any combination of external display devices can be connected to the output interface 172. Any of the display devices 182 can display a graphic user interface for operating the medical observation device 100.
[0095] The computer 186 or data processing device 170 can be connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission lines can be wired or wireless, or partially wired or partially wireless. The computer 186 and / or data processing device 170 are not physically incorporated into the medical observation device 100 and can be physically located at a position remote from the digital imaging system 102. For this purpose, the digital imaging system 102 and the computer 186 and / or data processing device 170 can be connected to a network, such as a LAN, WLAN or WAN, and at least one display device 182 can be further connected to this network. The network connection can be provided by the output interface 172.
[0096] According to one modification, the medical observation device 100 is a stereomicroscope device, but may include only two cameras, one for each stereochannel. In one stereochannel, a fluorescence camera 111 is used and is further configured to selectively record white light reflection as well. In the other stereochannel, a white light color camera 110 is used. With such an arrangement, when fluorescence is not used, a stereoscopic white light color image is provided, and when fluorescence is used, a white light color image for planar viewing and a fluorescence image for planar viewing are provided. The above and below descriptions similarly apply to this configuration.
[0097] According to another configuration, each stereochannel may include three or more cameras.
[0098] FIG. 3 shows a quantitative example of the first imaging spectrum 302 as recorded by the digital white light camera 110 and / or as represented in the digital white light color image 114. The intensity I over the wavelength / color λ is shown normalized. The first imaging spectrum 302 preferably extends over at least the visible spectrum 312.
[0099] By way of example only, the color space in which the first imaging spectrum 302 is recorded can be an RGB color space having three primary colors or color bands 304, 306, 308. One primary color 304 is blue, another primary color 306 is green, and the third primary color 308 is red. The sensitivities of the sensors of the white light color camera 110 at the respective different primary colors 304, 306, 308 are adjusted, and as a result, a sensitivity over as constant a visible spectrum 312 as possible is obtained.
[0100] When a color space other than RGB is used, the amount, position, and / or width of the color bands may be different.
[0101] The first imaging spectrum 302 preferably does not include the fluorescence excitation light and the fluorescence emission spectrum of at least one phosphor 116, 118. Accordingly, the first imaging spectrum 302 can include at least one stop band 310 that coincides with the fluorescence emission of at least one phosphor whose fluorescence is recorded by the fluorescence camera 111. The stop band 310 is formed, for example, by the white light filter 188. The number, width, and / or position of the stop band 310 depend on the number and type of phosphors observed in the object 106.
[0102] As an example of the second imaging spectrum 322, it has been shown that it can be recorded by the fluorescence camera 111 and / or can be represented in the digital fluorescence image 112. By way of example only, the color space in which the second imaging spectrum 322 is recorded is also the RGB color space. The sensitivities of the sensors of the fluorescence camera 111 in the respective different primary colors 304, 306, 308 are adjusted so that as constant a sensitivity as possible is obtained over the visible spectrum 312.
[0103] Preferably, the spectra 302, 322 are recorded in the same color space, but this is not essential.
[0104] The second imaging spectrum 322 can include one or more pass bands 324. The number, position, and / or width of the pass band depend on the number and type of phosphors used. At least one pass band 324 preferably corresponds to at least one stop band 310. At least one pass band can be formed, for example, by the fluorescence filter 190.
[0105] The first imaging spectrum 302 and the second imaging spectrum 322 are complementary to each other. The first imaging spectrum 302 and the second imaging spectrum 322 are preferably completed with each other in order to cover the whole, most, or almost all of the visible spectrum 312.
[0106] Each passband 324 of the second imaging spectrum 322 preferably overlaps with the fluorescence emission spectra 326 of the phosphors 116, 118 where fluorescence is recorded and also overlaps with one or more primary colors 304, 306, 308 of the color spectrum. For example, the fluorescence emission spectrum 326 of the phosphor 116 can overlap with all of the three primary colors 304, 306, 308.
[0107] Referring to FIG. 2, next, a process of combining the images 112, 114 using the overlay color 152 will be described. The data processing device 170 is configured to execute any step of the process.
[0108] To obtain color mixing from the background color and the overlay color 152 of the first pixel 150a that most closely corresponds to human color perception, preferably, the color mixing is performed in a uniform color space, such as the CIELAB color space, CIELUV color space, HSLuv color space, IPT color space, OKLab color space, and the three - stimulus color space.
[0109] In the case where the digital white - light color image 114 forming the background - color image in this example is recorded in a non - uniform color space, such as the RGB color space, in step 210, a conversion to a uniform color space is performed. Thus, in step 210, a color - space conversion to a uniform color space is executed for the first pixel 150a. The background - color coordinates 202 of the first pixel 150a in the uniform color space are {X BC , Y BC , Z BC}, where X, Y, Z are, for example, coordinates L, a, b in the CIELAB color space and coordinates L, U, V in the CIELUV color space.
[0110] If the fluorescence image 112 is a color image recorded in a non-uniform color space, such as the RGB color space, the same is done for the fluorescence image 112 as the second image. In step 210, a color space conversion is performed on the second pixel 150b. The color space coordinates of the first pixel 150a and the second pixel 150b must be converted to the same uniform color space. If the fluorescence image is a grayscale image, the color space conversion is not necessary.
[0111] Again, if the fluorescence image is a color image, the luminance value at the second pixel 150b is calculated. Note that the luminance calculation 212 can also be performed before the color space conversion 210. Thus, for the second pixel 150b, the light intensity value I 2 is obtained.
[0112] The overlay color coordinates 200 of the overlay color 152 are {X OC , Y OC , Z OC} in a uniform color space. Of course, if the overlay color coordinates do not exist in the uniform color space of the first pixel 150a and the second pixel 150b, the overlay color coordinates can be converted to a uniform color space.
[0113] In step 250, a color conversion is performed using the overlay color coordinates 200, the light intensity value 204, and the background color coordinates 202, and as a result, the output color coordinates 206 for the output pixel 150c, that is, the tuple {X O , Y O , Z O} is obtained. The output pixel 150c is preferably placed at the same position in the output color image 160 as the first pixel 150a and the second pixel 150b in the respective corresponding images 112, 114.
[0114] Next, the output color coordinates 206 can be converted to a color space suitable for the output device, such as the display devices 132, 182, for example, the RGB color space again. Thus, the color space conversion 210 is performed again.
[0115] Referring to FIG. 4, the color conversion 250 performed by the color conversion function 140 will be described in more detail. FIG. 4 shows a single uniform color space 400, specifically the CIELAB color space by way of example.
[0116] First, a luminance line 402 is calculated. The luminance line 402 starts from the overlay color space coordinates 200 and extends continuously along a point where only the luminance changes while the chroma remains constant. In a uniform color space with luminance as the coordinate, such as CIELAB, the luminance line 402 extends along the luminance coordinate. In another uniform color space, the luminance line 402 may not be straight.
[0117] The end point 406 of the luminance line 402 is determined by, for example, a predetermined value of the luminance related to the luminance of the overlay color space coordinates 200. For example, the end point 406 can be arranged at a location where the luminance is 20% of the luminance of the overlay color space coordinates 200 at the same chroma. Of course, any other predetermined ratio can also be used.
[0118] Using a predetermined luminance ratio, when the end point 406 is located outside the color gamut 452 of one of the predetermined display devices, such as display devices 132, 182, the end point 406 is located at the intersection of the luminance line 402 and the boundary of the color gamut 452.
[0119] Next, a first point 408 is calculated on the luminance line 402. The position of the first point 408 along the luminance line 402 corresponds to a position within the range of possible luminance values in the uniform color space 400 or the luminance 444 of the background color 202 within the color gamut with respect to the chroma of the background color. The range here extends between a predetermined maximum luminance value 440 and a predetermined minimum luminance value 442, including each of these values. The maximum luminance value 440 and / or the minimum luminance value 442 may depend on the chroma of the background color.
[0120] More specifically, the ratio of the distance 410 from the first point 408 to the overlay color space coordinates 200 to the distance 412 from the first point 408 to the end point 406 is the difference between the maximum luminance value 440 and the luminance of the background color coordinates {XBC , Y BC , Z BC} and is the same as the ratio of the difference between the minimum luminance value 442 and the luminance value of the background color coordinates.
[0121] For example, when the maximum luminance value 440 of the saturation of the background color is 251 in the color gamut 452, the minimum luminance value 442 of the saturation of the background color is 3 in the color gamut 452, and the luminance of the background color 202 is 134, the ratio of the length 410 to the length 412 is (251 - 134) / (134 - 3). The brighter the background color, the closer the first point 408 is to the overlay color coordinates 200.
[0122] When the first point 408 is determined, the overlay color line 404 is calculated. The overlay color line 404 extends linearly from the background color coordinates 202 to the first point 408 in an arbitrary uniform color space 400. The output color coordinates 206 are determined as the coordinates of the second point 414 located on the overlay color line 404.
[0123] The position of the second point 414 depends on the light intensity value 204 of the second pixel 150b. The distance from the second point 414 to the background color coordinates 202 corresponds to the relative intensity of the light intensity value 204 with respect to the maximum intensity range. The higher the light intensity value 204, the closer the second point 414 is to the luminance line 404. The light intensity value 204 is in the range from the maximum light intensity value 446 to the minimum light intensity value 448, and the maximum and minimum values are included in the range.
[0124] In particular, the ratio of the difference between the maximum light intensity value 446 and the light intensity value 204 and the difference between the light intensity value 204 and the minimum light intensity value 448 corresponds to the ratio of the length 420 of the overlay color line 404 from the second point 414 to the first point 408 and the length 422 of the overlay color line 404 from the background color coordinates 202 to the second point 414.
[0125] For example, when the maximum light intensity value 446 of the first pixel 150a is 1024, the minimum light intensity value 448 of the first pixel is 0, and the light intensity value of the second pixel 150b is 347, the ratio of the length 420 to the length 422 is (1024 - 347) / (347 - 0).
[0126] The maximum light intensity value and the minimum light intensity value can be determined by a predetermined color space or color gamut of a display device, or alternatively, by the range of a camera that records the second image. The maximum intensity value 446 and / or the minimum intensity value 448 can represent intermediate colors. For example, the minimum light intensity can be corresponded to black, and / or the maximum light intensity can be corresponded to white within a uniform color space and / or within a uniform color gamut.
[0127] The minimum luminance value and the maximum luminance value and / or the maximum intensity value and the minimum intensity value can be stored in the memory 194.
[0128] Although the medical observation device 100 has been described in the above description, the mixing method and the data processing device 170 are not limited to these devices. For example, instead of expressing fluorescence, the second image can also express reflection, IR light, microwaves, or other electromagnetic waves outside the visible spectrum. Furthermore, other image sources, such as X-ray images or MR images, can also be used as the background or the second image.
[0129] FIG. 5 shows a schematic diagram of a system 500 configured to implement the method described herein. The system 500 includes a microscope 510 and a computer system 520. The microscope 510 is configured to image and is connected to the computer system 520. The computer system 520 is configured to implement at least a portion of the method described herein. The computer system 520 may be configured to execute a machine learning algorithm. The computer system 520 and the microscope 510 may be separate entities or may be integrated within a single common housing. The computer system 520 may be part of the central processing system of the microscope 510 and / or the computer system 520 may be part of a sub-component of the microscope 510, such as a sensor, actuator, camera, or illumination unit of the microscope 510.
[0130] The computer system 520 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed at various locations such as local clients and / or one or more remote server farms and / or data centers). The computer system 520 may include any circuit or combination of circuits. In one embodiment, the computer system 520 may include one or more processors, which can be of any type. As used herein, a processor may contemplate any kind of computing circuit, such as, for example, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, but is not limited thereto. Other types of circuits that may be included in the computer system 520 may be custom circuits, application specific integrated circuits (ASICs), etc., such as, for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 520 may include one or more storage devices that may include one or more storage elements suitable for specific applications, such as main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives for handling removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 620 may include a display device, one or more speakers and a keyboard and / or a mouse, a trackball, a touch screen, a controller that may include a voice recognition device, or any other device that enables a user of the system to input information into the computer system 620 and to receive information from the computer system 620.
[0131] Some or all of the steps may be performed by, or using, a hardware device such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the critically important steps may be performed by such a device.
[0132] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is executable by a non-transitory recording medium, which is, for example, a digital recording medium such as a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, in which electronically readable control signals are stored that cooperate (or are capable of cooperating) with a programmable computer system to perform each method. Thus, the digital recording medium may be computer-readable.
[0133] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so that any of the methods described herein are performed.
[0134] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code, which operates to perform any method when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.
[0135] Another embodiment includes a computer program for implementing any of the methods described herein, stored on a machine-readable carrier.
[0136] Thus, in other words, an embodiment of the present invention is a computer program having program code for implementing any of the methods described herein when the computer program is executed thereon.
[0137] Thus, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for implementing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, including a processor and a recording medium.
[0138] Thus, another embodiment of the present invention is a data stream or signal sequence representing a computer program for implementing any of the methods described herein. The data stream or signal sequence may be configured to be transferred, for example, via a data communication connection such as the Internet.
[0139] Another embodiment includes a processing means, for example, a computer or programmable logic device configured or adapted to implement any of the methods described herein.
[0140] Another embodiment includes a computer having an installed computer program for implementing any of the methods described herein.
[0141] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.
[0142] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to execute some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to implement any of the methods described herein. Generally, and advantageously, the methods are implemented by any hardware device.
Explanation of Reference Numerals
[0143] 100 Medical observation device 101L Stereo sub-assembly for left channel 101R Stereo sub-assembly for right channel 102 Digital imaging system 104 Eyepiece 106 Object to be inspected 107 Biological tissue 110 White light color camera 111 Fluorescent camera (color or grayscale) 112 Fluorescent image / Second image (color or grayscale) 114 White light image / Background color image 116 Phosphor 118 Second phosphor 132 Integrated display device / Internal image display device 134 Direct optical path 136 Beam splitter 140 Color conversion function 150a First pixel in background color image 150b Second pixel in the second image 150c Output pixel in the output color image 152 Overlay color 160 Output color image 170 Data processing device 172 Output interface 174 Objective lens 176 Color separation assembly 178 Lighting assembly 179 Lighting filter 180 Lighting beam splitter 182 Image display device 184 Field of view 186 Computer 188 White light filter 190 Fluorescent filter 192 Dichroic beam splitter 194 Memory 196 Data transmission line 200 Overlay color coordinates, {X OC , Y OC , Z OC}, overlay color 202 Background color coordinates, {X BC , Y BC , Z B C}, background color 204 Light intensity value, I 2 206 Output color coordinates of the output color {X O , Y O , Z O}, output color 210 Color space conversion 212 Luminance calculation 250 Color conversion 302 First imaging spectrum 304, 306, 308 Color space coordinates 310 Stop band 312 Visible spectrum or light range 322 Second imaging spectrum 324 Pass band 326 Fluorescent emission spectrum of the phosphor 400 Uniform color space 402 Luminance line 404 Overlay color line 406 Endpoints of the luminance line 408 First point on the luminance line 410 Length from the first point to the overlay color coordinates 412 Length from the endpoint to the first point 414 Second point on the overlay color line 420 Length from the second point to the first point 422 Length from the background color coordinates to the second point 440 Maximum luminance value 442 Minimum luminance value 444 Luminance of the background color 446 Maximum light intensity value 448 Minimum light intensity value 450 Plane of constant luminance 452 Color gamut 500 System 510 Microscope 520 Computer system I Intensity I 2 Light intensity value R, G, B Primary colors of the RGB color space X BC , Y BC , Z BC Color space coordinates of the color of the first pixel in the uniform color space X OC , Y OC , Z OC Color space coordinates of the overlay color in the uniform color space X O , Y O , Z O Color space coordinates of the color of the output pixel in the uniform color space λ Wavelength
Claims
1. A data processing device (170) that calculates the output color of an output pixel (150c) in an output color image (160) by mixing the background color and overlay color of a first pixel (150a) in a background color image (114) depending on the light intensity value (204) of a second pixel (150b) in a second image (112), The aforementioned data processing device is The background color coordinates (202) of the background color within the uniform color space (400), which has a predetermined maximum brightness value (440) and a predetermined minimum brightness value (442) in the uniform color space, are obtained. The overlay color coordinates (200) of the overlay color within the aforementioned uniform color space (400) are obtained. A light intensity value (204) is obtained that is within a range extending from a predetermined maximum light intensity value (446), including a predetermined maximum light intensity value (446), to a predetermined maximum light intensity value (448), including a predetermined maximum light intensity value (448). Within the uniform color space (400), along the point having the saturation of the overlay color coordinate (200), calculate a luminance line (402) that extends from the overlay color coordinate (200) to an endpoint (406) having a luminance lower than the luminance at the overlay color coordinate. It is configured to determine the first point (408) on the luminance line (402), The ratio of the difference between the predetermined maximum luminance value and the luminance of the background color coordinate (202), and the ratio of the difference between the luminance of the background color coordinate (202) and the predetermined minimum luminance value, corresponds to the ratio of the length of the luminance line (410) from the first point (408) to the overlay color coordinate (200), and the length of the luminance line (412) from the endpoint (406) to the first point (408). The aforementioned data processing device is Calculate an overlay color line (404) that extends linearly from the background color coordinate (202) to the first point (408), It is configured to determine a second point (414) on the overlay color line (404), The ratio of the difference between the predetermined maximum light intensity value and the light intensity value, and the ratio of the difference between the light intensity value and the predetermined minimum light intensity value, corresponds to the ratio of the length (420) of the overlay color line (404) from the second point (414) to the first point (408), and the length (422) of the overlay color line (404) from the background color coordinate (202) to the second point (414). The data processing device is configured to output the color coordinates (206) of the second point (414) in the uniform color space (400) as the output color of the output pixel (150c). Data processing device (170).
2. The endpoint (406) of the luminance line (402) is located within the color gamut (452) of a predetermined display device (132, 182). The data processing device (170) according to claim 1.
3. The luminance line (402) is a straight line within the uniform color space (400). A data processing device (170) according to claim 1 or 2.
4. The data processing device is configured to convert the overlay color coordinates (200) from a non-uniform color space to a uniform color space (400) before acquiring the overlay color coordinates (200). The data processing device (170) according to claim 1.
5. The data processing device is configured to convert the background color coordinates (202) from a non-uniform color space to a uniform color space (400) before acquiring the background color coordinates (202). The data processing device (170) according to claim 1.
6. The data processing device is configured to convert the color coordinates (206) of the second point (414) from a uniform color space to a non-uniform color space before outputting the color coordinates of the second point. The data processing device (170) according to claim 1.
7. The aforementioned data processing device is From the background color image (114), a first pixel (150a) containing the background color coordinate (202) is obtained. The system is configured to obtain a second pixel (150b) containing the light intensity value (204) from the second image (112). The data processing device (170) according to claim 1.
8. The second image (112) is one of a grayscale image and a color image. The data processing device (170) according to claim 1.
9. The aforementioned data processing device is A background color image (114) containing a plurality of first pixels (150a) recorded in the first imaging spectrum (302) is acquired. The system is configured to acquire a second image (112) containing a plurality of second pixels (150b) recorded in a second imaging spectrum (322), The second imaging spectrum overlaps with the fluorescence emission spectrum (326) of at least one phosphor (116, 118), the second imaging spectrum is different from the first imaging spectrum, and both the first and second imaging spectra overlap with the visible spectrum (312). The data processing device (170) according to claim 1.
10. The ratio of the luminance at the overlay color coordinate (200) to the luminance at the endpoint of the luminance line is independent of the background color coordinate (202) of the first pixel (150a) and / or the light intensity value (204) of the second pixel (150b). The data processing device (170) according to claim 9.
11. The background color image (114) and the second image (112) are registered. The data processing device (170) according to claim 1.
12. In a computer implementation method for calculating the output color of an output pixel (150c) in an output color image (160) by mixing the background color and overlay color of a first pixel (150a) in a background color image (114) depending on the light intensity value of a second pixel (150b) in a second image (112), the computer implementation method comprises the following steps, namely: In a uniform color space (400) having a predetermined maximum brightness value (440) and a predetermined minimum brightness value (442), the steps include obtaining the background color coordinates (202) of the background color, The steps include obtaining the overlay color coordinates (200) of the overlay color in the aforementioned uniform color space, A step of obtaining a light intensity value (204) that is within a range extending from the minimum light intensity value (448), which includes the minimum light intensity value (448), to the maximum light intensity value (446), which includes the maximum light intensity value (446), In the aforementioned uniform color space, the steps include calculating a luminance line (402) that extends from the overlay color coordinate to an endpoint (406) having a luminance lower than the luminance at the overlay color coordinate, along a point having the saturation of the overlay color coordinate, The steps include determining a first point (408) on the brightness line, Includes, The ratio of the difference between the predetermined maximum luminance value and the luminance of the background color coordinate (202), and the ratio of the difference between the luminance of the background color coordinate (202) and the predetermined minimum luminance value, corresponds to the ratio of the length of the luminance line (410) from the first point to the overlay color coordinate (200) and the length of the luminance line (412) from the endpoint to the first point. The aforementioned computer implementation method is The steps include calculating an overlay color line (404) that extends linearly from the background color coordinate (202) to the first point, The steps include determining a second point (414) on the overlay color line, Includes, The ratio of the difference between the maximum light intensity value and the light intensity value (204) and the difference between the light intensity value (204) and the minimum light intensity value corresponds to the ratio of the length of the overlay color line from the second point to the first point (420) and the length of the overlay color line from the background color coordinate (202) to the second point (422). The computer implementation method includes the step of outputting the color coordinates (206) of the second point in the uniform color space as the output color. Computer implementation method.
13. A computer program product or computer-readable medium that includes instructions causing a computer to perform the method described in claim 12 when the program is executed by the computer.
14. A medical observation device (100) such as a microscope or endoscope, wherein the medical observation device is A white light color camera (110) configured to record a background color image (114), A fluorescence camera (111) configured to record a second image (112), The data processing device (170) according to claim 1, A medical observation device (100) including the following.
15. A method for operating a medical observation device (100) such as a microscope or endoscope, The method described above includes the computer implementation method described in claim 12, The aforementioned method, The steps include recording a background color image (114) of the object using a white light color camera (110), The steps include recording a second image (112) of the object using a fluorescent camera (111), A method that further includes this.