Data processing device and computer-implemented method for combining fluorescence emission signal with specular reflection signal in medical observation device

Abstract

The present invention relates to a data processing device (170) and a computer-implemented method for a medical observation device (100), such as a microscope or endoscope, for observing an object (106) comprising at least one fluorophore (116, 118), where the data processing device (170) is configured to: access input image data (120) representative of an image of the object (106), the input image data comprising: a reflected signal (202) representing an image of the object (106); the method includes receiving a reflection signal (206) representative of light (198) reflected from an object and including a specular reflection signal (206) representative of specular reflection (208) from the object, and a fluorescence emission signal (204) representative of fluorescence emitted by at least one fluorophore; extracting a specular reflection signal; and generating a digital output image (106) from a combination of the extracted specular reflection signal and the fluorescence emission signal.

Description

Technical Field

[0001] This invention relates to a data processing device and a computer-implemented method for use in medical observation equipment, such as a microscope or endoscope. The medical observation equipment is configured to observe objects containing fluorophores. The invention also relates to a medical observation equipment including a data processing device, and a method of operating the medical observation equipment, the method comprising a computer-implemented approach. Background Technology

[0002] Medical observation equipment can be used in surgical or laboratory settings. In surgery, the object can be part of the patient's body, while in a laboratory setting, such as in a biopsy, the object can be a clump of cells, a single cell, or a portion of a cell. Fluorescence is commonly used to detect specific properties of the object. The object may naturally contain one or more fluorophores—i.e., fluorescent materials—and / or one or more fluorophores may be artificially added to the object, for example, by injection into blood or cells. The fluorescence emission of the fluorophores can be used to label and highlight specific materials or portions of interest within the object.

[0003] For example, in surgery, some fluorophores, such as 5-aminolevulinic acid (5-ALA), are used to label tumors, while others, such as indocyanine green (ICG), are used to label blood flow. The autofluorescence of fluorophores naturally present in an object can be used to identify specific types of tissue.

[0004] The fluorescence of one or more fluorophores is triggered by illuminating an object with light that includes or is limited to a fluorescence excitation spectrum of at least one fluorophore.

[0005] Fluorescence from at least one fluorophore is represented by a fluorescence emission signal, which is recorded as part of the input image data by a medical observation device. If only the fluorescence emission signal is displayed, the user can quickly identify these parts of the object of interest. However, it is difficult for the user to quickly determine the location of the fluorescent object—whether it is inside an object or within the field of view of the medical observation device.

[0006] Therefore, the object of the present invention is to improve images generated by data processing devices, allowing users to more reliably determine the location of fluorescent fluorophores within an object. Summary of the Invention

[0007] This objective is achieved by a data processing device for observing objects containing fluorophores, such as a microscope or endoscope, wherein the data processing device is configured to: access input image data representing an image of the object, the input image data including: a reflection signal representing light reflected from the object and including a specular reflection signal representing specular reflection from the object; and a fluorescence emission signal representing fluorescence emitted by at least one fluorophore; extract the specular reflection signal; and generate a digital output image from the combination of the extracted specular reflection signal and the fluorescence emission signal.

[0008] The present invention also relates to a computer-implemented method for use in medical observation devices, such as microscopes or endoscopes, the method comprising the steps of: accessing input image data representing an image of an object containing at least one fluorophore, the input image data comprising a reflection signal representing light reflected from the object and a fluorescence emission signal comprising a specular reflection signal representing specular reflection from the object and the fluorescence emission signal representing fluorescence emitted by the object; extracting the specular reflection signal; and generating a digital output image from the combination of the extracted specular reflection signal and the fluorescence emission signal.

[0009] By combining the extracted specular reflection signal with the fluorescence emission signal into a single image, the digital output image provides the user with an easily identifiable positional reference frame. Users can quickly detect specular reflections in the reflection image of an object—or the object itself—and thus immediately identify the location of fluorescent areas on the object. Adding only the specular reflection signal does not further overload the digital output image with the signal, so the details of the fluorescence emission signal are not obscured or masked by other signals, and only a minimal positional reference is provided.

[0010] The above solution can be further improved by one or more features described below. These features can be combined independently of each other.

[0011] Furthermore, even if the following features are described only in the context of a data processing device or only in the context of a computer-implemented method, the corresponding features can also be used to improve the data processing device and / or the computer-implemented method. For example, features or steps described in the context of a computer-implemented method can generally be used only to improve the data processing device because the data processing device is configured to perform or carry out this method feature or step. Specifically, the data processing device may include routines that can be implemented in software, hardware, or a combination of software and hardware, which, when executed, perform the method features or steps. Conversely, features described only in the context of a data processing device can be used to improve a computer-implemented method.

[0012] According to one aspect, the input image data includes one or more digital input images. At least one digital input image can be a color input image or a monochrome image. The input image data can contain multiple input pixels. Each input pixel represents light received from a specific location on an object. Each input pixel corresponds to a specific location on the object. If the input image data contains more than one digital input image, a specific location on the object can be represented by more than one input pixel. As is common in the art, input pixels representing the same location on the object are referred to as "corresponding pixels".

[0013] Input pixels can be either colored or monochrome. Colored input pixels include color space coordinates, which use, for example, a color space to define the color of the input pixel. Color space coordinates represent color appearance parameters such as hue, lightness, brightness, chroma, saturation, and color intensity. If a tri-stimulus color space such as RGB is used, each input pixel includes three color space coordinates: R, G, and B. Color space coordinate R defines the intensity of the red band, color space coordinate G defines the intensity of the green band, and color space coordinate B defines the intensity of the blue band. For other color spaces, such as HSV, CIELAB, or CIELUV, other color space coordinates are used. Monochrome input pixels only indicate light intensity.

[0014] Throughout this document, if the digital input image is a color image, it is referred to as a "digital color input image." Multispectral or hyperspectral images are considered color images. If the digital input image is a monochrome image, it is referred to as a "digital monochrome input image." If it is not important in a particular context whether the digital input image is a color or monochrome image, the general term "digital input image" is used. Therefore, a "digital input image" can be either a "digital color input image" or a "digital monochrome input image." The same terminology is used for digital output images.

[0015] Each digital input image can contain multiple input pixels. In a digital color input image, each input pixel can be a color pixel. In a digital fluorescent color output image, each pixel, or more specifically, each output pixel, can be a color pixel. A color pixel contains a set of color space coordinates that, when combined, represent at least some color appearance parameters such as hue, lightness, brightness, chroma, and so on. A monochrome image can represent only light intensity values, for example, in grayscale.

[0016] In one particular embodiment, the input image data may include at least one of the following images: at least one digital color input image of an object; at least one digital color input image and at least one digital monochrome input image; and at least two digital monochrome input images. These are corresponding sets of digital input images that may contain reflective and fluorescent emission signals that can be reliably separated. However, the set of input images may also include only a single digital monochrome input image that contains or consists of specular reflection and fluorescent emission signals.

[0017] If the input image data includes multiple digital input images, it is preferable that each of the multiple digital input images represents a different spectrum. Specifically, the spectra of the different digital input images do not overlap, or at least have only minimal overlap. Furthermore, the spectra of the different digital input images can be complementary.

[0018] A digital output image may include multiple output pixels. The digital output image can be a monochrome image or a color image. Output pixels can be monochrome or color. For each output pixel, there is at least one corresponding input pixel in the input image data.

[0019] Reflection signals, specular reflection signals, and fluorescence emission signals can be considered as discrete and distinguishable signals superimposed on each other in input image data, such as in one or more digital input images. Each input pixel can represent a superposition of signals received from a specific location on the object. Each of the reflection, specular reflection, and fluorescence emission signals represents a digital color or monochrome image of the object and can be distinguished from the other signals by at least one color appearance parameter.

[0020] The reflected signal and the fluorescent emission signal can be contained together in a single digital input image, preferably in a single digital color input image. Alternatively, the reflected signal and / or the fluorescent emission signal can be contained in different digital input images. For example, the reflected signal can be contained in one digital input image, while the fluorescent emission signal can be contained in another digital input image.

[0021] Alternatively, the reflected signal and / or fluorescent emission signal can be distributed across multiple digital input images. In this case, the digital input image may contain only a portion of the reflected signal and / or fluorescent emission signal. If the digital input image contains more than one signal, it should be a color image so that the signals can be separated from each other.

[0022] If a reflected signal (in whole or in part) and a fluorescent emission signal (in whole or in part) are contained in a single digital input color image, the data processing device is preferably configured to extract the reflected signal and / or the fluorescent emission signal contained in the digital input color image. The data processing device may include separation or extraction routines for extracting and / or separating the fluorescent emission signal and / or the reflected signal from the digital input image data or the remainder of the digital input image composed of the input image data.

[0023] Extracting any signal from input image data or digital input images, such as reflected signals, fluorescent emission signals, and / or specular reflection signals, may involve linear transformations and / or spectral unmixing. Linear transformations and spectral unmixing are preferably performed on a pixel-by-pixel basis. Linear transformations may include matrix multiplication of input pixels in a digital input image or corresponding input pixels in different digital input images with a transformation matrix.

[0024] In this configuration, the digital input image includes at least a portion of a fluorescent emission signal and at least a portion of a reflected signal. The digital data processing apparatus is preferably configured to separate the reflected signal and / or specular reflection signal contained in the digital input image from the fluorescent emission signal contained in the digital input image. The data processing apparatus may include a specular reflection extraction routine for extracting the specular reflection signal from the extracted reflected signal or directly from the input image data or at least one digital input image containing the specular reflection signal.

[0025] If a portion of a signal, such as a reflected signal, specular reflection signal, and / or fluorescence emission signal, is contained in multiple digital input images, the data processing device can be configured to combine the extracted portions of the respective signals. This combination may include partial linear transformations such as addition, or α-blending (or mixing). This allows for the recovery of a more complete signal from the corresponding segments contained in the various digital input images.

[0026] For example, the input image data may include a first digital input image containing a portion of the reflected signal and at least a portion of the fluorescent emission signal; and a second digital input image containing an additional portion of the reflected signal. The data processing device may be configured to extract a portion of the reflected signal from the first digital input image and an additional portion of the reflected signal from the second digital input image. The data processing device may also be configured to calculate the reflected signal from the portion of the reflected signal contained in the first digital input image and the additional portion contained in the second digital input image. The reflected signal can be calculated by using a linear transformation and / or an α-mix combination of the two portions as described above.

[0027] According to another embodiment, the second or third input image may include an additional portion of the fluorescence emission signal. The data processing device may be configured to extract a portion of the fluorescence emission signal from the first digital input image and an additional portion of the fluorescence emission signal from the second or third digital input image. The data processing device may also be configured to calculate the fluorescence emission signal from the portion of the fluorescence emission signal included in the first digital input image and the additional portion included in the second or third digital input image. The fluorescence emission signal can be calculated by using a linear transformation and / or α-blending combination of the two portions as described above.

[0028] Since specular reflection corresponds to total internal reflection of light illuminating an object, it can represent the brightest spot in the object's reflected image. Therefore, according to one aspect, the data processing device can be configured to extract the specular reflection signal by retrieving these pixels from input image data or a digital input image based on their intensity. The specular reflection signal can be extracted directly from the input image data. Alternatively, the reflection signal can be extracted first, and then the specular reflection signal can be extracted from the reflection signal.

[0029] In one variant, pixels with an intensity greater than a predetermined intensity threshold are extracted from the input image data or digital input image. Therefore, only pixels with an intensity exceeding the predetermined intensity threshold are considered in the specular reflection signal.

[0030] To obtain consistent results when extracting specular reflection signals, it may be desirable to normalize the input image data, especially at least these digital input images containing reflection signals, before extracting the specular reflection signals.

[0031] Since specular reflection corresponds to total internal reflection, the color of the specular reflection corresponds to the color of the light illuminating the object. Therefore, according to another variation, the specular reflection signal can be extracted from the input image data by extracting the colors of the pixels. Specifically, the colors of the pixels in the input image data can be compared with a predetermined color, which can be stored in the memory of the data processing device. The predetermined color can correspond to the color of the illumination.

[0032] If the medical observation device has an adjustable light source, the stored predetermined colors can correspond to the corresponding current lighting color, because the medical observation device can be configured to automatically update the stored predetermined colors according to the current lighting color.

[0033] According to another approach, the data processing device can be configured to extract specular reflection signals by extracting pixels having colors contained in a predetermined color group. The predetermined color group can represent the current color of the light source illuminating the object. The medical observation device can automatically update the predetermined color group based on the current illumination. Using a predetermined color group instead of a single predetermined color makes the extraction of the specular reflection signal more robust. The predetermined color group or a single predetermined color can be obtained through calibration. The predetermined color group can correspond to a region in a color space, which can be continuous.

[0034] The data processing device can be configured to color the extracted specular reflection signal with a predetermined color, wherein at least one color appearance parameter of the predetermined color of the specular reflection signal in the digital output color image can depend on the intensity of the specular reflection signal in the corresponding pixel. Therefore, the specular reflection in the digital output image acquires its natural color.

[0035] Extracting pixels based on their intensity can be combined with extracting them based on their color. This makes the extraction of specular reflection signals more robust.

[0036] In some implementations, the input image data may include a digital fluorescence input image representing the fluorescence emission of an object and potentially including a portion of the reflected signal; and a digital reflection input image including at least a majority of the reflected signal or consisting of the reflected signal. The digital fluorescence input image may be a digital input image representing light recorded in a discrete narrow passband, which may lie within one or more fluorescence emission spectra of at least one fluorophore. The digital reflection input image may be a digital input image representing light recorded in at least one wide passband. Specifically, the reflected signal included in the digital reflection image may represent a white light reflection image of the object.

[0037] The reflected signal can represent the light in the fluorescence excitation spectrum of a fluorophore, while the fluorescence emission of the fluorophore is represented in the fluorescence emission signal. The low-frequency end of the fluorescence excitation spectrum can overlap with the high-frequency end of the fluorescence emission spectrum, as shown in the fluorescence emission signal. Therefore, illumination of an object used to trigger fluorescence can produce reflection, i.e., a portion of the reflected signal recorded in the fluorescence emission spectrum.

[0038] Digital fluorescence input images and digital reflectance input images can be color images.

[0039] The reflected signal can represent reflected light in the near-infrared (NIR) range, i.e., with wavelengths greater than 700 nm and preferably less than 1000 nm. This portion of the reflected signal may have been recorded in a passband, which is preferably confined to the NIR band. Reflected light in the NIR range represents light reflected well in the visible red range. Therefore, a portion of the reflected signal in the NIR range can be used to supplement the red areas where reflected images cannot be recorded in the visible red range. This may be the case if the fluorescence emission of at least one fluorophore is located in the visible red range.

[0040] In one embodiment, the input image data may include a first (partial) reflection signal and a second (partial) reflection signal, wherein the first reflection signal represents light reflected from an object and having a first polarization type, and the second portion of the reflection signal represents light reflected from the object and having a second polarization type, wherein the second polarization type is different from the first polarization type. In one embodiment, the first reflection signal may be a reflection signal without specular reflection, and the second reflection signal may be a reflection signal with specular reflection.

[0041] According to one aspect, the first polarization type can include unpolarized light. According to another aspect, the second polarization type can include linearly polarized light or circularly polarized light.

[0042] The data processing device can be configured to obtain a specular reflection signal from a combination of a first reflection signal and a second reflection signal. The combination of a portion of the reflection signal in the first digital input image with an additional portion of the reflection signal can include applying a linear transformation, specifically subtraction, to the first and second reflection signals. The data processing device can perform the subtraction pixel-by-pixel.

[0043] The first reflected signal and the second reflected signal can be contained in a single digital input image, or the first reflected signal can be contained in a first digital input image and the second reflected signal can be contained in a second digital input image. If the first reflected signal and the second reflected signal are contained in a single digital input image, they can constitute a first image channel and a second image channel, respectively.

[0044] According to another embodiment, the digital processing device can be configured to generate an edge detection signal from a reflected signal, particularly from a reflected signal without specular reflection. Using a reflected signal without specular reflection has the advantage of not generating false edges caused by specular reflection. The edge detection signal can be a monochrome signal or a color signal.

[0045] Digital processing devices can be configured to combine edge detection signals with fluorescence emission signals into a digital output image. Including edge detection signals in the digital output image is another way to provide a visual reference frame to facilitate the localization of the fluorescent emitting portions of an object. Like specular reflection signals, edge detection signals do not impair fluorescence emission signals. Edge detection signals can be combined with fluorescence emission signals and / or reflection signals, with or without extracted specular reflection signals, into the digital output image.

[0046] Generating an edge detection signal may include applying edge detection routines to the reflected signal. Edge detection routines may be part of a data processing device and may be implemented via hardware, software, or a combination of both.

[0047] Edge detection routines can be routines available from image processing libraries. Edge detection routines can be isotropic or anisotropic. Edge detection routines can be applied individually or simultaneously to one or more color space coordinates of the reflected signal.

[0048] According to another embodiment, the data processing device can be configured to generate a digital output image from a combination of at least two signals from a group consisting of a specular reflection signal, a reflection signal that does not contain a specular reflection signal, and a reflection signal and a fluorescence emission signal, the combination depending on the signal selected by the user.

[0049] Furthermore, the data processing device can be configured to combine at least two signals from a group comprising a specular reflection signal, a reflection signal without specular reflection, a reflection signal, an edge detection signal, and a fluorescence emission signal, wherein the ratio of the intensity of the combined signals in the digital output image depends on the user-selected signal.

[0050] The above configuration provides a variety of different viewing modes, in which users can select signals in the digital output image and optionally select the relative intensity between signals.

[0051] For example, a digital processing device can be configured to change the ratio of the intensity of specular reflection signals to the intensity of fluorescence emission signals in a digital output image based on user input signals.

[0052] The digital processing device can also be configured to change the ratio of the intensity of the edge detection signal to the intensity of the fluorescence emission signal.

[0053] The digital processing device can be configured to change the ratio of the intensity of the reflected signal (which does not contain specular reflection signals) to the intensity of the fluorescent emission signal.

[0054] The digital processing device can also be configured to change the ratio of the intensity of the specular reflection signal to the intensity of the edge detection signal. As mentioned above, the intensity ratio can be adjusted by changing the intensity of only one of the two signals separately, or by changing the intensity of both signals together.

[0055] To provide user selection signals, medical observation devices may include mode selectors, such as physical buttons, knobs or dials, and / or widgets.

[0056] Medical observation devices may include data processing equipment and at least one camera adapted to record input image data in any of the embodiments described above.

[0057] At least one camera can be at least one color camera, or at least one color camera and at least one monochrome camera, or at least two monochrome cameras. Multispectral cameras or hyperspectral cameras are considered color cameras. Each camera in a medical observation device can be configured to record a separate digital input image contained in the input image data.

[0058] If the medical observation device includes multiple cameras, each of the multiple cameras is preferably configured to record digital input images in a different spectrum. The different spectra are preferably non-overlapping and / or complementary. At least one camera can be configured to record in the NIR range.

[0059] Medical observation devices may include an observation filter assembly located in the optical path between an object and at least one camera. The optical filter assembly includes one or more passband filters having one or more passbands. At least one passband includes, is composed of, or is located within the fluorescence emission spectrum of at least one fluorophore.

[0060] At least one camera can be configured, for example, to resolve polarization by including an on-chip polarizer. Such a camera can provide a digital input image with two image or—similarly in this context—polarization channels, each polarization channel representing an image of an object in different polarized light. For example, one of these image channels may include a reflected signal without specular reflection, and the other image channel may include a reflected signal with specular reflection.

[0061] At least one camera may be a digital reflective camera, adapted to record at least a portion of the reflected signal. Specifically, a digital reflective camera may be adapted to record a digital color input image representing a white light reflection image of an object.

[0062] At least one camera may be a fluorescence camera adapted to record fluorescence emission signals. The fluorescence camera may also be configured to record a portion of the reflected signal in the NIR range. A portion of an observation filter assembly located between the object and the fluorescence camera may include at least one passband comprising or located within the fluorescence emission spectrum of at least one fluorophore.

[0063] Digital reflectometry (DRI) cameras can be configured to record a greater number of wavelengths in the visible light range than digital fluorescence (DFL). DRI cameras can record light within a wider passband than any passband of a DFL. The visible light range can extend from 380 nm to 700 nm.

[0064] One embodiment of the above-described implementation scheme for using a medical observation device may include the steps of recording input image data and performing the computer-implemented method of any of the above embodiments.

[0065] To generate a digital output image by combining signals, the data processing device may include a signal / image combination routine. Signal combination may include a combination of at least two signals comprising a group of signals including reflected signals, specular reflection signals, and fluorescent emission signals. Signal combination may include pixel-by-pixel addition of the color space coordinates of corresponding pixels in the respective signals to be combined in the digital output image. Optionally or cumulatively, signal combination may include α-mixing and / or linear transformations, such as multiplying the color space coordinates of all corresponding pixels by a transformation matrix.

[0066] The originally mentioned objective is also achieved through a method for using medical observation equipment, such as an endoscope or microscope, wherein the method includes steps of recording input image data and performing a computer-implemented method in any of the configurations described above.

[0067] Finally, the present invention also relates to a computer program product and / or a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the computer-implemented method in any of the above configurations.

[0068] As used herein, the term “and / or” includes any and all combinations of one or more related listed items and may be abbreviated to “ / ”.

[0069] Although some aspects are already described in the context of the apparatus, it is clear that these aspects also represent descriptions of the corresponding methods, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent descriptions of corresponding blocks, items, or features of the corresponding apparatus. Attached Figure Description

[0070] The invention will now be described by way of example with reference to various embodiments and accompanying drawings. The combination of features described and / or shown in the drawings and / or embodiments should not be considered limiting. For example, if a feature has a technical effect, such as that explained above, and is not needed in a particular application, the feature may be omitted from the implementation. Conversely, if the technical effect associated with a feature described above is beneficial in a particular application, that particular feature, which is not part of the implementation described below, may be added.

[0071] Throughout the specification and drawings, the same reference numerals are used for elements that correspond to each other in terms of function and / or structure.

[0072] In the attached diagram,

[0073] Figure 1 A schematic diagram of the medical observation equipment is shown;

[0074] Figure 2 A schematic diagram is shown showing the generation of a digital output image from specular reflection signals and fluorescence emission signals;

[0075] Figure 3 A schematic diagram is shown showing the extraction of specular reflection signals from a digital input image and the generation of a digital output image from a fluorescence emission signal and the extracted specular reflection signal;

[0076] Figure 4 A schematic diagram is shown illustrating the extraction of specular reflection signals using color space coordinates;

[0077] Figure 5 A schematic diagram is shown showing the extraction of mirror reflection signals using reflection signals representing light of different polarizations;

[0078] Figure 6 A schematic diagram of the combination of fluorescence emission signal and edge detection signal is shown;

[0079] Figure 7 A schematic diagram is shown showing the coloring and combination of edge detection signals and / or fluorescence emission signals;

[0080] Figure 8 Schematic diagrams of various combinations of different signals are shown;

[0081] Figure 9 A schematic diagram illustrating different display modes of a medical observation device is shown;

[0082] Figure 10 A schematic overview of the steps used to generate a digital fluorescence color output image is shown;

[0083] Figure 11A schematic diagram of a general medical observation device is shown. Detailed Implementation

[0084] Figure 1 A medical observation device 100 is schematically shown. The medical observation device 100 can be a microscope or an endoscope. The main difference between a microscope and an endoscope is that in an endoscope (not shown), the object 106 is viewed by introducing an optical fiber near the object under study, for example, by inserting it into the body containing the object; however, in a microscope, the objective lens 174 is pointed at the object. Although Figure 1 The most common medical observation equipment is the microscope, but the following description also applies to endoscopes.

[0085] The medical observation device 100 can be a medical observation device used in surgery. The medical observation device 100 can also be a medical observation device used in a laboratory, such as a laboratory microscope. The object to be studied 106 can consist of or include biological tissue 107. The object 106 can be part of a patient's body located within the field of view of the medical observation device 100.

[0086] Object 106 may contain one or more fluorophores 116, 118. At least one of these fluorophores 116 may be a fluorophore naturally present in the object. For example, bones and blood naturally contain fluorophores. Optionally or cumulatively, at least one fluorophore 118 may be artificially added to object 106, for example by injecting it into biological tissue 107. Examples of fluorophores 118 that may be artificially added to object 106 are ICG, luciferin, and / or 5-ALA.

[0087] The medical observation device 100 shown is a fluorescence imaging device. Therefore, the medical observation device is configured to view and preferably also excite the fluorescence of one or more fluorophores 116, 118.

[0088] Medical observation equipment 100 can be as follows Figure 1 The stereoscopic device is shown schematically. Therefore, it can include two identical sub-components 101L and 101R for each of the two stereoscopic channels. Since the two sub-components 101L and 101R are identical in function and structure, the following description focuses on the right sub-component 101R, but the same applies to the left stereoscopic channel 101L.

[0089] The medical observation device 100 may optionally be a monocular device (or single-field-of-view device). In this case, only one of the two sub-components 101L and 101R may exist. Therefore, the following description also applies to the monocular medical observation device 100.

[0090] The operating medical observation device 100 provides input image data 120. The input image data 120 represents the imaged scene, i.e., a portion of an object within the field of view 184 of the medical observation device. The medical observation device 100 is configured to record the fluorescence of at least one fluorophore 116, 118 in the input image data 120.

[0091] Input image data 120 may include one or more different digital input images 130. The digital input images 130 may correspond to frames in a video stream or sequence. If input image data 120 contains multiple digital input images 130, the different digital input images 130 should contain different spectral information. In this case, each digital input image 130 of the input image data may be recorded at a different wavelength, preferably without spectral overlap or, equivalently, with minimal spectral overlap. Preferably, the spectra of the different digital input images 130 recording the input image data 120 are complementary and / or non-overlapping.

[0092] To generate input image data 120, the digital imaging system 102 may include one or more digital cameras 108. The number of digital input images 130 included in the input image data 120 may depend on and / or correspond to the number of cameras 108 used to generate the input image data 120. Depending on the type and settings of the respective digital cameras 108, the digital input images 130 may be color images or monochrome images.

[0093] The input image data 120 contains a fluorescence emission signal representing the fluorescence emitted by the object 106, particularly the fluorescence emitted by at least one fluorophore 116, 118 contained therein. The fluorescence emission signal does not necessarily need to include all wavelengths of fluorescence emitted by the object 106; it may include only a portion of the fluorescence emitted by at least one fluorophore 116, 118, such as fluorescence emitted in one or more discrete passbands that overlap with the fluorescence spectrum.

[0094] The medical observation device 100 is also configured to record reflected signals, which represent light reflected from the object 106 and are also included in the input image data 120. The reflected light may have been generated by white light illumination, such as using a standard illuminator, or by illumination consisting of a fluorescence excitation spectrum comprising at least one fluorophore 116, 118 or a fluorescence excitation spectrum comprising at least one fluorophore 116, 118.

[0095] The reflected signal includes or may consist of a specular reflection signal representing a specular reflection from object 106.

[0096] In one embodiment, the reflected signal may include or consist of: a first reflected signal without specular reflection and a second reflected signal containing specular reflection. The reflected signal without specular reflection may be included in a digital input image 130 that is different from the reflected signal with specular reflection. Specifically, the reflected signal with specular reflection may represent light having polarization characteristics different from the reflected signal without specular reflection.

[0097] A single digital input image 130 may contain one or more signals, some of which may be entirely contained within the digital input image 130, while others may be partially contained. If the digital input image 130 contains more than one signal, it is preferably a digital color input image 130, such that the different signals can be distinguished from each other by, for example, the spectral characteristics of the signals. For example, a digital color input image 130 may contain a portion or all of a reflective signal and a portion or all of a fluorescent emission signal, with the remaining portion of the reflective signal and / or fluorescent emission signal contained in another digital input image 130. According to another embodiment, the reflective signal and the fluorescent emission signal may be contained in separate digital input images, and the different digital input images may be digital color or monochrome images 130.

[0098] By way of example only, the digital imaging system 102 may include a digital reflective camera 110 and one or more digital fluorescence cameras 111, 111a as a digital camera 108. A second (or third) digital fluorescence camera 111a is optional. Figure 1 In this diagram, the second digital fluorescence camera 111a is shown only in the left stereo channel 101L, but it could also exist in the right stereo channel 101R. Optionally, a digital fluorescence camera in one stereo channel can be used as the (first) digital fluorescence color camera 111, and a digital fluorescence camera in another stereo channel can be used as the second fluorescence camera 111a. Cameras 110, 111, and 111a can each be a color camera or a monochrome camera. Multispectral cameras or hyperspectral cameras are considered as color cameras.

[0099] Digital reflectance camera 110 is configured to record a digital reflectance input image 114, i.e., a digital input image 130, which represents the reflection of object 106 and therefore may include all or at least most of the reflection signal. Digital reflectance camera 110 is preferably configured to record digital input image 130 over a broad spectral range within the visible spectrum. Therefore, the digital input image 130 recorded by the digital reflectance camera can closely represent the natural color of object 106. It is important to use digital reflectance camera 110 to provide the user with an image of the object that is as close as possible to human perception. Digital reflectance camera 110 may be a charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), or a multispectral or hyperspectral camera.

[0100] The digital input image 130 recorded by the digital reflection camera 110 may also contain at least a portion of specular reflection signals and / or fluorescence emission signals.

[0101] The digital reflection camera 110 can be configured to resolve the polarization of the incident light, for example by including an on-chip polarizer 168. In this case, the digital reflection camera 110 can record reflection signals with specular reflection signals and reflection signals without specular reflection signals.

[0102] The fluorescence camera 111 can be configured to record digital fluorescence images only in one or more narrow spectral bands. These narrow spectral bands should overlap with or be included in the fluorescence spectra of one or more fluorophores 116, 118 whose fluorescence is to be recorded. Preferably, the fluorescence spectra of the different fluorophores 116, 118 are at least partially separated, preferably completely separated, i.e., do not overlap, so that the fluorescence camera 111 can record a digital color input image 130 representing two independent fluorescence bands spaced apart from each other.

[0103] Optionally, if two or more fluorescence cameras 111, 111a are provided, each fluorescence camera 111, 111a preferably captures the fluorescence emission of different fluorophores 116, 118, thereby providing two digital fluorescence input images 112. The fluorescence emission signal may include or consist of the fluorescence of only one fluorophore 116, 118, even if the fluorescence of more than one fluorophore 116, 118 is recorded, or if it may include or consist of the fluorescence of more than one fluorophore 116, 118.

[0104] Any fluorescence camera 111, 111a can also capture a portion of the reflected signal in its corresponding digital color input image 130. For example, a portion of the excitation spectrum of the fluorescence used to trigger fluorophores 116, 118, particularly at the low-frequency end, can overlap with the fluorescence spectrum of the fluorophore and then be recorded together with the fluorescence emission as a reflected signal or a portion of the reflected signal. Preferably, the intensity of the reflected signal is comparable to the intensity of the fluorescence, such that the reflected signal is no brighter than the fluorescence emission.

[0105] At least one fluorescence camera 111, 111a can be a monochrome camera, CCD, CMOS, or a multispectral or hyperspectral camera. Preferably, the reflection camera 110 and at least one fluorescence color camera 111 are of the same type, although this is not mandatory.

[0106] At least one fluorescence camera 111, 111a can be configured to resolve polarization, for example, by including an on-chip polarizer 168.

[0107] Any combination of cameras 110, 111, and 111a can be functionally combined into a single multispectral or hyperspectral camera. Multispectral or hyperspectral then represents the function of one or more of cameras 110, 111, and 111a. A digital color input image 130 recorded by a multispectral or hyperspectral camera can represent multiple digital input images 130 and can contain all fluorescence emission signals and all reflection signals.

[0108] The corresponding fields of view 184 of camera 108 are preferably aligned or even coincident and coaxial. Preferably, camera 108 provides the same field of view 184 with the same perspective and focal length. This results in the same geometric representation of object 106 in images 112, 114 generated by different cameras 108. Camera 108 can use the same objective lens 174. Preferably, camera 108 has the same resolution and / or number of pixels and / or aspect ratio.

[0109] If a perspective and field-of-view match cannot be generated optically, it can be generated by applying a matching or registration routine to the digital input image 130 for image processing, as explained further below.

[0110] Preferably, the cameras 108 operate synchronously. Specifically, the exposure times can be synchronized. Therefore, the medical observation device 100 can be configured to simultaneously generate digital input images 130.

[0111] Preferably, the gain of the cameras 108 is synchronized, meaning the gain is adjusted simultaneously in each camera 108. Furthermore, even if the gain in any one of the cameras 108 is changed, the ratio of the gain applied in one camera 110 to the gain applied in any other camera can remain constant. Gamma correction and color adjustment or white balance can be turned off or kept constant.

[0112] To separate the light recorded in one digital input image 130, such as a digital reflective input image 114, from the spectrum recorded in another digital input image 130, such as at least one digital fluorescence input image 112, an optical color separation component 176 may be provided. The color separation component 176 is used to optically separate the spectra arriving at each camera 108.

[0113] Color separation assembly 176 may include optical elements such as beam splitter 192, which may be dichroic. Color separation assembly 176 may further or optionally include optical observation filter group 188 and / or optical fluorescence filter group 190. Optical observation filter group 188 and fluorescence filter group 190 may be part of optical filter assembly 187.

[0114] The fluorescent filter group 190 is preferably configured to transmit light included in one or more fluorescence emission spectra of one or more fluorophores 116, 118, and to block light outside one or more fluorescence spectra.

[0115] The fluorescence filter group 190 may include one or more passbands and may include one or more optical filters. Each passband should overlap with the fluorescence emission spectrum of the corresponding fluorophore 116, 118 whose fluorescence is to be recorded. Since the fluorescence filter group 190 is located in the optical path between the beam splitter 192 and the fluorescence color camera 111, only wavelengths within the passbands of the fluorescence filter group 190 are transmitted to the fluorescence color camera 111.

[0116] If two fluorescence cameras 111, 111a are used to capture different fluorescence emission spectra, the fluorescence filter group 190 may include different optical bandpass filters in front of each fluorescence color camera 111, 111a. The passband of one bandpass filter may be contained in the fluorescence emission spectrum of one fluorophore 116, while the passband of the other bandpass filter may be contained in the fluorescence emission spectrum of another fluorophore 116, 118 in the object 106.

[0117] The observation filter group 188 is preferably configured to block light in one or more fluorescence spectra of one or more fluorophores 116, 118. The observation filter group 188 may also be configured to block light in fluorescence excitation spectra.

[0118] The observation filter group 188 is preferably configured as a bandstop filter, wherein the stopband corresponds to or at least includes the passband of the fluorescence filter group 190. The observation filter group 188 is located in the optical path between the beam splitter 192 and the reflection camera 110. Therefore, the reflection camera 110 records only wavelengths outside the stopband of the observation filter group 188, and thus also records wavelengths outside the passband of the fluorescence filter group 190.

[0119] Either the observation filter group 188 and the fluorescence filter group 190 may include a tunable filter.

[0120] If beam splitter 192 is a dichroic beam splitter, at least one of the filter groups 188 and 190 can be omitted, since optical spectral filtering is already integrated into the dichroic beam splitter in this case. The above description of the passband and stopband should then be applied to the dichroic beam splitter 192 with the necessary modifications.

[0121] The medical observation device 100 may also include an illumination component 178, which is configured to preferably illuminate the object 106 through an objective lens 174, and an imaging system 102 to record at least one digital image 112, 114 through the objective lens 174.

[0122] The illumination component 178 can be configured to selectively generate white light, i.e., light uniformly distributed across the entire visible spectrum; and fluorescent excitation light, which contains only light within the wavelengths that stimulate the fluorescence of at least one fluorophore 116, 118. The illumination light generated by the illumination component 178 can be fed into the objective lens 174 using an illumination beam splitter 180.

[0123] The illumination component 178 can be configured to simultaneously generate illumination light within multiple discrete, particularly narrow-band, wavelength bands. These wavelength bands may include any of the following wavelength bands or any combination thereof.

[0124] One such discrete wavelength band may lie entirely within the fluorescence excitation spectrum of fluorophore 116. Another such wavelength band may lie entirely within the fluorescence emission spectrum of another fluorophore 118. Yet another such wavelength band may be limited to wavelengths greater than 700 nm, for example, up to 1000 nm, and lie entirely within the NIR (near-infrared) range. The NIR wavelength band can be used as part of the reflected signal to represent the red component, particularly the R component of the RGB digital color input image 130.

[0125] As described above, illuminating an object simultaneously using any discrete wavelength bands can be accomplished by a light source 199, such as a tunable light source including multiple LEDs of different colors, particularly different primary colors, configured to simultaneously generate light of these wavelength bands. Optionally or additionally, wavelength bands can be generated by using an illumination filter 179 having multiple passbands, wherein the passbands preferably correspond to the aforementioned wavelength bands. If such an illumination filter 179 is used, the light source 199 can generate white light, which is then filtered through the illumination filter 179 so that only light within the passbands illuminates the object 106.

[0126] An illumination filter 179 may be provided based on at least one fluorophore whose fluorescence is to be triggered and its specific excitation spectrum. For example, if 5-ALA is used as the fluorophore, the illumination filter may have 90% to 98% transmittance at wavelengths up to 425 nm, 0.5% to 0.7% transmittance at wavelengths between 450 nm and 460 nm, no more than 0.1% transmittance between 460 nm and 535 nm, and almost zero transmittance at wavelengths above 535 nm. The illumination filter 179 may be configured to allow NIR light to pass through. For example, the illumination filter 179 may include a passband within the NIR spectrum. The illumination filter 178 may also include a passband that preferably lies entirely within the fluorescence excitation spectrum of another fluorophore.

[0127] By reconfiguring the color separation component 176, for example by replacing its optical elements such as filter groups 190 and / or 192, or dichroic beam splitter 180, the medical observation device 100 can be adapted to different fluorophores or fluorophore groups.

[0128] Using the observation filter system 188 as described above, if the corresponding wavelength passes through the passband of the observation filter system 188, at least a portion of the fluorescence emission signal can be recorded using the digital reflective camera 110. In this case, the illumination of the object 106 should preferably not include the fluorescence emission wavelength, because the fluorescence intensity is generally low, which may make the fluorescence signal more difficult to detect since the reflected light is brighter than the fluorescence. This also applies, with necessary modifications, to the reflected signal and at least one digital fluorescence camera 111, 111a.

[0129] Data processing device 170 processes input image data 120. Data processing device 170 may be an integral part of medical observation device 100. In one embodiment, it is a processor embedded in the medical observation device and also functions as a controller to control the hardware of medical observation device 100, such as the brightness and / or spectral emission of light source 199 and / or any objective lens of medical observation device 100 and / or any actuator of medical observation device 100. In another embodiment, data processing device 170 is part of a general-purpose computer connected to the medical observation device via wired or wireless means for one-way or two-way data transmission.

[0130] The data processing device 170 may be a hardware module such as a microprocessor or a software module. The data processing device 170 may also be a combination of hardware and software modules, for example, by using a software module configured to run on a specific processor such as a vector processor, floating-point graphics processor, parallel processor, and / or multiple processors. The data processing device 170 may be part of a general-purpose computer 186 such as a PC. In another embodiment, the data processing device 170 is an embedded processor of the medical observation device 100.

[0131] Data processing device 170 is configured to access input image data 120, such as in the form of one or more digital input images 130, such as digital reflectance input image 114 and digital fluorescence image 112. Data processing device 170 may be configured to retrieve digital input images 130 from memory 194 and / or directly from at least one camera 108. Memory 194 may be part of data processing device 170 or located elsewhere in medical observation device 100.

[0132] The data processing device 170 is also configured to calculate a digital reflection output image 160 from the input image data 120, specifically from the reflection signal and fluorescence emission signal contained in the input image data 120.

[0133] The digital output image 160 can be a color image, represented in a color space. The color space of the digital color output image 160 can be different from the color space of any digital color input image 130 included in the input image data 120. However, preferably, the color space of the digital color output image 160 is the same as the color space of any digital color input image 130. Alternatively, however, the digital output image 160 can be a monochrome image. The digital monochrome output image 160 represents only light intensity as luminance.

[0134] If it is necessary to process the fluorescence emission signal and / or reflection signal separately, the corresponding signals need to be separated or extracted from the input image data 120.

[0135] If the digital input image consists of only a single signal, such as only a fluorescent emission signal or a reflected signal, signal extraction is straightforward. Extracting a signal from such a digital input image 130 simply corresponds to using the corresponding digital input image 130 as the signal that constitutes it.

[0136] However, if any two signals from the group containing fluorescence emission signals, reflection signals, and specular reflection signals are contained in a single digital input image—in this case, the single digital input image should be a color image—they need to be separated from each other for individual processing. To achieve this, the digital processing device 170 is configured to separate the fluorescence emission signals, reflection signals, and / or specular reflection signals from each other. More specifically, the digital processing device 170 may include a separation or extraction routine 140 configured to perform this separation when applied to one or more digital input images 130. The separation or extraction routine 140 may be stored in the memory 194 of the digital processing device 170. The separation or extraction routine 140 may include a demixing routine 142, for example, for spectral demixing of the signals. To extract the specular reflection signal, a specular reflection extraction routine 146 may be constituted by the separation or extraction routine 142.

[0137] The extracted and separately processed signals can be combined after being processed using the signal / image combination routine 144 in the digital output image 160. The signal / image combination routine 144 can process the extracted signals as an image.

[0138] To detect the contours of the anatomical features of object 106, edge detection routine 148 can be applied to the reflected signal, particularly the reflected signal without specular reflection. Edge detection routine 148 can also be included in data processing device 170, for example, in memory 194.

[0139] Any of routines 140 to 148 can be a software routine, a routine implemented in hardware, or a routine in which software and hardware components are combined.

[0140] The medical observation device 100 may include a user input device 162 configured to be operated by a user. During operation, the user input device 162 may generate a user selection signal 164, which may be transmitted to a digital processing device 170. The user input device 162 may be, for example, a physical button, dial, slider, or lever, or a small component representing a physical button, dial, slider, lever, or widget.

[0141] By operating the user input device 162, the user can determine which signal or combination of signals is contained in the digital output image 160 and / or the strength of the signals contained in the digital output image 160. These different display modes are achieved through... Figure 1 The I, II, III, etc. in the text.

[0142] The digital output image 160 can be displayed on a monitor 132, which is integrated with the medical observation device 100. For example, the monitor 132 can be integrated into the eyepiece or eyepiece 104 of the medical observation device 100.

[0143] The digital output image 160 is preferably generated in real time, that is, the digital output image 160 is preferably generated at the same rate as the input image data 120 is recorded or refreshed. If the digital input image 130 is part of a video stream or sequence, the digital output image 160 is generated between two directly consecutive frames.

[0144] The medical observation device 100 may include a direct optical path 134 from object 106 through objective lens 174 to eyepiece 104. In this case, the display may be a translucent display 132 located in the direct optical path 134, or the display may be projected onto the direct optical path 134. A beam splitter 136 may be provided to split the light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed to the eyepiece 104.

[0145] Alternatively, the medical observation device 100 does not require a specific direct optical path 134, but only displays images from the integrated display 132. Alternatively, the medical observation device may not require any display at all.

[0146] The medical observation device 100 may include an output interface 172 to which one or more (external) displays 182 may be connected. For this purpose, the output interface 172 may include standardized connectors and data transmission protocols such as USB, HDMI, DVI, DisplayPort, Bluetooth, and / or other protocols. The external display may be a monitor, 3D glasses, eyepieces, etc. Any combination of external displays may be connected to the output interface 172.

[0147] Computer 186 and / or data processing device 170 are connected to digital imaging system 102 using one or more data transmission lines 196. The data transmission lines can be wired or wireless, or partially wired and partially wireless. Computer 186 and / or data processing device 170 do not need to be fully integrated into medical observation device 100 but can be physically located away from digital imaging system 102. For this purpose, digital imaging system 102 and computer 186 and / or data processing device 170 can be connected to a network, such as a local area network (LAN), wireless local area network (WLAN), or wide area network (WAN), with at least one display 182 also connected to the network.

[0148] According to the modification, the medical observation device 100 can be stereoscopic but includes only two cameras, each for one stereoscopic channel. In one stereoscopic channel, a fluorescence camera 111 is used and configured to selectively record white light reflection, while in the other stereoscopic channel, a reflection camera 110 is used. If fluorescence is not used, this arrangement provides a stereoscopic reflection color input image, and if fluorescence is used, this arrangement provides a single-field white light color input image and a single-field fluorescence color input image. The description above and below also applies to this configuration.

[0149] In one embodiment, the input image data 120 may include at least one digital color input image 130, which contains reflected signals and fluorescence emission signals. This occurs if all the fluorescence emitted by the object 106 is recorded in at least one digital color input image.

[0150] In another embodiment, the input image data 120 may include at least one digital monochrome input image and at least one digital color input image 130. The digital monochrome input image 130 includes at least a portion of a fluorescence emission signal or at least a portion of a reflection signal. If the digital monochrome color input image includes a reflection signal, the digital color input image 130 may include a fluorescence emission signal, or if the digital monochrome input image includes a fluorescence emission signal, the digital color input image 130 may include a reflection signal.

[0151] The input image data 120 may include multiple input pixels. Input pixels can be color pixels or monochrome pixels. Monochrome pixels represent only intensity, such as in a grayscale image. Color pixels include information about at least some of the color appearance parameters such as hue, lightness, brightness, chroma, saturation, and color intensity. Color pixels are recorded using a digital color camera using color bands or equivalent color channels or primary colors of a color space.

[0152] A color space can include at least three color channels. Each color channel in a color space is represented by different color space coordinates. Conversion between different color spaces can be achieved using color space transformations. The same color is represented by different color space coordinates in different color spaces. Each pixel of the digital color input image 130 includes a set of color space coordinates, which together represent the color of the corresponding pixel. Therefore, each color band can be considered as representing a color space axis, and each color can be considered as a point in the color space defined by a vector pointing to that color—that is, color space coordinates. Adding two colors can correspond to vector addition. If one color has color space coordinates {x1, y1, z1} and another color has color space coordinates {x2, y2, z2}, the sum of these two colors has color space coordinates {x1+x2, y1+y2, z1+z2}.

[0153] In one embodiment, the digital input image 130, or more generally, the input image data 120, can be recorded in the RGB color space using primary colors, color bands, or color space coordinates R, G, B. Optionally, the digital input image 130 can be recorded in different color spaces, and / or represent multispectral or hyperspectral color input images. The digital input images 130 of the digital input image group, such as the digital white light color input image 114 and the digital fluorescent color input image 112, do not need to be recorded in the same color space, although recording in the same color space is preferred.

[0154] In the RGB color space, each color is represented by a triplet of three color space coordinates in integer form, where each integer indicates the intensity of one of the primary colors R, G, and B. For example, the strongest red is indicated by the triplet [255, 0, 0]. The strongest green is indicated by [0, 255, 0], and the strongest blue is indicated by [0, 0, 255]. Therefore, the RGB color space is a three-dimensional space, and the CMYK color space would be a four-dimensional space. A multispectral or hyperspectral color space with n color bands will correspondingly produce an n-dimensional color space, where each color is represented by an n-tuple of color space coordinates.

[0155] exist Figure 2The diagram schematically illustrates that input image data 120 may contain two digital input images 130, each digital input image 130 including input pixels 230. (As shown in...) Figure 1 In the context described, the two digital input images 130 may, in one example, be a digital fluorescence input image 112 and a digital reflection input image 114.

[0156] The sample spectrum of the digital fluorescence input image 112 is indicated at reference numeral 252. The digital fluorescence input image 112 represents light recorded in one or more passbands 220; that is, the spectral content represented in the digital fluorescence input image 112 may be limited to one or more spectral bands, each of which is represented by a passband 220. Each passband 220 may be contained in, or contain the fluorescence emission spectra of one or more fluorophores 116, 118.

[0157] For example, the digital fluorescence input image may include a fluorescence emission signal 204, which includes a first fluorescence emission signal 222 of a first fluorophore 118 and a second fluorescence emission signal 224 of an optional second fluorophore 116. Correspondingly, the spectrum 262 of the fluorescence emission signal 204 may include the fluorescence emission spectrum 264 of the first fluorophore 118 and the fluorescence spectrum 266 of the second fluorophore 116. For example, the first fluorophore may be 5-ALA. Spectrum 266 may correspond to the autofluorescence spectrum of object 106.

[0158] The digital fluorescence image 112 may also include at least a portion of the reflection signal 202. The reflection signal 202 in the digital fluorescence image 112 may include a specular reflection signal 208. The sample spectrum 260 of the reflection signal 202 is indicated in spectrum 252.

[0159] By way of example only, the reflected signal 202 may contain spectral components, which may be narrow-band and located within or even limited to the near-infrared (NIR) light range 272. Other components of the reflected signal may be located within individual wavelength bands of the visible light range 270. For example, these components may correspond to the light in the fluorescence excitation spectrum of a fluorophore used to trigger the fluorescence of fluorophore 116. The low-frequency portion of the illumination of the object used to trigger fluorescence may overlap with the high-frequency components of the fluorescence emission spectrum 260 of the corresponding fluorophores 116, 118. Since the illumination used for fluorescence excitation has an intensity greater than that of fluorescence emission, the reflected illumination leaking into the passband 220 is brighter than the fluorescence emission and is therefore part of the reflected signal. Figure 2 As indicated in the document, two different excitation belts can be used to trigger the fluorescence of two different fluorophores.

[0160] Figure 2The diagram also shows a sample spectrum 250 of the digital reflectance input image 114. Preferably, the digital input images 112 and 114 are recorded with complementary spectra, particularly non-overlapping spectra. That is, the corresponding passbands 220 in spectra 252 and 250 do not show overlap, or at most show minimal overlap. Preferably, they complement each other to record the entire visible spectrum 270, and may also record the NIR range 272.

[0161] In the digital reflective input image 114, the passband 220 is preferably wider, such that the portion of the visible light range 270 covered by the digital reflective input image 114 is larger than that covered by the digital fluorescence input image 114. The wider coverage of the visible spectrum 272 in the digital reflective input image ensures that the digital reflective input image 114 represents a good approximation of the white light image and the natural color of the object 106.

[0162] Fluorescence signal 204 and reflection signal 202 are extracted from input image data 122. For example, spectral unmixing and / or linear transformation can be used to separate the fluorescence emission signal 204 and reflection signal 202 (if present) of digital fluorescence image 112 from each other.

[0163] If the digital fluorescent input image 114 consists of fluorescent emission signals, then the extracted fluorescent emission signal 280 can simply correspond to the digital fluorescent input image 114. If multiple digital input images 130 consist of portions of fluorescent emission signals 130, then the extracted fluorescent emission signal 280 simply corresponds to a combination of multiple digital input images 130, for example, by simple addition.

[0164] The digital reflection image 114 is already composed of a portion of the reflection image 202, so there is no need to extract the latter. However, the specular reflection signal 208 can be extracted from either the digital reflection image 114 or the reflection signal 292.

[0165] A portion of the reflection signal 202 extracted from the digital fluorescence image 114 can be combined with the reflection signal 202 in the digital reflection image 114—that is, the digital reflection image 114 as a whole—for example, by addition, especially pixel-by-pixel addition.

[0166] In this way, extracted fluorescence emission signal 280, extracted reflection signal 282, and extracted specular reflection signal 284 can be obtained. Each of these extracted signals 280, 282, and 284 represents an image of object 106 with pixel 232 under different wavelengths (fluorescence emission signal and reflection signal) and intensities or polarizations (spectral reflection signal). The extracted signals 280, 282, and 284 can correspond to color images or monochrome images. If the signals 280, 282, and 284 are monochrome, each pixel 232 of the extracted signal represents only the intensity of fluorescence emission at the position of object 106 corresponding to the position of pixel 230.

[0167] Extracting routine 146 using specular reflection ( Figure 1 The specular reflection signal 284 is obtained. According to a specific specular reflection extraction routine 146, it is not necessary to first extract the reflection signal 202 from the input image data 120 and then extract the specular reflection signal 208 from the extracted reflection signal 282. Instead, the specular reflection signal can be obtained directly from the input image data 122.

[0168] A digital output image 160, which may include output pixels 234, may be obtained, for example, from combining extracted fluorescence emission signals 280 and extracted specular reflection signals 284. Here, a signal / image combination routine 144 may be applied to the extracted fluorescence emission signals 280 and extracted specular reflection signals 284. The signal / image combination routine may include linear transformations of signals 280, 284, such as addition or multiplication with a transformation matrix. For example, a union of the color space coordinates of the input pixels 232 containing signals 280, 284 at each pixel 232 may be multiplied with a transformation matrix. This can be accomplished using either color pixels or monochrome pixels 232.

[0169] exist Figure 3 The diagram illustrates how the specular reflection signal 206 can be extracted and combined with the extracted fluorescence emission signal 204.

[0170] The digital input image 130 or the extracted reflection signal 202 includes a specular reflection 208. At 300, the variation of intensity I along line 304 in the digital input image 130 or the extracted reflection signal 282 is shown. Line 304 passes through the specular reflection 208 and is composed of a series of adjacent pixels 232.

[0171] According to one embodiment of the specular reflection extraction routine 146, all pixels 232 in the digital input image 130 or the extracted reflection signal 282 with an intensity I exceeding the intensity threshold 302 are considered to belong to the specular reflection signal 206. All pixels 232 with an intensity I equal to or less than the threshold 302 are considered to belong to the remainder of the reflection signal 202 and are not considered in the specular reflection signal 206. Therefore, the extracted specular reflection signal 284 consists only of pixels 232 with an intensity I greater than the intensity threshold 302.

[0172] Then, signal combination routine 144 combines the extracted specular reflection signal 284 with the extracted fluorescence emission signal 204. When the extracted fluorescence emission signal 280 and the extracted specular reflection signal 284 are combined, different colors 308 and 310 can be assigned to the extracted signals 280 and 284, respectively. Colors 308 and 310 can be stored in memory 194. Figure 1 A portion of color group 306 in ). The assignment of colors 308 and 310 can be independent of whether the extracted signals 280 and 284 are represented in color or monochrome, and can be selected according to the user selection signal 164.

[0173] For example, a first color 308 can be assigned to the specular reflection signal 284. The first color 308 can correspond to the natural color of the illumination in which the object 106 is illuminated. The object 106, for example, is illuminated by a light source 199 (…). Figure 1 The currently illuminated color 308 can also be stored in memory 194. Therefore, at each output pixel 234, the component representing the extracted specular reflection signal 284 can have a color corresponding to the illumination of object 106. By assigning the first color 308 to the extracted specular reflection signal 284, a more natural appearance for specular reflection can be obtained. False or pseudo-colors can be used instead of the natural color as the first color 308 to clearly mark the specular reflection 208.

[0174] A second color 310 can be assigned to the extracted fluorescence emission signal 280. The second color 310 can be a natural color of fluorescence emission as perceived by human vision, or a false color. The intensity of color 310 at output pixel 234 can depend on the intensity of the extracted fluorescence emission signal 280 at the corresponding pixel 232. Optionally, a false color or pseudo-color can be assigned to the extracted fluorescence emission signal 280.

[0175] The intensity of the first and / or second colors 308 and 310 in the extracted signals 280 and 284 contained in the digital output image 160 can depend on the intensity of the corresponding pixel 232. If pseudo-color is used, the hue of colors 308 and 310 can depend on the intensity.

[0176] In addition to extracting the specular reflection signal 206 using intensity threshold 302, or instead of extracting the specular reflection signal using intensity threshold 302, the specular reflection signal can be extracted or separated from the input image data 120 by the spectral content of the specular reflection signal. (This reference...) Figure 4 To explain.

[0177] exist Figure 4 The image shows a sample reflectance spectrum 408 at the location of the object, corresponding to pixels 230 and 232. At pixels 230 and 232, spectrum 408 is recorded after passing through one or more passbands 220. Therefore, at pixels 230 and 232, color 414 corresponding to color space coordinates {R1, G1, B1} is recorded.

[0178] Each color space coordinate of a pixel can be recorded by a different sensor. Each sensor records light of a specific color band. The spectral sensitivity of these sensors varies with wavelength, as indicated by spectral sensitivity curves 402, 404, and 406. In the case of an RGB image, spectral sensitivity curve 406 quantitatively represents the B color space coordinates, spectral sensitivity curve 404 quantitatively represents the G coordinates, and spectral sensitivity curve 402 quantitatively represents the R coordinates. Therefore, light arriving in at least one passband 220 of the sensor at the pixel will likely trigger responses from all three sensors, as can be seen from sensitivity curves 402, 404, and 406, resulting in color space coordinates R1, G1, B1.

[0179] Specular reflection is total internal reflection of light illuminating object 106. Therefore, the color of the specular reflection corresponds to the color of the illumination. This can be used to identify specular reflection 208 using the color of a pixel: if color 414 corresponds to a (known) color of the illumination spectrum, input pixel 232 is considered to represent a specular reflection.

[0180] To allow for small deviations in color 414, region 412 in color space 410 can be predetermined. Region 412 may contain the color 414 that can be expected by pixels 230, 232, which correspond to positions on object 106 where illumination with spectrum 408 is completely reflected. Region 412 can be obtained through calibration.

[0181] Therefore, if the colors 414 of pixels 230 and 232, such as {R1, G1, B1}, are contained in region 412, then pixels 230 and 232 are considered to represent specular reflection.

[0182] Figure 4 and Figure 5 The extractions shown can be combined to identify specular reflection signals 206 even more reliably.

[0183] The following is for reference. Figure 5 Explain another method for extracting the specular reflection signal 206. Figure 5 The routines or methods used to extract specular reflection signals can be used alone or in combination. Figure 3 and Figure 4 One or two methods or routines can be used in combination.

[0184] according to Figure 5 The input image data 120 includes a reflection signal 500 and a second reflection signal 502, which can be considered as part of the reflection signal 202. The first reflection signal 500 may be included in a digital input image 130, which may consist only of the first reflection signal 500. The second reflection signal 500 may be included in another digital input image 130, which may consist only of the second reflection signal 502. Optionally, the first reflection signal 500 and the second reflection signal 502 may be components of a single digital input image 130, such as two image channels forming a single digital input image 130.

[0185] The first reflection signal 500 and / or the second reflection signal 502 can be color or monochrome digital input images. The first reflection signal 500 can represent a reflection image of light reflected from an object, the reflected light having a polarization different from that represented in the second reflection signal 502. The first reflection signal 500 may include a specular reflection signal 206, while the second reflection signal 502 may not include a specular reflection signal 206, i.e., it corresponds to a reflection signal 210 without a specular reflection signal.

[0186] For example, the first reflected signal 500 can be recorded by a camera 108 without a polarizing filter or by a camera 108 having a polarizing filter that is different from or oriented differently than the polarizing filter of another camera 108 recording the second reflected signal 502. In one embodiment, the polarizing filter can be part of an observation filter assembly 187, particularly an observation filter group 188 and / or a fluorescence filter group 190. In an alternative configuration, the first reflected signal 500 and the second reflected signal 502 can also be recorded by a single camera 108, which may include an on-chip polarizer.

[0187] In this case, the specular reflection signal 206 can be extracted by combining the first reflection signal 500 and the second reflection signal 502, in particular by subtracting them. This subtraction can be part of the separation or extraction routine 140, and in particular the specular reflection extraction routine 146.

[0188] exist Figure 6In another embodiment shown, the edge detection signal 600 is generated from the reflection signal 210, which has no specular reflection signal. Although the edge detection signal 600 could also be generated from the reflection signal 202, which includes a specular reflection signal, this is not preferred because the specular reflection 208 would result in additional edges that do not reflect the structure and / or shape of the object.

[0189] An edge detection signal 600 can be obtained by applying edge detection routine 148 to the reflected signal 210. The edge detection routine can be based on the first and / or second derivatives of one or more color channels or luminance, and it can be based on phase stretching transform and / or other edge detection routines typically included in image processing libraries. The edge detection routine can be anisotropic or isotropic.

[0190] The edge detection signal 600 can be normalized and combined with the fluorescence emission signal 204, and in particular the extracted fluorescence emission signal 280, to obtain a digital output image 160.

[0191] Then, the edge detection signal is combined with the extracted fluorescence emission signal 280 in the same manner as the extracted specular reflection signal 284 described above to produce a digital output image 160.

[0192] like Figure 7 As shown, when generating the digital output image 160, at least one of the extracted fluorescence emission signal 280 and edge detection signal 600 can be colored.

[0193] The extracted edge detection signal 600 can be colored as color 702. Color 702 can be the natural color at the corresponding input pixel of the digital input image 130 to which edge detection has been applied. Optionally, false color or pseudo-color can also be applied. Optionally, the extracted edge detection signal 600 can be normalized and converted to grayscale using signal / image combination routine 144.

[0194] exist Figure 8The diagram illustrates that the extracted fluorescence emission signal 280 can be combined with at least one signal from the group comprising the extracted reflection signal 282 (with or without specular reflection), the edge detection signal 600, and the extracted specular reflection signal 284. Any possible combination produces different output images 160a, 160b, 160c, and 160d. For example, digital output image 160a is generated by a combination of (only) the extracted fluorescence emission signal and the edge detection signal. Digital output image 160b can be generated by a combination of (only) the extracted fluorescence emission signal 280, the extracted specular reflection signal 284, and the edge detection signal 600. Another digital output image 160c can be generated by a combination of (only) the extracted digital reflection signal 282 without specular reflection, the edge detection signal 600, and the extracted fluorescence emission signal 280. Another digital output image 160d can be generated by a combination of (only) the extracted emission signal 280 and only the specular reflection signal 284.

[0195] Users can operate the mode selector 162 ( Figure 1 The user can select which digital output image 160a, 160b… to generate. Furthermore, the user can determine the relative intensity of each signal 280, 282, 600, 284 in the corresponding digital output images 160a, 160b… even if one or more signals are darkened relative to one or more other signals contained in the digital output image 160. Now refer to… Figure 9 This needs to be explained.

[0196] exist Figure 9 In this configuration, the mode selector 162 is displayed, for example, as a rotatable knob, which can be physical or a widget. The mode selector 162 has multiple positions I, II, ..., VIII. The number of positions depends on the application and implementation and can be more or fewer than displayed. Each of the positions I, II, ..., VIII corresponds to a different digital output image 160-I, 160-II, ..., 160-VIII; for example, it could correspond to... Figure 8 The digital output images are 160a, 160b, ..., 160d. Positions I, II, ..., VIII can be discrete, or there can be intermediate positions between positions I, II, ..., VIII, or continuous transitions between positions I, II, ..., VIII are possible.

[0197] Each of the different digital output images 160-I, 160-II… contains different combinations of extracted signals 282, 600, 284, and 280. By changing the state of the mode selector 162, a user selection signal 164 is generated. Figure 1The signal is sent to the digital processing unit 170, which then combines at least one signal 282, 600, 284, 280 at relative intensities predetermined by the corresponding positions of the mode selector 162.

[0198] For example, digital output image 160-I may consist solely of the extracted fluorescence emission signal 280. Digital output image 160-II may consist of a combination of the extracted fluorescence emission signal 280 and the extracted specular reflection signal 284 at their respective maximum intensities. At an intermediate position between positions I and II, the data processing device 170 may be configured to increase the intensity of the extracted specular reflection signal 284 relative to the extracted fluorescence emission signal 280 until position II is reached. Alternatively, this can be accomplished using additional input devices, such as widgets, physical buttons, or dials, which can be individually assigned to the signals. In the latter configuration, the relative intensity of each signal can be adjusted individually.

[0199] At position III, data processing device 170 is triggered by user selection signal 164 to generate digital output image 160-III, which includes a combination of edge detection signal 600, extracted specular reflection signal 284, and extracted fluorescence emission signal. If data processing device 170 receives user input signal 164 corresponding to position III of mode selector 162, data processing device 170 can be configured to generate digital output image 160-III. The closer the mode selector 162 is to position III, the stronger the edge detection signal 600 can be.

[0200] The digital output image 160-IV can correspond to a combination of the extracted fluorescence emission signal 280 and the edge detection signal 600. The closer the mode selector 162 is to position V, the weaker the intensity of the specular reflection signal 284 in the digital output image 160. Then, as the mode selector moves from position IV to position VI and the digital output image 160 transforms from digital output image 160-IV to 160-V, the scheme can continue to add the extracted reflection signal 282 without specular reflection, with the intensity gradually increasing.

[0201] Next, as the mode selector 162 moves from position V to position VI, the extracted specular reflection signal 284 can be blended. Further, the fluorescence emission signal 204 can be slowly blended away until it is no longer included in the digital output images 160-VII, and the digital output images 160-VII contain only the extracted reflection signal 282 with or without specular reflection. Of course, the data processing device 170 can be configured to simultaneously generate more than one digital output image 160 and display them simultaneously. Then, adjustments to each of the simultaneously generated digital output images 160 may be performed as described above.

[0202] Figure 10 An overview of method steps for generating a digital output image 160, for example, performed by a data processing device 170, is provided.

[0203] In step 1000 and optionally steps 1002 and 1004, input image data 122 is recorded. For example, in step 1000, a single digital color input image 130 may be recorded, which includes a reflection signal of specular reflection and a fluorescence emission signal. In steps 1002 and / or step 1004, additional input images 130 may be recorded. As described above, the digital input image 130 may contain any combination of reflection signals and fluorescence emission signals or portions thereof.

[0204] In step 1006, the fluorescence emission signal and / or reflection signal are extracted. The term extraction also extends to cases where a single digital input image 130 consists of the signal to be extracted. In this case, extraction simply involves using the corresponding digital input image as the signal to be extracted. Step 1006 may also include extracting the specular reflection signal from the extracted reflection signal 282 or directly from the input image data 122.

[0205] As a result of step 1006, extracted fluorescence signal 280, extracted specular reflection signal 284, and optionally extracted reflection signal 282 are obtained. As explained above, which of these signals is extracted can be determined by user selection signal 164.

[0206] In step 1010, an edge detection signal 600 can be calculated from the extracted reflection signal 282, which preferably does not include the specular reflection signal 284. Whether to calculate the edge detection signal 600 may depend on a user-selected signal.

[0207] In step 1012, for example as described above, any extracted signal 280, 282, 284 or 600 can be colored.

[0208] In step 1014, the relative intensity and / or color of the corresponding extracted signals 280, 600, 282, and 284 can be changed according to the user input signal 164. Coloring is optimal when the corresponding signals have already been normalized in previous steps, such as step 1006 or step 1012.

[0209] In step 1016, the digital output image 160 is generated by combining the extracted fluorescence emission signal 280 with one or more of the extracted signals 600, 282, and 284. Which signals in the combined digital output image 160 can be determined by a user-selectable signal.

[0210] Then, the digital output image 160 can be displayed in step 1018.

[0211] Some implementations involve microscopes, which include information about Figures 1 to 10 One or more of the systems described herein. Optionally, the microscope may be about Figures 1 to 10 It is part of or connected to one or more of the systems described in the text. Figure 11 A schematic diagram of a system 1100 configured to perform the methods described herein is shown. System 1100 includes a microscope 1110 and a computer system 1120. Microscope 1110 is configured to capture images and is connected to computer system 1120. Computer system 1120 is configured to perform at least a portion of the methods described herein. Computer system 1120 may be configured to execute machine learning algorithms. Computer system 1120 and microscope 1110 may be separate entities, but may also be integrated together in a common housing. Computer system 1120 may be part of the central processing system of microscope 1110 and / or computer system 1120 may be part of sub-components of microscope 1110, such as sensors, actors, cameras, or illumination units of microscope 1110.

[0212] Computer system 1120 may be a local computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in different locations, such as at local clients and / or one or more remote server farms and / or data centers). Computer system 1120 may include any circuitry or combination of circuitry. In one embodiment, computer system 1120 may include one or more processors, which may be of any type. As used herein, a processor may mean any type of computing circuitry, such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multi-core processor, field-programmable gate array (FPGA) (e.g., a microscope or microscope component (e.g., a camera)), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 1120 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuitry) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 1120 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for disposing of removable media such as CDs, flash memory cards, digital video discs (DVDs), etc. Computer system 1120 may also include a display device, one or more speakers and a keyboard and / or controller, the controller including a mouse, trackball, touchscreen, voice recognition device, or any other device that allows system users to input and receive information from computer system 1120.

[0213] Some or all of the method steps can be performed by (or using) hardware devices (e.g., processors, microprocessors, programmable computers, or electronic circuits). In some embodiments, such devices can perform one or more of the most important method steps.

[0214] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be carried out using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM, or flash memory) having electronically readable control signals stored thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to cause the corresponding methods to be executed. Therefore, the digital storage medium can be computer-readable.

[0215] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0216] Generally, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0217] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0218] In other words, therefore, an embodiment of the present invention is a computer program having program code that, when run on a computer, performs one of the methods described herein.

[0219] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) comprising a computer program stored thereon, which, when executed by a processor, is used to perform one of the methods described herein. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0220] Therefore, another embodiment of the invention represents a data stream or signal sequence for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0221] Another embodiment includes a processing component, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0222] Another embodiment includes a computer on which a computer program is installed for performing one of the methods described herein.

[0223] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0224] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0225] Figure Labels

[0226] 100 Medical Observation Equipment

[0227] 101L Stereo Subassembly for the Left Channel

[0228] 101R Stereo Subassembly for the Right Channel

[0229] 102 Digital Imaging System

[0230] 104 Eyepiece

[0231] 106 objects

[0232] 107 biological tissues

[0233] 108 digital camera

[0234] 110 Digital Reflective Camera

[0235] 111 Digital Fluorescence Camera

[0236] 111a Second Digital Fluorescence Camera

[0237] 112 Digital fluorescence input image

[0238] 114 Digital reflection input image

[0239] 116. Fluorescein, such as naturally occurring fluorophores in an object.

[0240] 118. Fluorescein, such as those artificially added to an object.

[0241] 120 Input image data

[0242] 130 digital input images

[0243] 132 Internal Display

[0244] 134 Direct Optical Path

[0245] 136 beam splitter

[0246] 140 Isolation or Extraction Routine

[0247] 142 Edge Detection Routine

[0248] 144 Signal / Image Combination Routine

[0249] 146 Specular Reflection Extraction Routine

[0250] 148 Edge Detection Routine

[0251] 160 digital output image

[0252] 160a, 160b, ... different digital output images

[0253] 160-I, 160-II, ... different digital output images

[0254] 162 Mode Selector

[0255] 164 User Selection Signal

[0256] 166 arrows

[0257] 168 On-chip polarizer

[0258] 170 Data Processing Equipment

[0259] 172 Output Interface

[0260] 174 Objective Lens

[0261] 176 Color Separation Components

[0262] 178 Lighting Components

[0263] 179 Illumination Filter

[0264] 180° Illumination Beam Splitter

[0265] 182 monitor

[0266] 184° field of view

[0267] 186 Computer

[0268] 187. Observe the filter assembly.

[0269] 188. Observe the filter group.

[0270] 190 Fluorescent Filter Set

[0271] 192 Dichroic Beam Splitter

[0272] 194 Memory

[0273] 196 Data Transmission Lines

[0274] 198 illumination light

[0275] 199 Light Source

[0276] 202 Reflected Signal

[0277] 204 fluorescence emission signal

[0278] 206 Specular reflection signal

[0279] 208 Specular Reflection

[0280] 210 Reflected signals without specular reflection

[0281] 212 Optional components of the reflected signal in NIR

[0282] 214 Components of the reflected signal

[0283] 220 passband

[0284] 222 Fluorescence emission signal of a fluorophore

[0285] 224 Fluorescence emission signal of another fluorophore

[0286] 230 input pixels

[0287] 232 pixels

[0288] 234 output pixels

[0289] Spectrum of 250 digital reflectance input image

[0290] Spectrum of 252 digital fluorescence input images

[0291] 260 Spectrum of reflected signal

[0292] 262 Spectrum of fluorescence emission signal 204

[0293] 264 Fluorescence emission spectrum 222

[0294] Spectrum of fluorescence emission signal 224 (266)

[0295] 270 Visible light

[0296] 272 NIR range

[0297] 280 Extracted fluorescence emission signal

[0298] 282 Extracted reflected signal

[0299] 284 Extracted specular reflection signal

[0300] Intensity distribution along line 304 (300)

[0301] 302 Intensity Threshold

[0302] 304 Reflected Signal / Lines in Digital Input Image

[0303] 306 color group

[0304] 308 (First) Color

[0305] 310 (Second) Color

[0306] 402 Sensitivity curve of the first color space coordinates

[0307] 404 Sensitivity curve of second color space coordinates

[0308] 406 Sensitivity curve of third color space coordinates

[0309] 408 The spectrum of light reflected from object 408

[0310] 410 color space

[0311] Color / area groups in the 412 color space

[0312] 414 pixels / specular reflection color

[0313] 500 First reflected signal

[0314] 502 Second Reflection Signal

[0315] 600 edge detection signal

[0316] 702 Color used for edge detection signals

[0317] 1000 Record the first digital input image

[0318] 1002 Record the second digital input image

[0319] 1004 Record the third color input image

[0320] 1006 Extraction / separation of fluorescence emission and / or reflection signals

[0321] 1010 Edge Detection

[0322] 1012 Coloring of one or more signals

[0323] 1014 Adjust / Change Relative Intensity

[0324] 1016 Combined Signal / Digital Output Image

[0325] 1018 Displays digital color output images

[0326] 1100 system, medical observation equipment

[0327] 1110 Microscope

[0328] 1120 Computer System

[0329] B, B1, Color space coordinates

[0330] G, G1 Color space coordinates

[0331] I Intensity

[0332] R, R1 Color space coordinates

[0333] λ wavelength

Claims

1. A data processing device (170) for observing an object (106) containing at least one fluorophore (116, 118), such as a microscope or an endoscope, wherein the data processing device (170) is configured to: - access input image data (120) representing an image of the object (106), the input image data containing: = a reflection signal (202) representing light (198) reflected from the object and containing a specular reflection signal (206), ≡ the specular reflection signal representing a specular reflection (208) from the object, and = a fluorescence emission signal (204) representing fluorescence emitted by the at least one fluorophore; - extract the specular reflection signal; and - generate a digital output image (106) from a combination of the extracted specular reflection signal and the fluorescence emission signal.

2. The data processing device (170) according to claim 1, wherein the data processing device is configured to: - extract the specular reflection signal (206) by extracting pixels (230) from the input image data (120) according to an intensity (I) of the pixels (230).

3. The data processing device (170) according to claim 1 or 2, wherein the data processing device is configured to: - extract the specular reflection signal (206) by extracting pixels (230) from the input image data (120) according to a color ({R1, G1, B1}) of the pixels (230).

4. The data processing device (170) according to any one of claims 1 to 3, wherein the image data (120) comprises a digital input image (130) containing at least a portion of the fluorescence emission signal (204) and at least a portion of the reflection signal (202).

5. The data processing device (170) according to claim 4, wherein the digital data processing device (170) is configured to - separate the reflection signal (202) contained in the digital input image (130) from the fluorescence emission signal (204) contained in the digital input image.

6. The data processing device (170) according to claim 4 or 5, wherein the image data (120) comprises a second digital input image (130) containing an additional portion of the reflection signal (202), and wherein the data processing device is configured to: - calculate the reflection signal (202) from the portion of the reflection signal (202) contained in the digital input image (114) and the additional portion of the reflection signal contained in the second digital input image (112).

7. The data processing device (170) according to any one of claims 4 to 6, wherein the data processing device is configured to: - calculate the fluorescence emission signal (204) from the portion of the reflection signal (202) contained in the digital input image (114) and the additional portion of the reflection signal contained in the second digital input image (112). wherein the image data (120) comprises a digital input image (130) containing the further portion of the fluorescent emission signal (204) and being the second digital input image (130) or a third digital input image (130); wherein the data processing device is configured to: - calculate the fluorescent emission signal (204) from the portion of the fluorescent emission signal (204) and the further portion of the fluorescent emission signal (204) contained in the digital input image (114).

8. The data processing device (170) according to any one of claims 1 to 7, wherein the input image data (120) comprises a first reflection signal (500) and a second reflection signal (502), wherein the first reflection signal (500) represents light (198) reflected from the object (106) and having a first polarization type; wherein the second reflection signal (502) represents light (198) reflected from the object and having a second polarization type, which is different from the first polarization type; wherein the first reflection signal (500) and the second reflection signal (502) are part of the reflection signal (202); and wherein the data processing device is configured to: - obtain the specular reflection signal (206) from a combination of the first digital input image (500) and the second digital input image (502).

9. The data processing device (100) according to any one of claims 1 to 8, wherein the data processing device (170) is configured to: - acquire a hue representing the color of the light (198) illuminating the object (106) and recorded in the reflection signal (202); and - assign the hue to the specular reflection signal in a digital output color image.

10. The data processing device (170) according to any one of claims 1 to 9, wherein the data processing device (170) is configured - to selectively generate the digital output image (160) from a combination of the specular reflection signal (206) and the fluorescent emission signal (204) or from a combination of the fluorescent emission signal (204) and the extracted reflection signal (210) without the specular reflection signal (206) in dependence on a user selection signal (164).

11. A medical observation device (100), such as a microscope or an endoscope, for observing an object (106) containing a fluorophore (116, 118), wherein the medical observation device (100) comprises: - a data processing device (170) according to any one of claims 1 to 10; - at least one camera (110, 111, 111a) adapted to record the input image data (120).

12. A computer-implemented method for a medical observation device (100), such as a microscope or an endoscope, the method comprising the steps of: - obtaining input image data (120) representing a reflection signal (202) of light (198) reflected from an object (106) containing a fluorophore (116, 118); and - processing the input image data (120) to obtain a digital output image (160) representing the object (106) and comprising a fluorescent emission signal (204) and a specular reflection signal (206). - accessing input image data (120) representing an image of an object (106), the object (106) comprising at least one fluorescent fluorophore (116, 118) that fluoresces, = the input image data comprising a reflection signal (202) and a fluorescent emission signal (204), ≡ the reflection signal representing light (198) reflected from the object, and ≡ comprising a specular reflection signal (206) representing a specular reflection from the object, = the fluorescent emission signal representing fluorescence emitted by the object; - extracting the specular reflection signal; and - generating a digital output image (106) from a combination of the extracted specular reflection signal and the fluorescent emission signal.

13. A method for using a medical observation device, the method comprising the steps of: - recording input image data (120); and - performing the computer-implemented method of claim 12.

14. A computer program product computer readable medium comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 12.

15. A computer readable medium comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 12.

Images