Data processing device and computer implementation invention for medical observation devices aimed at visualizing autofluorescence signals and fluorescence emission signals.

Abstract

The present invention relates to a data processing device (170) for a medical observation device (100), such as a microscope or endoscope, for observing an object (106). The data processing device (170) is configured to access input image data (120). The input image data includes an autofluorescence signal (224), which represents fluorescence emitted by a fluorophore (116) naturally present in the object, and a fluorescence emission signal (204), which represents fluorescence emitted by a fluorophore (118) artificially added to the object. Finally, the data processing device (170) is configured to generate a digital fluorescence color output image (160) from a combination of a fluorescence emission signal colored with a first color (308) and an autofluorescence signal colored with a second color (310) different from the first color.

Description

[Technical Field]

[0001] The present invention relates to a data processing device and a computer implementation method for medical observation devices such as microscopes and endoscopes used for observing objects. The present invention also relates to a medical observation device comprising such a data processing device, and a method for using a medical observation device, including a computer implementation method.

[0002] Medical observation devices such as microscopes and endoscopes are often used in conjunction with fluorophores.

[0003] One or more fluorophores may be artificially added to an object to mark specific parts of it. For example, in the case of medical observation devices, one or more fluorophores may be injected into a patient's body.

[0004] Some artificially added fluorophores attach to certain parts of an object, such as specific antibodies or (bi)chemicals. For example, the fluorophore 5-ALA is used to mark tumors. In another example, fluorophores can be used to mark the pathways through which they are transported. For example, ICG can be used to highlight blood flow.

[0005] Objects observed by medical observation devices may contain one or more naturally occurring fluorophores. The fluorescence of these naturally occurring fluorophores is called autofluorescence. While autofluorescence can be used as an additional diagnostic tool, it can also obscure the fluorescence of artificially added fluorophores, making the simultaneous use of both autofluorescence and fluorescence difficult.

[0006] Therefore, an object of the present invention is to provide a device and method that enables more efficient use of both autofluorescence and fluorescence emission from artificially added fluorophores.

[0007] This objective is addressed by a data processing device for medical observation devices such as microscopes and endoscopes used to observe objects. The data device accesses input image data that includes an autofluorescent signal representing fluorescence emitted by fluorophores naturally present in the object, and a fluorescence emission signal representing fluorescence emitted by fluorophores artificially added to the object. The device is configured to generate a digital fluorescence color output image from the synthesis of a fluorescence emission signal colored with a first color and an autofluorescent signal colored with a second color different from the first color.

[0008] The above objectives can also be achieved by a computer implementation method for processing input image data in medical observation devices such as microscopes and endoscopes, the method comprising: accessing input image data including an autofluorescence signal of an object, which represents fluorescence emitted by fluorophores naturally present in the object; and a fluorescence emission signal of an object, which represents fluorescence emitted by fluorophores artificially added to the object; and generating a digital fluorescence color output image from a synthesis of a fluorescence emission signal colored with a first color and an autofluorescence signal colored with a second color different from the first color.

[0009] Both the data processing device and the computer implementation method described above allow for easier distinction between the autofluorescence signal and the fluorescence emission signal, as they are colored with different colors.

[0010] Data processing devices and computer implementations can be further improved by one or more of the following features, which can be combined independently of each other. Any of the features described below can be used independently to improve data processing devices and / or computer implementations, even if each feature is described only in the context of data processing devices or only in the context of computer implementations.

[0011] For example, the first color and / or the second color may be pseudocolors, false colors, or natural colors. If the first color and / or the second color are pseudocolors, different colors are assigned to different intensities of the autofluorescence signal and / or fluorescence emission signal. If the first color and / or the second color are false colors or natural colors, the intensity or brightness of the color depends on the intensity of the autofluorescence signal and the fluorescence emission signal, respectively. In such cases, it is preferable that the hue of the first color and / or the second color is independent of the intensity of the autofluorescence signal and / or fluorescence emission signal, respectively. If the first color and / or the second color are false colors, they differ from the colors of the fluorescence signal and / or autofluorescence signal as perceived by human color vision. If the first color and / or the second color are natural colors, they correspond at least closely to the colors of the autofluorescence signal and / or fluorescence emission signal as perceived by human color vision.

[0012] In another embodiment, the distance between the first color and the second color in the color space of the fluorescent color output image may be greater than the distance between the natural color of the autofluorescence signal and the natural color of the fluorescence emission signal. This facilitates the visual distinction between these two signals. In another embodiment, the first color may have the same hue as the natural color of the fluorescence emission signal, and / or the second color may have the same hue as the natural color of the autofluorescence signal.

[0013] The input image data may consist of one or more digital input images, or may contain one or more digital input images. Each digital input image may contain multiple input pixels. If the input image data consists of two or more digital input images, these digital input images may have the same number of pixels or different numbers of pixels, may be represented in the same color space or different color spaces, and / or may have the same image aspect ratio or different image aspect ratios.

[0014] Throughout this specification, if a digital input image is a color image, it will be referred to as a “digital color input image.” Multispectral or hyperspectral images are considered color images. If a digital input image is a monochrome image, it will be referred to as a “digital monochrome input image.” In certain contexts, the general term “digital input image” is used regardless of whether the digital input image is color or monochrome. Therefore, a “digital input image” can be either a “digital color input image” or a “digital monochrome input image.”

[0015] In a digital color input image, each pixel, more specifically, the input pixel, can be a color pixel. In a digital fluorescent color output image, each pixel, more specifically, the output pixel, can be a color pixel. A color pixel, at least in combination, contains a set of color space coordinates that represent at least some of the color appearance parameters, such as hue, lightness, brightness, saturation, and color perception. A monochrome image may represent, for example, only light intensity values ​​on a grayscale.

[0016] In one specific example, the input image data may include at least one of the following: at least one digital color input image, at least one digital color input image and at least one digital monochrome input image, and at least two digital monochrome input images of the object. These are the minimum set of digital input images, each of which may contain both autofluorescence signals and fluorescence emission signals that can be reliably separated from one another.

[0017] A digital color input image may include at least one of an autofluorescence signal and a fluorescence emission signal, or at least a portion of the autofluorescence signal and / or at least a portion of the fluorescence emission signal. If a digital color input image includes two or more signals, these signals may be separated from each other by spectral unmixing.

[0018] A digital monochrome input image may contain only one of the autofluorescence signal or the fluorescence emission signal. Since a monochrome image contains only light intensity information, it is difficult to separate the two signals contained in a single monochrome image. Therefore, if the input image data includes two digital monochrome input images, one of the two digital monochrome input images may contain, or preferably consist only of, an autofluorescence signal, and the other of the two digital monochrome input images may contain, or preferably consist only of a fluorescence emission signal. In such a configuration, since the autofluorescence signal and the fluorescence emission signal are already contained in the two different digital input images, the autofluorescence signal and the fluorescence emission signal can be separated from each other with little effort. Of course, it is also possible for one digital monochrome input image to contain the sum of the autofluorescence signal and the fluorescence emission signal, and the other digital monochrome input image to contain only one of the autofluorescence signal or the fluorescence emission signal. In such a configuration, subtraction of the two digital monochrome input images yields a signal that is not contained in the digital monochrome input image containing only a single signal.

[0019] Input image data may also include a reflectance signal representing light reflected from the object. The reflectance signal, or a portion thereof, may be included in a digital color input image that includes at least one of a fluorescence emission signal and an autofluorescence signal. The reflectance signal may not be necessary to distinguish between the autofluorescence signal and the fluorescence emission signal. Therefore, it is advantageous if the data processing device is configured to separate either or both of the autofluorescence signal and the fluorescence emission signal from the reflectance signal.

[0020] If the digital input image includes two or more signals, such as any combination or subset of the group including autofluorescence signals, fluorescence emission signals, and reflectance signals, the contributions of these signals may be included in any one input pixel.

[0021] If the autofluorescence signals are to be processed separately from the fluorescence emission signals, and when they or parts thereof are included in a single digital input image, both signals need to be separated from each other or extracted from each other. This means that for each input pixel, the respective contribution of the signals included in the respective digital input image needs to be calculated. As mentioned above, one way to do this is to use spectral unmixing. Another way to do this is to apply a linear transformation to one or more input pixels containing the signals that need to be separated from each other. The linear transformation may comprise applying a transformation matrix to the color space coordinates of at least one input pixel, and the at least one input pixel may be either a color pixel or a monochrome input pixel.

[0022] The term extracting is considered synonymous with separating and isolating.

[0023] The data processing device may be configured for spectral unmixing of input image data and / or any digital input image included in the input image data. The data processing device may also be configured to apply a linear transformation to the input image data or any digital input image included in the input image data. This may be achieved in that the data processing device comprises an extraction or separation routine configured to separate or extract the autofluorescence signal and / or to extract or separate the fluorescence emission signal. The extraction or separation routine may comprise a linear transformation, in particular a transformation matrix. The extraction or separation routine may, independently thereof, comprise a spectral unmixing routine.

[0024] When the input image data includes two or more digital input images, the two or more digital input images are preferably aligned relative to each other. They preferably have the same field of view. The cameras used for recording different digital input images preferably have the same field of view and / or coaxial optical axes. In this context, it is further advantageous if two or more digital input images have the same number of pixels and the same aspect ratio, which results in a high correlation between their respective positions in each digital input image and the position on the object represented by the input pixels.

[0025] Aligning digital input images facilitates particularly processing them together on a pixel-by-pixel basis, since the aligned digital input images have corresponding input pixels, that is, input pixels that represent the same position of the object despite being located in different digital input images.

[0026] If the digital input images are not aligned and / or have different numbers of pixels and / or aspect ratios, corresponding pixels can be calculated using feature analysis. One method for identifying corresponding pixels is described, for example, in U.S. Patent No. 6,711,293 B. Alternatively, scale-invariant feature transform or the SURF algorithm from ETH Zurich may be used.

[0027] Both the fluorescence emission signal and the autofluorescence signal represent a color or monochrome image of the object in the respective fluorescence emission spectrum. For example, the fluorescence emission signal represents an image of the object in the fluorescence emission spectrum of at least one fluorophore artificially added to the object. The autofluorescence signal corresponds to an image of the object in at least a part of the fluorescence emission spectrum of fluorophores naturally contained in the object. According to another aspect, the autofluorescence signal and / or the fluorescence emission signal may include only a part of the respective fluorescence emission spectrum. The spectrum recorded in the autofluorescence signal and / or the fluorescence emission signal may be determined by an optical filter in front of the camera used to record them.

[0028] In an example of a digital monochrome input image consisting solely of or containing only autofluorescent signals, each input pixel represents the intensity of the autofluorescent signal at the location of the object corresponding to the input pixel.

[0029] In another independent example of a digital monochrome input image consisting solely of or containing only fluorescence emission signals, the input pixels represent the intensity of the fluorescence emission signal at the location of the object corresponding to the location of the input pixel.

[0030] In an example of a digital color input image consisting solely of or containing only autofluorescent signals, each input pixel represents at least some of the color appearance parameters of the fluorescent emission signal emitted from the object corresponding to the input pixel.

[0031] In an example of a digital color input image containing both fluorescence and autofluorescence signals, the input pixel represents a mixture of at least several color appearance parameters of the fluorescence and autofluorescence signals emitted by a single location of an object corresponding to the location of the input pixel.

[0032] The intensity of the autofluorescence signal and / or fluorescence emission signal may be 0 at input pixels representing locations within an object where no autofluorescence signal and / or fluorescence emission signal is emitted, or where emitted fluorescence signal and / or emitted autofluorescence signal is too weak to be recorded.

[0033] In a digital fluorescent color output image, each output pixel may be formed from a combination of the fluorescence emission signal at the corresponding input pixel of a digital input image containing a fluorescence emission signal colored with a first color, and the autofluorescence signal at the corresponding pixel of a digital input image containing an autofluorescence signal colored with a second color. As shown, the input pixels and the output pixels generated from the input pixels are corresponding pixels.

[0034] For example, the intensity of a first color in an output pixel of a digital fluorescent color output image may depend on the preferably normalized intensity of the fluorescence emission signal in the corresponding input pixel of a digital input image that includes a fluorescence emission signal. Similarly, the intensity of a second color in an output pixel of a digital fluorescent color output image may depend on the preferably normalized intensity of the autofluorescence signal in the corresponding input pixel of a digital input image that includes an autofluorescence signal.

[0035] Coloring a fluorescence emission signal with a first color may involve assigning an intensity to the output pixel that depends on the intensity of the fluorescence emission signal in the corresponding input pixel in the input image data.

[0036] Coloring the autofluorescent signal with a second color may include assigning an intensity to the second color in the output pixel that depends on the intensity of the fluorescence emission signal in the corresponding input pixel.

[0037] According to one embodiment, the input image data may include a digital color input image containing a fluorescence emission signal and an autofluorescence signal, and the data processing device is configured to extract the fluorescence emission signal from the digital color input image before coloring the fluorescence emission signal with a first color, and to extract the autofluorescence signal from the digital color input image before coloring the autofluorescence signal with a second color. As described above, the extraction of the fluorescence emission signal and the autofluorescence signal allows for the processing of these two signals separately.

[0038] According to another embodiment, the data processing device may be configured to extract at least one of an autofluorescence signal and a fluorescence emission signal from input image data by spectral unmixing. Various embodiments for coloring the autofluorescence signal and fluorescence emission signal are described below.

[0039] According to one embodiment, a data processing device may be configured to generate a digital fluorescent color output image in a color space having a set of color space coordinates. According to one embodiment, the color space may be an RGB color space. A first color may be represented by a first subset of the set of color space coordinates of the color space, and a second color may be represented by a second subset of the set of color space coordinates of the color space. The intersection of the first subset and the second subset may be empty. In that case, the data processing device may form a composite of a fluorescent emission signal colored with the first color and an autofluorescence signal colored with the second color by forming a union of the first subset and the second subset.

[0040] The intersection of the first subset and the second subset being empty means that the color space coordinates are contained in either the first subset or the second subset, or they are not contained in either the first subset or the second subset.

[0041] A first color of a fluorescence emission signal, particularly one of a predetermined intensity, may correspond to a predetermined first value of the color space coordinates of a first subset. For example, if the intensity of the fluorescence emission signal is normalized, a fluorescence emission signal intensity of 1 may be assigned to a predetermined value of the first color {255,0,0}. The value of the first color space coordinate in the output pixel can then be adjusted to the intensity of the fluorescence emission signal in the input pixel. For example, if the input pixel contains a fluorescence emission signal of intensity 0.5, the fluorescence emission signal of the output image will be colored as {127,0,0}.

[0042] For a preferably normalized autofluorescent signal with an intensity of 1, a given second color {0,255,0} can be assigned to, for example, the RGB color space. For an input pixel with an intensity of 0.5, the autofluorescent signal at the output pixel may be colored as {0,127,0}. For an input pixel having a fluorescence emission signal of intensity 0.5 and an autofluorescent signal of intensity 0.25, the combination of the first and second colors results in a final color that, in that case, arises from the combination of two subsets, such as {127,68,0}.

[0043] Therefore, the first subset of color space coordinates includes all color space coordinates except those in the second subset. Naturally, neither the first nor the second subset can be empty.

[0044] The output pixels of a digital fluorescent color output image may be formed pixel by pixel-by-pixel union of a first color color space coordinate whose intensity is adjusted according to the intensity of the fluorescence emission signal in the input pixel corresponding to the output pixel, and a second color color space coordinate in the input pixel whose intensity is adjusted according to the intensity of the autofluorescence signal in the input pixel.

[0045] In another embodiment, the data processing device may be configured to form a composite of a first-colored fluorescent emission signal and a second-colored autofluorescence signal by performing vector addition of the color space coordinates of a first-colored fluorescent emission signal and a second-colored autofluorescence signal at each output pixel of the digital fluorescent color output image.

[0046] In particular, the color space coordinates of the output pixels in a digital fluorescent color output image can be calculated by vector summing the color space coordinates of the corresponding fluorescence emission signal in the corresponding input pixel and the color space coordinates of the autofluorescence signal in the corresponding input pixel, and both input pixels may be in the same digital input image or in different digital input images.

[0047] In another embodiment, the data processing device may be configured to form a composite of a fluorescent emission signal colored with a first color and an autofluorescence signal colored with a second color by simultaneously applying a linear transformation to the fluorescent emission signal and the autofluorescence signal.

[0048] The linear transformation may be performed pixel by pixel, and the linear transformation may be applied to both input pixels containing only autofluorescence signals and input pixels containing only fluorescence emission signals.

[0049] For example, a union may be formed between the set of color space coordinates of input pixels containing only the fluorescence emission signal and the corresponding set of color space coordinates of input pixels containing only the autofluorescence signal. The resulting union has several color space coordinates corresponding to the sum of the color space coordinates of the input pixels. A linear transformation is then applied to this union to obtain the color space coordinate values ​​in the corresponding output pixels. In one example, the linear transformation may involve multiplication of the union, which can be treated as a vector, with a transformation matrix. In this case, the linear transformation is reduced to matrix multiplication, which can be quickly implemented and executed on a computer. The transformation matrix has a first dimension corresponding to the sum of the color space coordinates of two input pixels and a second dimension corresponding to the number of color space coordinates in the digital fluorescence color output image.

[0050] In some cases, it may be desirable to be able to change the relative intensity of the autofluorescence signal of the second color and the fluorescence emission signal of the first color. This facilitates the distinction between these two fluorescence signals. To achieve this, the data processing device may be configured to change the ratio of the fluorescence emission signal intensity to the autofluorescence signal intensity in the digital fluorescence color output image in response to a user-selected signal. In particular, the data processing device may be configured to change the ratio of the intensities of the first color and the second color in the digital fluorescence color output image in response to a user-selected signal.

[0051] Therefore, the user-selected signal can darken at least one of the autofluorescence signal and fluorescence emission signal of the output digital fluorescence color relative to the other.

[0052] A data processing device may be configured to receive a user selection signal.

[0053] To receive input image data and / or user selection signals, the data processing device may include a data interface that may have connectors and / or wireless connections for one-way or two-way data exchange.

[0054] To generate a user selection signal, the medical observation device may include a user input device such as a widget, or a manual operation component such as a dial, knob, switch, or button.

[0055] The objective described above can also be achieved in another embodiment by a data processing device configured to access a digital color input image containing a fluorescence emission signal representing fluorescence emitted by fluorophores artificially added to an object, and an autofluorescence signal representing autofluorescence naturally emitted by the object; extract the fluorescence emission signal from the digital color input image; color the extracted fluorescence signal to a first color set and extract the autofluorescence signal; color the extracted autofluorescence signal to a second color different from the first color; and generate a digital fluorescence color output image from the synthesis of the extracted fluorescence signal colored to the first color and the extracted autofluorescence signal colored to the second color.

[0056] According to another embodiment, a medical observation device, such as a microscope or endoscope, for observing an object containing at least two fluorophores, may comprise a data processing device in any of the above configurations and at least one digital camera adapted to record input image data.

[0057] According to one embodiment, a medical observation device may include a digital color camera for recording at least one of a fluorescence emission signal and an autofluorescence signal. The digital color camera may also be configured to record a reflectance signal instead of, or in addition to, the fluorescence emission signal and the autofluorescence signal.

[0058] In another embodiment, the medical observation device may comprise a digital color or monochrome camera for recording a fluorescence emission signal and another digital color or monochrome camera for recording an autofluorescence signal. In yet another embodiment, the medical observation device may comprise a first digital monochrome camera for recording a fluorescence emission signal and a second digital monochrome camera for recording an autofluorescence signal.

[0059] A digital color input image containing fluorescence emission signals and autofluorescence signals may be a digital fluorescence color input image recorded by a fluorescence camera of a medical observation device. Such a fluorescence camera may be combined with a set of fluorescence filters having passbands that are included in or corresponding to the fluorescence emission spectra of the respective fluorescence emission signals and autofluorescence signals. Thus, the fluorescence camera may be configured to record fluorescence specifically and not reflectance. In another variant, the fluorescence camera may also be configured to record reflectance signals, or a portion of reflectance signals, particularly a portion of reflectance signals included in the NIR (near-infrared) and / or fluorescence excitation spectra of at least one artificially added fluorophore.

[0060] None of the aforementioned digital cameras need to be configured to record the entire fluorescence emission signal and / or autofluorescence signal. For example, part of the fluorescence emission signal may be recorded by a first digital color or monochrome camera, and the other part by a second digital color or monochrome camera.

[0061] The objectives mentioned earlier are also achieved by methods for using medical observation devices such as endoscopes and microscopes, the methods comprising the steps of recording input image data and performing a computer implementation method in one of the above configurations.

[0062] Finally, the present invention also relates to a computer program product and / or computer-readable medium that, when executed by a computer, includes instructions causing the computer to execute a computer implementation method in any of the above configurations.

[0063] As used herein, the term "and / or" includes any combination of one or more of the related enumerated items and may be abbreviated as " / ".

[0064] While some embodiments are described in the context of the apparatus, it is clear that these embodiments also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, embodiments described in the context of a method step also represent a description of the corresponding block, item, or feature of the corresponding apparatus.

[0065] The present invention will be described below illustratively with reference to various examples and the drawings. No combination of features described and / or illustrated in the drawings and / or examples should be considered limiting. For example, features may be omitted from embodiments if they have technical effects, such as those described above, that are not required for a particular application. Conversely, features described below that are not part of the embodiments may be added if the technical effects associated with that particular feature are beneficial for a particular application.

[0066] Throughout the descriptions and drawings, the same reference numerals are used for elements that correspond to each other in terms of function and / or structure. [Brief explanation of the drawing]

[0067] [Figure 1] A schematic diagram of a medical observation device for generating a digital fluorescence color output image from at least one digital color input image is shown. [Figure 2] This diagram shows a schematic representation of the generation of a digital fluorescence color output image from the autofluorescence signal and fluorescence emission signal contained in two digital input images of the input image data. [Figure 3] This diagram shows a schematic representation of the generation of a digital fluorescence color output image from the autofluorescence signal and fluorescence emission signal contained in a single digital input image data. [Figure 4] A schematic diagram of an example of combining the extracted fluorescence emission signal and the extracted autofluorescence signal is shown. [Figure 5] A schematic diagram of another example of combining the extracted fluorescence emission signal and the extracted autofluorescence signal is shown. [Figure 6]A schematic diagram of another example of combining the extracted fluorescence emission signal and the extracted autofluorescence signal is shown. [Figure 7] This diagram provides a schematic overview of the steps involved in generating a digital fluorescent color output image. [Figure 8] A schematic diagram of a typical medical observation device is shown.

[0068] Figure 1 schematically shows a medical observation device 100. The medical observation device 100 may be a microscope or an endoscope. The main difference between a microscope and an endoscope is that, in an endoscope (not shown), the object 106 is observed via an optical fiber brought near the object 106 being studied, such as by insertion into the body containing the object, whereas in a microscope, the objective lens 174 is pointed above the object. The medical observation device in Figure 1 is a microscope, but the following explanation also applies to endoscopes.

[0069] The medical observation device 100 may be a medical observation device used in surgery. The medical observation device 100 may also be a medical observation device used in a laboratory, such as a laboratory microscope. The object of study 106 may consist of or contain biological tissue 107. The object 106 may be a part of a patient's body located within the field of view of the medical observation device 100.

[0070] The object 106 may contain one or more fluorophores 116, 118, 120. At least one fluorophore 116 may be a fluorophore naturally present in the object. For example, bone and blood naturally contain fluorophores. At least one fluorophore 118, 120 may be artificially added to the object 106, for example, by injecting it into biological tissue 107. Examples of fluorophores 118, 120 that can be artificially added to the object 106 are ICG, fluorescein, and / or 5-ALA.

[0071] The illustrated medical observation device 100 is a fluorescence imaging device. Therefore, the medical observation device is configured to observe, and preferably excite, the fluorescence of one or more fluorophores 116, 118, and 120.

[0072] The medical observation device 100 may be a stereoscopic device, as illustrated in Figure 1. Therefore, it may comprise two identical subassemblies 101L and 101R for each of the two stereoscopic channels. Since the two subassemblies 101L and 101R are identical in function and structure, the following description focuses on the right subassembly 101R, but applies equally to the left stereoscopic channel 101L.

[0073] The medical observation device 100 may, alternatively, be a monocular device. In this case, only one of the two subassemblies 101L and 101R may exist. Therefore, the following description also applies to the monocular medical observation device 100.

[0074] The operating medical observation device 100 provides input image data 122. The input image data 122 represents an imaged scene, i.e., a portion of an object within the field of view 184.

[0075] The input image data 122 may include one or more different digital input images 130. If the input image data 122 includes multiple digital input images 130, the different digital input images 130 should contain different spectral information. In such cases, each digital input image 130 of the input image data may be recorded at different wavelengths, preferably without spectral overlap or with synonymously minimal spectral overlap. Preferably, the spectra on which the different digital input images 130 of the input image data 122 are recorded are complementary.

[0076] To generate input image data 122, the digital imaging system 102 may include one or more digital cameras 108. The number of digital input images 130 included in the input image data 122 may depend on the number of cameras 108 used to generate the input image data 122. Depending on the type or configuration of the digital camera 108, the digital input images 130 may be color images or monochrome images.

[0077] The medical observation device 100 is configured to record in the input image data 122 both the fluorescence of a fluorophore 116 naturally present in the object and the fluorescence of at least one artificially added fluorophore 118, 120. For example, one or more fluorophores 118, 120 may be injected into the patient's body to mark a specific area of ​​interest, such as a tumor.

[0078] The fluorescence of naturally occurring fluorophores, also known as autofluorescence, is represented by the autofluorescence signal in the input image data 122. The autofluorescence signal is a component of the input image data 122 and represents the image of the object in the fluorescence spectrum of the fluorophore 116.

[0079] The fluorescence of artificially added fluorophores is represented by a fluorescence emission signal in the input image data 122. The fluorescence emission signal is also a component of the input image data 122 and represents an image of the object in the fluorescence emission spectra of at least one fluorophore 118, 120.

[0080] Each of these two fluorescence signals has a different spectral signature due to the different spectral characteristics of their respective fluorescence emission.

[0081] The input image data 122 may also include additional signals as extra components. For example, the medical observation device 100 may also be configured to record light reflected from an object in the input image data 122. The light reflected from the object is represented as a reflectance signal in the input signal. The reflectance signal represents the reflectance image of the object.

[0082] A single digital input image 130 may contain one or more signals in all or part of each of them. If a digital input image 130 contains two or more signals, it is preferable that it be a digital color input image so that the different signals can be distinguished from each other, for example, by their spectral signatures. Alternatively, a single digital input image 130 may be a digital monochrome image. In this case, it is preferable that the digital input image 130 is a digital monochrome input image.

[0083] As just one example, the digital imaging system 102 may include a digital reflectance camera 110 and one or more digital fluorescence cameras 111, 111a as a digital camera 108. The second (or third) digital fluorescence camera 111a is optional. In Figure 1, the second digital fluorescence camera 111a is shown only for the left stereoscopic channel 101L, but of course, it may also be present for the right stereoscopic channel 101R. Alternatively, the digital fluorescence camera for one stereoscopic channel may be used as the (first) digital fluorescence color camera 111, and the digital fluorescence camera for the other stereoscopic channel may be used as the second fluorescence camera 111a. Cameras 110, 111, and 111a may each be a color camera or a monochrome camera. A multispectral camera or hyperspectral camera is considered a color camera.

[0084] The digital reflectance camera 110 is configured to record a digital reflectance input image 114, i.e., a digital input image 130, which represents the reflectance of the object 106 and therefore may include all or at least a major portion of the reflectance signal. Preferably, the digital reflectance camera 110 is configured to record the digital input image 130 over a wide spectral range within the visible light spectrum. Thus, the digital input image 130 recorded by the digital reflectance camera faithfully represents the natural color of the object 106. This is important when the digital reflectance camera 110 is used to provide the user with an image of the object that is as close as possible to human perception of the object. The digital reflectance camera 110 may be a CCD, CMOS, or a multispectral or hyperspectral camera.

[0085] The digital input image 130 recorded by the digital reflectance camera 110 may also include at least a portion of the signal and / or fluorescence emission signal.

[0086] Each of at least one digital fluorescence camera 111, 111a is configured to record a different digital fluorescence image 112, i.e., a digital input image 130, which represents the fluorescence of the object 106 and therefore may include all or at least a major portion of the autofluorescence signal and the fluorescence emission signal. Fluorescence camera 111 may be configured to record the digital fluorescence input image 114 with only one or more fluorescence spectra of at least one fluorophore 116, 118, 120. Each fluorescence camera 111, 111a may be configured to record the fluorescence of different fluorophores. If the digital fluorescence image 112 is a color image, it may include both the autofluorescence signal and the fluorescence emission signal.

[0087] The fluorescence camera 111 may be configured to record a digital fluorescence image using only one or more narrowbands of light. These narrowbands must overlap with one or more fluorescence spectra of one or more fluorophores 116, 118, 120 whose fluorescence is to be recorded. Preferably, the fluorescence spectra of different fluorophores 116, 118, 120 are at least partially separated, preferably completely separated, i.e., do not overlap, so that the fluorescence camera 111 can record a digital color input image 130 representing two separate fluorescence bands spaced apart from each other.

[0088] Alternatively, if two fluorescence cameras 111, 111a are provided, each fluorescence camera 111, 111a preferably captures the fluorescence emission of different fluorophores and thus provides two digital fluorescence input images 112 containing different fluorescence emission signals. For example, fluorescence camera 111 captures the fluorescence emission signal and fluorescence camera 111a captures the autofluorescence signal. In this case, fluorescence cameras 111, 111a may be monochrome or color.

[0089] At least one fluorescence camera 111, 111a can capture a portion of the reflectance signal into the digital color input image 130. For example, a portion of the low-frequency side of the excitation spectrum used to induce fluorescence of a fluorophore can overlap with the fluorescence spectrum of this fluorophore and be recorded.

[0090] At least one fluorescence camera 111, 111a may be a monochrome camera, CCD, CMOS, or a multispectral or hyperspectral camera. Preferably, the white light color camera 110 and at least one fluorescence color camera 111 are of the same type, but this is not required.

[0091] Any combination of cameras 110, 111, and 111a may be combined to form a single multispectral or hyperspectral camera.

[0092] The fields of view 184 of cameras 110, 111, and 111a, if present, are preferably aligned or coincident and coaxial. It is preferable that cameras 110 and 111 provide identical fields of view 184 having the same viewpoint and focal length. This results in identical representations of the object 106 in images 112 and 114 produced by different cameras 110 and 111. Both cameras 110 and 111 may use the same objective lens 174.

[0093] If optical generation of a matching viewpoint and field of view is not possible, the matching can be generated by image processing, by applying a matching or alignment routine to the digital input image 130, as further described below.

[0094] Cameras 110, 111, and 111a, if present, are preferably operated in synchronous manner. Specifically, their exposure times may be synchronized. Thus, the medical observation device 100 may be configured to simultaneously generate digital input images 130.

[0095] Preferably, the gains of at least two cameras 110, 111, and 111a are synchronized, i.e., adjusted simultaneously in at least two cameras 110, 111, and 111a. Furthermore, the ratio of the gain applied to camera 110 to the gain applied to camera 111 and, if present, camera 111a may remain constant even when the gains are changed. Gamma correction and color adjustment or white balance may be switched off or kept constant.

[0096] An optical color separation assembly 176 may be provided to separate the light recorded in the digital reflectance input image 114 from the spectrum recorded in at least one digital fluorescence input image 112, that is, to separate the reflectance spectrum from the fluorescence spectrum. The color separation assembly 176 may include optical elements such as a beam splitter 192, and the beam splitter may be dichroic. The color separation assembly 176 may further or alternatively include an optical observation filter set 188 and / or an optical fluorescence filter set 190. The optical observation filter set 188 and the fluorescence filter set 190 may be part of the optical filter assembly 187.

[0097] The fluorescence filter set 190 is preferably configured to transmit light from one or more spectra of one or more fluorophores 116, 118, and 120, and to block light outside of one or more fluorescence spectra.

[0098] The fluorescence filter set 190 may comprise one or more optical bandpass filters, each containing one or more passbands. Each passband must overlap with the fluorescence emission spectra of the respective fluorophores 116, 118, and 120 whose fluorescence is to be recorded. Since the fluorescence filter set 190 is located in the optical path between the beam splitter 192 and the fluorescence color camera 111, only wavelengths within the passbands of the fluorescence filter set 190 are transmitted to the fluorescence color camera 111.

[0099] If two fluorescence cameras 111, 111a are used to capture different fluorescence emission spectra, the fluorescence filter set 190 may be equipped with different optical bandpass filters in front of each of the fluorescence color cameras 111, 111a. The passband of one bandpass filter may be included in the fluorescence emission spectrum of one fluorophore 116, and the passband of the other bandpass filter may be included in the fluorescence emission spectra of other fluorophores 116, 118 within the object 106.

[0100] The observation filter set 188 is preferably configured to block light from one or more fluorescence spectra of one or more fluorophores 116, 118. The observation filter set 188 may also be configured to block light from fluorescence excitation spectra.

[0101] The observation filter set 188 is preferably configured as a band-erasing filter, whose erasure band corresponds to, or at least includes, the passband of the fluorescence filter set 190. The observation filter set 188 is located in the optical path between the beam splitter 192 and the white light camera 110. Therefore, the white light camera 110 records only wavelengths outside the erasure band of the observation filter set 188, and thus outside the passband of the fluorescence filter set 190.

[0102] Either the observation filter set 188 or the fluorescence filter set 190 may be a tuning filter.

[0103] If beam splitter 192 is a dichroic beam splitter, then at least one of filter sets 188, 190 may be omitted, as optical spectral filtering in this case is already incorporated into the dichroic beam splitter. In that case, the above descriptions of the passband and erase band should apply to dichroic beam splitter 192 with the necessary modifications.

[0104] The medical observation device 100 may further comprise an illumination assembly 178, which is preferably configured to illuminate an object 106 via an objective lens 174 through which an imaging system 102 records at least one digital image 112, 114.

[0105] The illumination assembly 178 may be configured to selectively generate fluorescence excitation light that includes only white light, i.e., light uniformly distributed across the entire visible spectrum, and light of wavelengths that stimulate the fluorescence of at least one fluorophore 116, 118. The illumination light generated by the illumination assembly 178 may be supplied to the objective lens 174 using an illumination beam splitter 180.

[0106] The illumination assembly 178 may be configured to generate illumination light simultaneously in multiple discrete, particularly narrow-band wavelength ranges. These wavelength ranges may include any one or any combination of the following wavelength ranges:

[0107] One such discrete wavelength band may be entirely located within the fluorescence excitation spectrum of fluorophore 116. Another such wavelength band may be entirely located within the fluorescence emission spectrum of another fluorophore 118. Another such wavelength band may be limited to wavelengths greater than 700 nm and may be entirely located within the NIR range.

[0108] Simultaneous illumination of an object by any of the discrete wavelength bands described above can be achieved by a tuned light source 199, such as a light source comprising multiple LEDs of different colors, particularly different primary colors, configured to simultaneously generate light in these wavelength bands. Alternatively or additionally, the wavelength bands may be generated by using an illumination filter 179 having multiple passbands, the passbands preferably corresponding to the above wavelength bands. When such an illumination filter 179 is used, the light source 199 can generate white light, which is then filtered by the illumination filter 179 so that only light in the passbands illuminates the object 106.

[0109] The illumination filter 179 may be provided depending on at least one fluorophore whose fluorescence is to be induced and its particular excitation spectrum. For example, when 5-ALA is used as the fluorophore, the illumination filter may have a transmittance of 90% to 98% up to a wavelength of 425 nm, a transmittance of 0.5% to 0.7% at wavelengths of 450 nm to 460 nm, a transmittance of 0.1% or less at wavelengths of 460 nm to 535 nm, and substantially zero transmittance at wavelengths above 535 nm. The illumination filter 179 may be configured to allow NIR light to pass through. For example, the illumination filter 179 may include an NIR passband. The illumination filter 179 may further include a passband preferably entirely located within the fluorescence excitation spectrum of another fluorophore.

[0110] The medical observation device 100 can be adapted to different fluorophores or fluorophore sets by reconfiguring the color separation assembly 176, for example by replacing its optical elements such as filter sets 190 and / or 192 or a dichroic beam splitter 180.

[0111] Using the observation filter system 188 described above, the digital reflectance color camera 110 may be used to record at least a portion of the fluorescence emission signal and / or at least a portion of the autofluorescence signal when each wavelength passes through the passband of the observation filter system 188. In such cases, the illumination of the object 106 should preferably not include fluorescence emission wavelengths, as the fluorescence intensity is often low and the fluorescence signal may be difficult to detect. The same applies to the reflected signal and at least one digital fluorescence camera 111, 111a with necessary modifications.

[0112] The input image data 122 is processed by a data processing device 170. The data processing device 170 may be an integral part of the medical observation device 100. In one example, the processor is integrated into the medical observation device and also serves as a controller for controlling the hardware of the medical observation device 100, such as the brightness and / or emission spectrum of the light source 199 and / or any objective lens of the medical observation device 100 and / or any actuators of the medical observation device 100. In another example, the data processing device 170 is part of a general-purpose computer connected to the medical observation device for wired or wireless one-way or two-way data transfer.

[0113] The data processing device 170 may be a hardware module such as a microprocessor, or a software module. The data processing device 170 may also be a combination of both hardware and software modules, by using software modules configured to run on specific processors such as vector processors, floating-point graphics processors, parallel processors, and / or multiple processors. The data processing device 170 may also be part of a general-purpose computer 186, such as a PC. In another embodiment, the data processing device 170 is an embedded processor in a medical observation device 100.

[0114] The data processing device 170 is configured to access input image data 122 in the form of one or more digital input images 130, such as a digital white light color input image 114 or a digital fluorescence image 112. The data processing device 170 may also be configured to retrieve the digital input images 130 directly from memory 194 and / or from cameras 110, 111, and 111a, if present. Memory 194 may be part of the data processing device 170 or may reside elsewhere within the medical observation device 100.

[0115] The data processing device 170 is further configured to calculate a digital fluorescence color output image 160 from the input image data 122, specifically from the autofluorescence signal and fluorescence emission signal contained in the input image data 122.

[0116] The digital fluorescent color output image 160 is a color image represented in a color space. The color space of the digital fluorescent color output image may differ from the color space of any digital color input image included in the input image data 122. However, it is preferable that the color space of the digital fluorescent color output image 160 is the same as the color space of any of the digital color input images 130.

[0117] If the autofluorescence signal and / or fluorescence emission signal need to be processed separately, each signal needs to be separated or extracted from the input image data 122.

[0118] If the digital input image consists entirely of fluorescence emission signals or entirely of autofluorescence signals, extraction is straightforward. Processing such a digital input image 130 corresponds to processing its constituent signals.

[0119] However, if any two signals (or parts thereof) of a signal group including an autofluorescence signal, a fluorescence emission signal, and a reflectance signal are included in a single digital input image, this digital input image should in this case be a digital color input image, and each signal may need to be separated from the other signals so that it is processed separately. To achieve this, the digital processing device 170 is configured to separate or extract any signal from the signal group including the autofluorescence signal, the fluorescence emission signal, and the reflectance signal from the rest of the group.

[0120] For such separation or extraction of a signal, the digital processing device may include a separation or extraction routine 140 which can be stored in the memory 194 of the digital processing device 170. The separation or extraction routine 140 may include, for example, an unmixing routine 142 for spectrally unmixing a signal.

[0121] The extracted signals can be combined using the signal / image synthesis routine 144. The signal / image synthesis routine 144 can treat the extracted signals as an image.

[0122] A linear transformation routine 146, which may include a transformation matrix 148, may be used to transform the color of the extracted signal. The linear transformation may be included in the signal / image synthesis routine 144.

[0123] Any of routines 140 to 146 may be a software routine, a hardware-implemented routine, or a routine that combines software and hardware components.

[0124] The medical observation device 100 may include a user input device 162, which, when operated by the user, may generate a user selection signal 164 that can be communicated to the digital processing device 170. The user input device 162 may be, for example, a physical button, dial, slide or lever, or a widget representing a physical button, dial, slide, lever or widget.

[0125] By operating the user input device 162, the user can determine which signals or combinations of signals extracted from the digital input image 130 are displayed, and / or the intensity at which each signal is displayed, either absolutely or relative to one or more other signals. These different display modes are shown as I, II, III, etc.

[0126] The digital fluorescent color output image 160 can be displayed on a display 132 integrated with the medical observation device 100. For example, the display 132 may be integrated with the eyepiece or eye lens 104 of the medical observation device 100.

[0127] The digital fluorescence color output image 160 is preferably generated in real time, that is, the digital fluorescence color output image 160 is generated from the set of digital color input images 130 before the next set is generated by at least two cameras 110, 111, and 111a.

[0128] The medical observation device 100 may have a direct optical path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In such a case, the display may be a translucent display 132 located within the direct optical path 134, or the display may be projected into the direct optical path 134. A beam splitter 136 may be provided to split the light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed towards the eyepiece 104.

[0129] Alternatively, the medical observation device 100 may not have a direct optical path 134 and may only display images from the integrated display 132. As a further alternative, the medical observation device may not have a display at all.

[0130] The medical observation device 100 may have an output interface 172 to which one or more (external) displays 182 can be connected. For this purpose, the output interface 172 may have standardized connectors and data transmission protocols such as USB, HDMI, DVI, DisplayPort, Bluetooth, and / or others. The external displays may be monitors, 3D goggles, eyepieces, etc. Any combination of external displays can be connected to the output interface 172.

[0131] The computer 186 and / or data processing device 170 are connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission lines may be wired or wireless, or partially wired or partially wireless. The computer 186 and / or data processing device 170 do not have to be physically integrated with the medical observation device 100 and may be located physically separate from the digital imaging system 102. For this purpose, the digital imaging system 102 and the computer 186 and / or data processing device 170 may be connected to a network such as a LAN, WLAN or WAN, to which at least one display 182 is also connected.

[0132] In one modification, the medical observation device 100 is stereoscopic but may have only two cameras, one for each stereoscopic channel. One stereoscopic channel uses a fluorescence color camera 111, configured to selectively record white light reflectance, while the other stereoscopic channel uses a white light color camera 110. Such an arrangement provides a stereoscopic white light color input image when fluorescence is not used, and provides a monocular white light color input image and a monocular fluorescence color input image when fluorescence is used. The above and below descriptions apply equally to this configuration.

[0133] In one embodiment, the input image data may include at least one digital color input image 130 that includes both an autofluorescence signal and a fluorescence emission signal. This is the case when all fluorescence emitted by the object is recorded in at least one digital color input image.

[0134] In another embodiment, the input image data 122 may include at least one digital monochrome input image and at least one digital color input image. The digital monochrome input image includes at least a portion of a fluorescence emission signal or at least a portion of an autofluorescence signal. The digital color input image may include a fluorescence emission signal if the digital monochrome color input image includes an autofluorescence signal, or may include a portion of an autofluorescence signal if the digital monochrome color input image includes an autofluorescence signal.

[0135] Input image data contains multiple pixels. Pixels may be color pixels or monochrome pixels. Monochrome pixels represent only intensity, for example, as a grayscale image. Color pixels contain information about at least some of the color appearance parameters, such as hue, lightness, brightness, saturation, color sensitivity, and saturation. Color pixels are recorded using a digital color camera, using color bands, or equivalently, color channels or primary colors in a color space.

[0136] A color space contains at least three of such color channels. In a color space, each color channel is represented by different color space coordinates. Color space transformations can be used to convert between different color spaces. In different color spaces, the same color is represented by different color space coordinates. Each pixel of the digital color input image 130 contains a set of color space coordinates that together represent the color of each pixel. Thus, each color band may be considered to represent a color space axis, and each color may be considered a point in the color space defined by a vector that points to this color, i.e., a color space coordinate. Thus, adding two colors is equivalent to vector addition. If one color has color space coordinates {x1, y1, z1} and a second color has color space coordinates {x2, y2, z2}, then the sum of these two colors corresponds to the color {x1+x2, y1+y2, z1+z2}.

[0137] For example, a digital color input image 130, or more generally, an input image data 120, may be recorded in the RGB color space using primary colors or color bands, i.e., color space coordinates R, G, and B. Alternatively, each digital color input image 130 may be recorded in a different color space and / or represent a multispectral color input image or a hyperspectral color input image. A set of digital input images, such as a digital white light color input image 114 and a digital fluorescent color input image 112, does not need to be recorded in the same color space, but is preferred.

[0138] In the RGB color space, each color is represented by a triplet of three color space coordinates in integer form, where each integer represents the intensity of one of the primary colors R, G, and B. For example, the strongest red is represented by the triplet [255,0,0], the strongest green by [0,255,0], and the strongest blue by [0,0,255]. Thus, the RGB color space is a three-dimensional space, and the CMYK color space is a four-dimensional space. A color can be thought of as a point in the color space pointed to by a vector such as [0,0,255]. A multispectral or hyperspectral color space with n color bands correspondingly results in an n-dimensional color space where each color is represented by a triplet of n color space coordinates.

[0139] Figure 2 shows an example of how a digital fluorescent color output image 160 is generated from input image data 122. As shown in the figure, the input image data 122 may include two digital input images 130, each of which may be either a color image or a monochrome image. Each of the digital input images 130 includes or consists of input pixels 230. If each digital input image 130 is a color image, its input pixels 230 become color pixels. If each digital input image 130 is a monochrome image, its input pixels 230 become monochrome pixels.

[0140] The first digital color input image 212 of the input image data 122 may correspond to the digital reflectance input image 114 described above. The second digital input image 214 of the input image data 122 may correspond to the fluorescence input image 114.

[0141] The first digital input image 212 includes an autofluorescent signal 224. Therefore, each input pixel 230 includes a contribution from the autofluorescent signal 224. This contribution may be zero for some pixels that were unable to receive the autofluorescent signal.

[0142] Reference numeral 240 shows an exemplary spectrum of the first digital input image 212. Spectrum 240 shows the intensity I at wavelength λ of light received by the first digital color input image 212. The spectrum 250 of autofluorescence 224, corresponding to the autofluorescence emission spectrum, is found to be at least partially included in the passband 220 of, for example, the optical filter assembly 187. The passband 220 may be wider or narrower than spectrum 250.

[0143] If the first digital input image 212 is a monochrome image, it should contain only a single signal. In the case of a monochrome image, the first digital input image 212 may consist of an autofluorescence signal 224. If the first digital input image 212 is a color image, additional signals may be included. In the case of a color image, the first digital input image 212 may, for example, include at least a portion of the reflectance signal 202 in addition to the autofluorescence signal 224. The spectrum 252 of the reflectance signal 202 is quantitatively shown in spectrum 240.

[0144] In some input pixels 230, the reflectance signal 202 and the autofluorescence signal 224 may overlap. Such input pixels 230 contain both a local portion of the reflectance signal 202 and a local portion of the autofluorescence signal 224. If the input pixel 230 is a color pixel, the two signals 202 and 224 may be separated from each other.

[0145] The reflectance signal 202 can be generated, for example, by illuminating the object 106 with light having wavelengths of the fluorescence excitation spectra of other fluorophores 118, 120 whose fluorescence is to be induced. A portion of such illumination spectrum may overlap with the passband 220. Naturally, the spectrum 240 of the first digital input image may include additional components of other wavelengths λ, preferably outside the bandpass 220.

[0146] If the first digital input image 130 contains only the autofluorescence signal 224, extraction of the autofluorescence signal 224 is unnecessary. In this case, the first digital input image 130 already represents the autofluorescence signal 224.

[0147] However, if the first digital input image 212 contains another signal, such as a reflectance signal 202, the autofluorescence signal 224 must be separated or extracted, if it should be processed separately. Such extraction or separation can be performed, in particular, using spectral unmixing by the data processing device 170.

[0148] Once the autofluorescent signal 224 is extracted or separated as indicated by reference numeral 290, it may be colored, for example, by color 218. The intensity of color 218 may depend on the intensity of the extracted autofluorescent signal 224 in the input pixel 230. If the first digital input image 130 is a color image, the color of the extracted autofluorescent signal 290 may be preserved, but its intensity and / or brightness may be normalized.

[0149] Color 218 may be a pseudocolor, a false color, or a natural color. In the case of a pseudocolor, different colors are assigned to the pixel depending on the intensity of the autofluorescence signal 224 in that pixel. In the case of a false color or a natural color, the brightness or intensity of this color may depend on the intensity of the autofluorescence signal 224 in that pixel. A false color is a color different from the color of fluorescence emitted by an autofluorescent fluorophore as perceived by the human eye. A natural color corresponds to the color of fluorescence emitted by an autofluorescent fluorophore as perceived by the human eye.

[0150] The second digital input image 214 includes the fluorescence emission signal 204 in the example shown in Figure 2. Again, the second digital input image 214 may include additional signals, such as a portion of the autofluorescence signal 224 and / or a portion of the reflectance signal 202. Similar to the first digital input image 212, the second digital input image 214 is preferably a color image so that the fluorescence emission signal 204 can be separated from other signals contained in the second digital input image 214. If the second digital input image 214 is a monochrome image, it should contain only the fluorescence emission signal 204.

[0151] A sample spectrum of light recorded by the second digital input image 214 is shown in reference numeral 242. The second digital input image 214 records light passing through the passband 220 of the optical filter assembly 187. If the autofluorescence emission spectrum 250 overlaps with the passband 220 containing the fluorescence emission spectrum 254 of at least one fluorophore 116, 118 such as 5-ALA and / or ICG, as shown in just one example, a portion of the autofluorescence signal 224 may leak into the second digital input image 214. The second digital input image 214 may correspond to the fluorescence input image 114.

[0152] In some cases, it may be preferable for the object to be illuminated in an additional narrowband to provide additional information. For example, the object 106 may be illuminated in the NIR range 272 outside the visible light range 270.

[0153] If the second digital input image 214 contains only the fluorescence emission signal 204, extraction or separation of this signal is unnecessary. However, if the second digital image 214 contains additional signals such as the autofluorescence signal 224 and / or the reflectance signal 202, or parts thereof, extraction or separation of the fluorescence emission signal 204 is necessary so that it can be processed separately. Separation of the fluorescence emission signal 204 from the second digital input image 130 may be performed, for example, using spectral unmixing. Spectral unmixing can be performed by the data processing device 170 described above.

[0154] The extracted autofluorescence signal 290 is preferably colored using color 216. The intensity and / or luminance or any other color parameters of color 216 may depend on the intensity of the extracted fluorescence signal 290 in the input pixel 230. If the extracted fluorescence emission signal 280 is already a color image, its color may also be preserved. As described above in the context of color 218, color 216 may be a pseudocolor, a false color, or a natural color.

[0155] The digital fluorescent color output image 160 is obtained by combining an autofluorescence signal 290, which is (optionally) extracted, colored, and preferably normalized, with a fluorescence emission signal 280, which is (optionally) extracted, colored, and preferably normalized. Each output pixel 232 of the digital fluorescent color output image 160 includes a combination of an autofluorescence signal 224 contained in the corresponding input pixel 230 of the first digital input image 212 and a fluorescence emission signal 204 in the corresponding input pixel 230 of the second digital image 214. The digital fluorescent color output image 160 may have overlaps 222 where both the autofluorescence signal and the fluorescence emission signal overlap.

[0156] Figure 3 shows an embodiment in which the input image data 122 includes only a single digital color input image 130, which may be, for example, a digital fluorescent color input image 112. The digital color input image 130 includes both an autofluorescence signal 224 and a fluorescence emission signal 204. Furthermore, the digital fluorescent input image 130 may include a reflectance signal 202. A sample spectrum of such a digital color input image 130 is shown by reference numeral 300. As previously stated, the autofluorescence signal 224 and the fluorescence emission signal 204 need to be extracted for each input pixel 230, yielding extracted autofluorescence signals 290 and extracted fluorescence emission signals 280. Once the extracted signals 280, 290 are obtained, further steps may be as described with reference to Figure 2.

[0157] Referring to Figures 4 to 6, we will now explain how the extracted autofluorescence signal 290 and the extracted fluorescence emission signal 280 can be combined using different colors 216 and 218 to make them more distinguishable to the user. As just one example, the digital fluorescence color output image 160 is represented in an RGB color space with three color bands R, G, and B, and the color of the pixels is represented accordingly in three color space coordinates [rgb].

[0158] As shown in Figure 4, the fluorescence emission signal 280 and the autofluorescence signal 290 can each be assigned to different color channels in the color space. For example, the extracted autofluorescence signal 290 may be assigned to the green color channel G, and the extracted fluorescence emission signal 280 may be assigned to the red color channel R. The blue color channel B is not assigned a signal and may therefore be set to, for example, 0.

[0159] This synthesis is simple and useful, especially when the color bands correspond at least approximately to the hue of the autofluorescence signal and the hue of the fluorescence emission signal, respectively. This may be the case when the fluorescence emission of each fluorophore 116, 118, and 120 has a hue corresponding to the color of the color band as perceived by human color vision.

[0160] For example, the fluorescence emission of 5-ALA has a red hue. Therefore, the extracted fluorescence emission signal 280 representing the fluorescence of 5-ALA may be assigned the red color channel R. The autofluorescence signal 290 may be assigned the green color channel, in particular, if the autofluorescence is perceived as greenish by human perception.

[0161] As shown in Figure 5, the color space coordinates of the two corresponding pixels 230 of the extracted autofluorescence signal 290 and the extracted fluorescence emission signal 280 can be simply added together to obtain the color space coordinates of the corresponding output pixels 232 of the digital fluorescence color output image 160. Therefore, if the pixel 230 of the extracted autofluorescence signal 290 has color space coordinates {r1, g1, b1} and the corresponding pixel 230 of the extracted fluorescence emission signal 280 has color space coordinates {r2, g2, b2}, then vector summing these color space coordinates yields the color space coordinates {r1+r2, g1+g2, b1+b2}.

[0162] According to Figure 6, the extracted autofluorescent signal 290 is in color space coordinate x at each pixel 230. i It can be expressed by, where i=1,...,M. Therefore, in the case of a monochrome image, i=1, and x1 simply represents the intensity or brightness of pixel 230. When the extracted fluorescence emission signal 280 is expressed in the RGB color space, y1=r, y2=g and y3=b. Naturally, y i This may represent any color space, or even multispectral or hyperspectral color coordinates.

[0163] Regardless of the representation of the extracted fluorescence emission signal 280, the extracted autofluorescence signal 290 may be monochrome or color. Here, the color space coordinates at each pixel are y j It may also be expressed as such, where j=1,...,N, and N does not necessarily have to be equal to M, so both signals may be represented in different color spaces (considering the monochrome representation as a very simple color space).

[0164] Color space coordinate x of the corresponding pixel 230 of the extracted autofluorescence signal 290 and the extracted fluorescence emission signal 280 i , y j for converting, linear transformation 146 may be applied to the color space coordinate x i , y j . The linear transformation 146 may include a transformation matrix M indicated by reference numeral 148 in FIG. 1 as part of the data processing device 170.

[0165] The linear transformation matrix M is multiplied by a vector consisting of coordinates x i , y j {{x i}{y j}}. Therefore, the input vector is in the form of {x1,…,x M ,y1,…,y N} and thus has a dimension of (M+N). The result of the linear transformation is the color space coordinate or color vector {r k}, k=1,…,K at the corresponding output pixel 232 of the digital fluorescence color output image 160. When the digital fluorescence color output image 160 is represented in the RGB color space, it corresponds to three RGB color space coordinates. Therefore, the linear transformation matrix M has a dimension of (M+N) in one direction and K in the other direction.

[0166] By assigning different colors 216, 218 and using the synthesis scheme shown in FIGS. 5 to 6, it becomes possible to spread the spectra of the autofluorescence signal and the fluorescence emission signal over a wider range of colors, thereby providing better visualization of the output pixel 232 containing both the autofluorescence signal 224 and the fluorescence emission signal 204.

[0167] FIG. 7 shows an overview of steps that may be used to generate the digital fluorescence color output image 160.

[0168] In the first step 700, input image data 122 is recorded, retrieved, or accessed. This may include recording one or more digital input images 130 with a corresponding number of cameras. For example, in step 702, a first digital input image 130 may be recorded, in step 704, a second digital input image 130 may be recorded, and in step 706, a third digital input image 130 may be recorded. Steps 702, 704, and 706 may be performed simultaneously or immediately after each other.

[0169] In step 708, the autofluorescence signals are separated from the input image data or each digital input image 130. This step may be performed, for example, by the extraction routine 144. As a result, the extracted fluorescence emission signal 280 and the extracted autofluorescence signal 290 are obtained. If the digital input image 130 consists of signals 204, 224 that would otherwise have needed to be extracted, step 708 is unnecessary. In such cases, the digital input image 130 can be used as the signal to be extracted.

[0170] In step 710, intermediate processing may be performed on the extracted fluorescence emission signal 280. Intermediate processing may include normalizing the extracted fluorescence emission signal. Intermediate processing 710 may also include assigning a color to the extracted fluorescence emission signal, either alternatively or cumulatively.

[0171] In step 712, intermediate processing may be performed on the extracted autofluorescent signal 290. Step 712 may correspond to step 710 described above.

[0172] In step 714, the extracted autofluorescence signal 290 may be combined with the extracted fluorescence emission signal 280 to generate a digital fluorescence color output image 160. Step 714 may be performed, for example, by a signal / image synthesis routine 144. Step 714 may include the step of coloring the extracted autofluorescence signal 280 with a different color than the extracted fluorescence emission signal 290.

[0173] In step 716, the digital fluorescent color output image 160 is displayed on, for example, at least one of the displays 132, 182.

[0174] Some embodiments relate to a microscope comprising a system such as those described in relation to one or more of Figures 1 to 7. Alternatively, the microscope may be part of or connected to a system such as those described in relation to one or more of Figures 1 to 7. Figure 8 shows a schematic diagram of a system 800 configured to perform the method described herein. The system 800 comprises a microscope 810 and a computer system 820. The microscope 810 is configured to take images and is connected to the computer system 820. The computer system 820 is configured to perform at least a portion of the method described herein. The computer system 820 may be configured to perform machine learning algorithms. The computer system 820 and the microscope 810 may be separate entities or they may be integrated into a single common housing. The computer system 820 may be part of the central processing system of the microscope 810, and / or the computer system 820 may be part of a subcomponent of the microscope 810, such as a sensor, actor, camera, or illumination unit of the microscope 810.

[0175] The computer system 820 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed to various locations, e.g., local clients and / or one or more remote server farms and / or data centers). The computer system 820 may comprise any circuit or combination of circuits. In one embodiment, the computer system 820 may include one or more processors, which may be of any type. As used herein, a processor may mean, but is not limited to, any type of computing circuit, such as a microprocessor, microcontroller, composite instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multicore processor, field-programmable gate array (FPGA) for, for example, a microscope or microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 820 may include custom circuits, application-specific integrated circuits (AS1Cs), and one or more circuits (such as communication circuits) for use in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 820 may also include one or more storage devices, which may include one or more memory elements suited to a particular application, such as main memory in the form of random-access memory (RAM), one or more hard drives, and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, and digital video discs (DVDs).The computer system 820 may also include a display device, one or more speakers, and a keyboard and / or controller, the controller of which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input and receive information from the computer system 820.

[0176] Some or all of the method steps may be performed by (or using) a hardware device such as a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0177] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be carried out using a digital storage medium storing electronically readable control signals that cooperate (or can cooperate) with a programmable computer system as each method performs, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, and non-temporary storage medium such as EPROM, EEPROM, or FLASH memory. Thus, the digital storage medium can be computer-readable.

[0178] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein is performed.

[0179] Generally, embodiments of the present invention can be implemented as a computer program product having program code, the program code operates to perform one of the methods when the computer program product runs on a computer. The program code may be stored, for example, in a machine-readable carrier.

[0180] Other embodiments include a computer program stored in a machine-readable carrier for performing one of the methods described herein.

[0181] In other words, thus, one embodiment of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is running on a computer.

[0182] Accordingly, a further embodiment of the present invention is a storage medium (or data carrier, or computer-readable medium) storing a computer program for performing one of the methods described herein when performed by a processor. A data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transient. A further embodiment of the present invention is the apparatus described herein, comprising a processor and a storage medium.

[0183] Accordingly, a further embodiment of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0184] Further embodiments include processing means, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0185] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0186] Further embodiments of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0187] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the method herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods herein. Generally, the method is preferably performed by any hardware device. [Explanation of Symbols]

[0188] 100 Medical Observation Devices 101L Stereoscopic subassembly for left channel 101R Stereoscopic subassembly for right channel 102 Digital Imaging System 104 Eyepiece 106 Object 107 Living tissue 108 Digital Cameras 110 Digital reflectance camera 111 Digital fluorescent camera 111a Second digital fluorescent camera 112 Digital fluorescence input image 114 Digital reflectance input image 116 Fluorophores naturally present in the subject 118 Fluorophores artificially added to the object 120 Another fluorophore artificially added to the object 122 Input image data 130 Digital Input Images 132 All-in-One Display 134 Direct optical path 136 Beam Splitter 140 Extraction / Separation Routines 142 Unmixing Routine 144 Signal / Image Synthesis Routines 146 Linear Transformation 148 Transformation Matrix 160 Digital fluorescent color output images 162 User Input Devices 164 User Selection Signal 170 Data Processing Devices 172 Output Interfaces 174 Objective lens 176 Color Separation Assembly 178 Lighting Assembly 179 Lighting Filter 180 Lighting Beam Splitter 182 displays 184 Field of view 186 Computers 187 Optical filter assembly 188 Observation Filter Set 190 Fluorescent Filter Set 192 Dichroic Beam Splitter 194 memory 196 data transmission lines 198 Illumination light 199 Light source 200 input image sets 202 Reflectance signal 204 Fluorescence emission signal 212 First digital input image 214 Second digital input image 216 The first color 218 Second Color 220 Passband 222 Overlap 224 Autofluorescence signal 230 input pixels 232 output pixels 240 Spectrum of the first digital input image 242 Spectrum of the second digital input image 250 Autofluorescence Signal Spectrum 252 Spectrum of the reflectance signal 254 Spectrum of fluorescence emission signal 270 visible light 272 NIR range 280 Extracted fluorescence emission signals 290 Extracted autofluorescence signals 300 Spectra of a digital color input image 402 The first color space coordinate, e.g., the sensitivity curve of B 404 Second color space coordinates, e.g., sensitivity curve of G 406 A third color space coordinate, e.g., the sensitivity curve of R. 700 Recording digital input image data. 700 Record the first digital input image. 702 Recording a second digital input image 704 Record the third color input image. 706 Extraction / separation of fluorescence emission signal and reflectance signal. 710 Intermediate Processing 712 Intermediate Processing 714. To synthesize and create digital fluorescent color output images. 900 System 910 Microscope 920 Computers x, B, B1, color space coordinates y, G, G1 color space coordinates I strength I, II, III… Imaging Modes I, j, k coordinate indices M ik Linear transformation matrix R, R1 color space coordinates r k Color space coordinates x j Color space coordinates y j Color space coordinates λ wavelength

Claims

1. A data processing device (170) for a medical observation device (100) such as a microscope or endoscope, for observing an object (106), The aforementioned data processing device (170) An autofluorescent signal (224) that represents fluorescence emitted by a fluorophore (116) naturally contained in the object, and A fluorescence emission signal (204) that represents fluorescence emitted by a fluorophore (118) artificially added to the object. Access the input image data (120) that includes, A digital fluorescent color output image (160) is generated by combining the fluorescent emission signal colored with a first color (216) and the autofluorescence signal colored with a second color (218) that is different from the first color. A data processing device (170) is configured as follows.

2. The input image data (120) includes a digital color input image (114). The digital color input image includes the fluorescence emission signal (204) and the autofluorescence signal (224), The aforementioned data processing device is Before coloring the fluorescence emission signal with the first color, the fluorescence emission signal is extracted from the digital color input image. Before coloring the autofluorescent signal with the second color (218), the autofluorescent signal is extracted from the digital color input image. It is structured in such a way. The data processing device (170) according to claim 1.

3. The aforementioned data processing device is At least one of the autofluorescence signal (224) and the fluorescence emission signal (204) is extracted from the input image data (120) by spectral unmixing. It is structured in such a way. A data processing device (170) according to claim 1 or 2.

4. The aforementioned input image data (120) includes input pixels (230), The aforementioned digital fluorescent color output image (160) includes output pixels (232), The intensity (I) of the first color (216) in the output pixel (232) depends on the intensity (I) of the fluorescence emission signal (204) in at least one corresponding input pixel (230). The intensity of the second color (218) in the output pixel depends on the intensity of the autofluorescence signal (224) in at least one corresponding input pixel. A data processing device (170) according to any one of claims 1 to 3.

5. The aforementioned data processing device is The digital fluorescent color output image (160) is generated in a color space (410) having a set of color space coordinates (402, 404, 406) ({R, G, B}). It is configured in such a way, The first color (216) is represented by a first subset of the color space coordinate set of the color space, The second color (218) is represented by a second subset of the color space coordinate set of the color space, and the intersection of the first subset and the second subset is empty. The data processing device is configured to form a combination of the fluorescence emission signal (204) colored with the first color (216) and the autofluorescence signal (224) colored with the second color (218) by forming a union of the first subset and the second subset. A data processing device (170) according to any one of claims 1 to 4.

6. The data processing device is configured to perform vector addition (140) at each output pixel of the digital fluorescent color output image between the color space coordinates of the fluorescent emission signal colored with the first color and the color space coordinates of the autofluorescence signal colored with the second color, thereby forming a composite of the fluorescent emission signal (204) colored with the first color (216) and the autofluorescence signal (224) colored with the second color (218). A data processing device (170) according to any one of claims 1 to 4.

7. The aforementioned data processing device is By simultaneously applying a linear transformation (144) to the fluorescence emission signal and the autofluorescence signal, a composite of the fluorescence emission signal (204) colored with the first color (216) and the autofluorescence signal (224) colored with the second color (218) is formed. It is structured in such a way. A data processing device (170) according to any one of claims 1 to 4.

8. The linear transformation includes multiplying the intermediate image by a transformation matrix having a first dimension corresponding to the total number of color space coordinates in the fluorescence emission signal and the autofluorescence signal, and a second dimension corresponding to the total number of color space coordinates in the color space of the digital fluorescence color output image. The data processing device (170) according to claim 7.

9. The aforementioned data processing device is The ratio of the intensity of the fluorescence emission signal to the intensity of the autofluorescence signal in the digital fluorescence color output image (160) is changed according to the user selection signal (164). It is structured in such a way. A data processing device (170) according to any one of claims 1 to 8.

10. The input image data (120) is of the object, At least one digital color input image (212), At least one digital color input image and at least one digital monochrome input image (214), At least two digital input images (214) that are monochrome images. Including one of the following, A data processing device (170) according to any one of claims 1 to 9.

11. A medical observation device (100), such as a microscope or endoscope, for observing an object (106) containing at least two fluorophores (116, 118, 120), The aforementioned medical observation device is A data processing device (170) according to any one of claims 1 to 9, At least one digital camera (110, 111, 111a) is adapted to record the input image data (120), and A medical observation device (100) comprising the above.

12. A computer implementation method for processing input image data (122) in a medical observation device (100) such as a microscope or endoscope, The aforementioned method, The autofluorescence signal (224) of the object (106) is an autofluorescence signal that represents fluorescence emitted by a fluorophore (116) naturally contained in the object, and The fluorescence emission signal (204) of the object, which represents fluorescence emitted by a fluorophore (118) artificially added to the object. The steps include accessing input image data (120) that includes, The steps include generating a digital fluorescent color output image (160) from the synthesis of the fluorescent emission signal colored with a first color (216) and the autofluorescence signal colored with a second color (218) different from the first color, and Computer implementation methods, including those mentioned above.

13. A method for using a medical observation device, The aforementioned method, A step of recording input image data (120), The steps of performing the computer implementation method described in claim 11 and Methods that include...

14. A computer program product that, when executed by a computer, includes instructions causing the computer to perform the method described in claim 12.

15. A computer-readable medium containing instructions that cause the computer to perform the method described in claim 12 when the program is executed by the computer.

Images