Data processing device and computer-implemented method for displaying blood oxygenation and concentration values in medical observation device, medical observation device and method of use thereof

Abstract

The data processing device (170) is configured to acquire at least one digital input image (130) representing a reflected light image of a biological object (106) and comprising a plurality of input pixels (230, 232). A blood concentration value is determined at an input pixel (230, 232) of the plurality of input pixels (230, 232), the blood concentration value representing a blood volume at a location of the subject (106) imaged in the input pixel (230, 232). Further, a blood oxygenation value is determined at the input pixel (230, 232), the blood oxygenation value representing an amount of deoxygenated hemoglobin and / or an amount of oxygenated hemoglobin at the location of the subject (106). Output pixels (230, 234) of the digital output color image (160) are generated by assigning a color (222) to each output pixel (230, 234), the color (222) depending on the blood oxygenation value and the blood concentration value.

Description

Technical Field

[0001] This invention relates to a medical observation device, such as a microscope or endoscope. Background Technology

[0002] Especially in surgical procedures assisted by microscopes and / or endoscopes, it is important for surgeons to identify the areas associated with arteries or veins as accurately as possible.

[0003] Therefore, the object of the present invention is to provide a data processing device and a computer-implemented method to improve the accuracy of the representation of blood vessels and their corresponding types in images. Summary of the Invention

[0004] According to the present invention, this objective is achieved by a data processing apparatus for a medical observation device, such as a microscope or endoscope, used to image a biological object, wherein the data processing apparatus is configured to: acquire at least one digital input image, the at least one digital input image representing an image of the object formed by light reflected from the object, and the at least one digital input image comprising a plurality of input pixels; determine a blood concentration value at one of the plurality of input pixels, the blood concentration value representing the amount of blood at a location of the object imaged in the input pixel; determine a blood oxygenation value at the input pixel, the blood oxygenation value representing the amount of deoxyhemoglobin and / or oxyhemoglobin at the location of the object; and generate a digital output color image having a plurality of output pixels; wherein the output pixels are generated by assigning colors to the output pixels, the colors depending on the blood oxygenation value and the blood concentration value.

[0005] The above objectives are also achieved by a computer-implemented method for processing images from medical observation devices such as microscopes or endoscopes, the method comprising the steps of: acquiring at least one digital input image, the at least one digital input image representing an image of an object formed by light reflected from the object, and the at least one digital input image containing a plurality of input pixels; determining a blood concentration value at one of the plurality of input pixels, the blood concentration value representing the amount of blood at a location of the object, the location being imaged in the input pixel; determining a blood oxygenation value at the input pixel, the blood oxygenation value representing the amount of deoxyhemoglobin and / or oxyhemoglobin at the location of the object; generating a digital output color image having a plurality of output pixels; and generating an output pixel among the plurality of output pixels by assigning a color to the output pixel, the color depending on the blood oxygenation value and the blood concentration value.

[0006] The above-described equipment and methods facilitate the identification of blood vessels because they are now visible not only by the blood oxygenation value (i.e., the amount of oxygen in the blood, or synonymously, the blood oxygen saturation) at the location of the object, but also by the blood concentration value. Therefore, areas can be identified by assigning colors to regions with high or very high (or low or very low) oxygenation values ​​but only low blood concentration values, thus avoiding misidentification of these areas as blood vessels.

[0007] The aforementioned apparatus and method can be further improved by any of the following features, which can be arbitrarily combined with each other, each feature having its own technical effect. Each of the following features can be used in conjunction with the data processing apparatus and the computer-implemented method, regardless of whether the particular feature has been described in the context of the data processing apparatus or the computer-implemented method. If, for example, a feature has been described in the context of the computer-implemented method, the data processing apparatus can be similarly configured to perform that feature. If a feature has been described in the context of the data processing apparatus, that feature can be performed as a step in the computer-implemented method.

[0008] Input image data may contain multiple input pixels. Each input pixel represents light received from a specific location on an object. Each input pixel corresponds to a specific location on the object. If the input image data includes more than one digital input image, a specific location on the object can be represented by more than one input pixel. Input pixels representing the same location on the object are commonly referred to in the art as "corresponding pixels".

[0009] Input pixels can be either colored or monochrome. Colored input pixels include color space coordinates that define the color of the input pixel using, for example, a color space. Color space coordinates represent color appearance parameters such as hue, lightness, brightness, chroma, colorfulness, and saturation. If a tristimulus color space (such as RGB) is used, each input pixel includes three color space coordinates: R, G, and B. Color space coordinate R defines the intensity of the red band, color space coordinate G defines the intensity of the green band, and color space coordinate B defines the intensity of the blue band. Other color spaces (such as HSV, CIELAB, or CIELUV) use different color space coordinates. Monochrome input pixels only indicate the intensity of light in the spectral band of the recorded light. Therefore, it has only one color space coordinate.

[0010] Throughout this document, if the digital input image is a color image, it is referred to as a "digital color input image." Multispectral or hyperspectral images are considered color images. If the digital input image is a monochrome image, it is referred to as a "digital monochrome input image." If, in a particular context, it is irrelevant whether the digital input image is color or monochrome, the general term "digital input image" is used. Therefore, "digital input image" can be either a "digital color input image" or a "digital monochrome input image." The same terminology definition is applied to digital output images.

[0011] Depending on the context, the color assigned to the output pixel can be either pseudo-color or false-color. In the case of pseudo-color, different hues can be assigned to the output pixel depending on the blood oxygenation and / or blood concentration values. In the case of false-color, the same hue but different intensities can be assigned to the output pixel depending on the blood oxygenation and / or blood concentration values. In both cases, if the blood concentration value is below a predetermined threshold, the color assigned to the output pixel can be black or have a very low intensity. Using the latter can help identify blood vessels because areas with blood concentration values ​​below the threshold are less visible, or even appear black.

[0012] When assigning color to output pixels, any color appearance parameter can be assigned to the output pixel, depending on the blood oxygenation value and blood concentration value. Color appearance parameters include any one of hue, lightness, brightness, chroma, saturation, and color intensity.

[0013] The output pixels and input pixels are preferably corresponding pixels. According to another aspect, the digital output color image and at least one digital input image have the same number of pixels and / or the same aspect ratio and / or are represented in the same color space.

[0014] According to another option, it is preferred to determine the blood concentration value and / or blood oxygenation value based solely on at least one digital input image.

[0015] A digital output image may contain multiple output pixels. The digital output image can be a monochrome image or a color image. Output pixels can be monochrome or color. For each output pixel, there is at least one corresponding input pixel in the input image data.

[0016] In one embodiment, at least one digital input image is a set of two or more monochrome images, a set including at least one digital color input image and at least one digital monochrome input image, or a set including at least one digital color input image.

[0017] Blood concentration values ​​can be considered an indicator of the presence of blood vessels, through which blood is transported and thus accumulates. Therefore, according to another aspect, the data processing device can be configured to assign hue to the output pixel based on the blood oxygenation value of the input pixel, and to assign intensity, brightness, and / or saturation to the output pixel based on the blood concentration value of the input pixel when assigning color to the output pixel.

[0018] For accurate determination of blood oxygenation and blood concentration values, it is advantageous to record at least one digital input image across as many color bands as possible. Since each color band can be represented as color space coordinates, this is equivalent to recording at least one digital input image in a color space with as many color space coordinates as possible. Therefore, in one embodiment, the at least one digital input image may be a digital multispectral input image. The term multispectral input image also includes a hyperspectral input image. Specifically, multispectral input images may be represented in color spaces having more than three or more than four spectral bands or color space coordinates. In a preferred embodiment, the multispectral input image comprises six color bands or six color space coordinates.

[0019] According to another advantageous embodiment, the digital multispectral input image can be generated from multiple digital color input images, which are preferably recorded in different image spectra. Each of the multiple digital color input images can represent the reflectance image of an object. For example, each of the multiple digital color input images can have been recorded in a color space with three color space coordinates, such as RGB or HSV color space.

[0020] To facilitate the generation of a digital multispectral input image from multiple digital color input images, it is preferable to register the digital color input images relative to each other. In the registered image, pixels at corresponding positions in the image represent the corresponding positions of the imaged object.

[0021] When recording multiple digital input images using a corresponding number of digital cameras, the same field of use, the same optical axis, and the same focal length can be used to obtain the registered images.

[0022] Additionally or cumulatively, software can be used to register digital input images.

[0023] The multiple digital input images used to generate the multispectral input image are preferably all represented in the same color space (such as RGB or any other color space).

[0024] A digital multispectral input image can contain multiple input pixels, and each input pixel of a multispectral input image contains a set of color space coordinates. In one case, the set of color space coordinates of a multispectral image can be obtained by forming the union of the sets of color space coordinates of the corresponding color input pixels of multiple digital color images.

[0025] If the input image data comprises multiple digital input images, it is preferable that each of the multiple digital input images represents a different spectrum. Specifically, the spectra of different digital input images in the multiple digital input images do not overlap, or at least have only a minimal degree of overlap. Further, the spectra of different digital input images in the multiple digital input images may be complementary to each other. In a particular example, the non-overlapping image spectra of the multiple digital input images complement each other to form a continuous spectrum. Each imaged spectrum may include at least one passband and at least one stopband. Thus, within the spectral range, where one digital input image in the multiple digital input images includes a passband, the other digital input images in the multiple digital input images may all include stopbands.

[0026] At least one of the multiple imaging spectra may include a passband that includes, is defined within, or is contained within the fluorescence emission spectrum of at least one fluorophore in the biological object. The at least one fluorophore may be a naturally occurring fluorophore in the biological object. Additionally or cumulatively, at least one fluorophore may have been artificially added to the biological object. Examples of fluorophores that may have been artificially added to the biological object are ICG, fluorescein, or 5-ALA / PpIX. For example, the artificially added fluorophore may have been injected into a patient.

[0027] According to another approach, the imaging spectra of at least two digital color input images used to generate a digital multispectral input image may include near-NIR (near-infrared) wavelengths. NIR wavelengths can be used to add additional reflectance information that allows for a more accurate determination of blood concentration and oxygenation levels.

[0028] According to one embodiment, the data processing device can be configured to determine the deoxyhemoglobin value at an input pixel, wherein the deoxyhemoglobin value represents the deoxyhemoglobin concentration at the location of the object represented by the input pixel. The deoxyhemoglobin value can be obtained, for example, by spectral unmixing. In spectral unmixing, a reference reflectance spectrum of deoxyhemoglobin can be used as an endmember.

[0029] According to another embodiment, the data processing device can be configured to determine the oxyhemoglobin value at an input pixel, wherein the oxyhemoglobin value represents the oxyhemoglobin concentration at the location of the object represented by the input pixel. The oxyhemoglobin value can be determined, for example, by a spectral demixing technique using the reflectance spectrum of oxyhemoglobin as an endmember.

[0030] If spectral unmixing is used to determine the values ​​of deoxyhemoglobin or oxyhemoglobin, the values ​​of the signal descriptors obtained from the spectral unmixing can be used to determine the blood concentration values. The greater the contribution of the endmember to the spectrum at the input pixel, the higher the blood concentration value can be.

[0031] The oxyhemoglobin and / or deoxyhemoglobin values ​​at the location of the object corresponding to the input pixel can be obtained in different modalities without using the input pixels of the digital input image to determine the oxyhemoglobin and deoxyhemoglobin values.

[0032] According to another embodiment, the data processing device can be configured to determine the blood oxygenation value at an input pixel by calculating the difference between the oxyhemoglobin value and the deoxyhemoglobin value at the input pixel. This difference can be a ratio, a difference, or a combination of both between the oxyhemoglobin and deoxyhemoglobin values. The blood oxygenation value thus represents the oxygen saturation at the location of the object being mapped or imaged at the input pixel.

[0033] Alternatively, the data processing device can be configured to calculate the blood concentration value at an input pixel by calculating the aggregate value of the oxyhemoglobin and deoxyhemoglobin values ​​at that pixel. The aggregate value can be the sum, product, or a combination of the oxyhemoglobin and deoxyhemoglobin values. It therefore represents the amount of hemoglobin at that location. The amount of hemoglobin at that location, in turn, represents the amount of blood at that location.

[0034] In some cases, it may be necessary to locate blood vessels identified by blood oxygenation and blood concentration values ​​within an anatomical setting. Therefore, according to one aspect, a digital output color image may be combined with at least one digital input image, which represents the reflection of an object. For example, the digital output image may be superimposed on one of the digital input images. The digital input image combined with the digital color output image may itself have already been generated by combining multiple digital input images, each representing light reflected (preferably in the spectrum) by the object. For example, before combining the multispectral input image generated from multiple input images with the digital color output image, the multispectral input image may be converted into a color input image with three color space coordinates, such as an RGB color image.

[0035] When a digital color output image is combined with a digital input image representing a reflection, the data processing device can be configured to generate output pixels by mixing the colors assigned to the output pixels with the colors of the corresponding input pixels.

[0036] Blending can include alpha blending, such as assigning transparency to the color of an output pixel (where transparency may depend on the intensity of the color), and overlaying the color of an output pixel onto the color of an input pixel.

[0037] Mixing can include vector addition between the color space coordinates of the color assigned to the output pixel and the color space coordinates of the input pixel.

[0038] According to another approach, blending may include a linear transformation that maps the color space coordinates of the pseudo-color and the color space coordinates of the input pixels to a set of color space coordinates (e.g., RGB color space coordinates) of the output pixels. The linear transformation may include a color transformation matrix multiplied by the color space coordinates of the color assigned to the output pixel and the color space coordinates of the corresponding input pixel.

[0039] The present invention also relates to a medical observation device for imaging biological objects, such as an endoscope or microscope, wherein the medical observation device includes a data processing device according to any embodiment and / or a data processing device including any of the features described above, and at least one color camera for recording digital input images.

[0040] In another embodiment, the medical observation device may include two color cameras configured to record digital color input images in two different imaging spectra, which are preferably non-overlapping and / or complementary.

[0041] The medical observation device may further include a third camera for recording a third digital input image in an imaging spectrum that is complementary to the imaging spectra of the two digital cameras.

[0042] At least one camera can be configured to record the fluorescence emission of fluorophores contained in a biological object. The imaged spectrum recorded by the camera preferably includes a passband that includes or is defined by the fluorescence emission spectrum of at least one fluorophore.

[0043] At least one camera in a medical observation device can be configured to record light in the NIR range. Such a camera can also be configured to record the fluorescence emission of fluorophores.

[0044] Medical observation equipment may include an illumination system configured to illuminate the object using a standard light source. Further, the illumination system may be configured to illuminate the object in the NIR range.

[0045] In one example, the lighting system is configured to illuminate an object using a bandpass filter that is configured to allow light with wavelengths greater than about 600 nm, particularly light with wavelengths greater than about 750 nm, to pass through.

[0046] A camera in a medical observation device may be a digital fluorescence color camera configured to record a first color image representing the reflectance of an object in a first imaging spectrum, the first imaging spectrum including wavelengths included in the fluorescence spectrum of at least one fluorophore in the biological object. The first imaging spectrum may include the NIR range. The digital fluorescence color camera may be a camera typically used to record fluorescence images of fluorophores.

[0047] Medical observation equipment may include a digital white light color camera, which is adapted to record a second color image in a second imaging spectrum that is complementary to the first spectrum.

[0048] The medical observation device may include a first set of optical observation filters configured to transmit a first imaged spectrum and a second set of optical observation filters configured to transmit a second imaged spectrum. The passband in the second imaged spectrum preferably corresponds to the stopband in the first imaged spectrum. The stopband in the second imaged spectrum therefore includes the fluorescence excitation spectrum of at least one fluorophore.

[0049] The present invention also relates to a method for operating a medical observation device, such as a microscope or endoscope, for observing biological objects, wherein the method includes the steps of: recording at least one digital input image, preferably representing light reflected by the object; and performing a computer-implemented method of any of the embodiments described above.

[0050] Finally, the present invention relates to a computer program product containing instructions and / or a computer-readable medium containing instructions that, when executed by a computer (such as the data processing apparatus described above), cause the computer to perform a computer implementation method of any of the embodiments described above.

[0051] As used herein, the term “and / or” includes any and all combinations of one or more related listed items and may be abbreviated as “ / ”.

[0052] Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a box or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding box, item, or feature of the corresponding apparatus.

[0053] In the following description, the invention is illustrated by way of example with reference to various examples and the accompanying drawings. The combinations of features described and / or shown in the drawings and / or examples should not be considered limiting. For example, if a feature has a technical effect that is not required in a particular application (e.g., as explained above), that feature may be omitted from that embodiment. Conversely, if the technical effect associated with the above-described feature, which is not part of the embodiments described below, is beneficial in a particular application, that particular feature may be added. Attached Figure Description

[0054] Throughout the specification and drawings, the same reference numerals are used for elements that correspond to each other in terms of function and / or structure.

[0055] In the attached diagram,

[0056] Figure 1 A schematic representation of a medical observation device for generating a digital color output image from at least one digital color input image is shown;

[0057] Figure 2 A schematic representation of generating digital color output is shown;

[0058] Figure 3 This illustrates another schematic representation of generating a digital color output image;

[0059] Figure 4 A schematic representation of generating a digital multispectral input image from two digital color input images is shown;

[0060] Figure 5 A schematic representation of a method for generating digital color output images is shown;

[0061] Figure 6 A schematic representation of the microscope system is shown. Detailed Implementation

[0062] Figure 1 A medical observation device 100 is schematically shown. This medical observation device 100 can be a microscope or an endoscope. The main difference between a microscope and an endoscope is that, in an endoscope (not shown), the object 106 is observed via an optical fiber brought to the vicinity of the object 106 to be studied (e.g., by insertion into the body containing the object), while in a microscope, an objective lens 174 is guided onto the object. Although Figure 1 The medical observation equipment mentioned is a microscope, but the following description also applies to endoscopes.

[0063] The medical observation device 100 can be a medical observation device used during surgery. The medical observation device 100 can also be a medical observation device used in a laboratory (such as a laboratory microscope). The object to be studied 106 can be composed of or contain biological tissue 107. The object 106 can be part of a patient's body located within the field of view of the medical observation device 100.

[0064] Object 106 may contain one or more fluorophores 116, 118, and 120. At least one fluorophore 116 may be a fluorophore naturally present in the object. For example, bones and blood naturally contain fluorophores. At least one fluorophore 118 or 120 may be artificially added to object 106 (e.g., by injecting the fluorophore into biological tissue 107). Examples of fluorophores 118 or 120 that may be artificially added to object 106 are ICG, luciferin, and / or 5-ALA.

[0065] The illustrated medical observation device 100 can be a fluorescence imaging device. Therefore, the medical observation device is configured to observe and preferably also excite the fluorescence of one or more fluorophores 116, 118, 120.

[0066] Medical observation equipment 100 can be as follows Figure 1 The stereoscopic device is illustrated in the example. Therefore, it may include two identical sub-components 101L and 101R for each of the two stereoscopic channels. Since the two sub-components 101L and 101R are identical in function and structure, the following description focuses on the right sub-component 101R, but also applies to the left stereoscopic channel 101L.

[0067] The medical observation device 100 may alternatively be a monoscopic device. In this case, only one of the two sub-components 101L and 101R may exist. Therefore, the following description also applies to the monoscopic medical observation device 100.

[0068] The operating medical observation device 100 provides input image data 122. The input image data 122 represents the imaged scene, that is, the portion of the object within the field of view 184 of the medical observation device 100.

[0069] The input image data 122 may include one or more different digital input images 130. Specifically, the digital input images may be different digital color input images 130, different digital monochrome input images 130, or a combination of at least one digital color input image 130 and at least one digital monochrome input image 130. If the input image data 122 contains multiple digital input images 130, the different digital input images 130 should contain different spectral information. In this case, each digital input image 130 of the input image data can be recorded at a different wavelength, preferably with no spectral overlap or at least minimal spectral overlap. Preferably, the imaged spectra are non-overlapping, and the different digital input images 130 of the input image data 122 are recorded in the imaged spectrum. According to another aspect, the imaged spectra are complementary. In this case, the stopband and passband of the imaged spectrum complement each other to form a continuous input spectrum that is seamlessly or at least almost seamlessly segmented into multiple imaged spectra.

[0070] To generate input image data 122, the digital imaging system 102 may include one or more digital cameras 108, preferably digital color cameras. The number of digital input images 130 contained in the input image data 122 may depend on (in particular equal to) the number of cameras 108 used to generate the input image data 122. Depending on the type and settings of the digital cameras 108, the digital input images 130 may be color images or monochrome images.

[0071] Medical observation device 100 can be configured to record both the fluorescence of a naturally occurring fluorophore 116 and the fluorescence of at least one artificially added fluorophore 118, 120 in input image data 122. For example, one or more fluorophores 118, 120 may have been injected into a patient to mark a specific region of interest, such as a tumor. To record the fluorescent fluorophore, at least one digital camera 108 (i.e., digital fluorescence camera 111) has an imaging spectrum that includes or contains the (known) fluorescence emission spectrum of the fluorophore whose fluorescence is to be recorded.

[0072] The medical observation device 100 can also be configured to record light reflected from objects in the input image data 122. This can be done simultaneously or sequentially with recording the fluorescence emission of fluorophores.

[0073] Instead of using a digital camera to record the reflection of the object, another digital camera can be used to record at least one fluorescent fluorophore 116, 118. This digital camera 108, namely digital reflection camera 110, can be used to provide a preferred white light reflection image of the object 106, particularly in wavelengths excluding those recorded by the other digital camera 108 (i.e., the fluorescence emission spectrum or a portion thereof).

[0074] However, according to one embodiment, it is preferred that the medical observation device 100 is configured to use both a digital camera for recording the fluorescence emission of at least one fluorophore and a digital camera for recording the reflection image to record a reflection image of an object. A reflection image recorded by two or more cameras 108 has additional spectral information compared to a reflection image recorded by only one camera 108. Specifically, the medical observation device 100 can be configured to generate a digital multispectral reflection input image from a digital input image 130 generated by two or more digital cameras 108.

[0075] Therefore, in one embodiment, the digital imaging system 102 may include a digital camera 108, a digital reflection camera 110, and one or more digital fluorescence cameras 111, 111a. A second (or third) digital fluorescence camera 111a is optional. Each fluorescence camera should record light in a different and preferably non-overlapping imaged spectrum, which is preferably complementary to all other imaged spectra.

[0076] exist Figure 1 In this diagram, the second digital fluorescence camera 111a is shown only in the left stereo channel 101L, but it could also exist in the right stereo channel 101R. Alternatively, a stereo channel digital fluorescence camera can be used as the (first) digital fluorescence color camera 111, and a other stereo channel digital fluorescence camera can be used as the second fluorescence camera 111a. Cameras 110, 111, and 111a can each be a color camera or a monochrome camera. Multispectral or hyperspectral cameras are considered as color cameras.

[0077] Digital reflectance camera 110 is configured to record a digital reflectance input image 114 (i.e., digital input image 130), which represents the reflection of object 106 and may therefore include all or at least the main portion of the reflected signal. Digital reflectance camera 110 is preferably configured to record digital input image 130 over a broad spectral range within the visible light spectrum. Therefore, digital input image 130 recorded by digital reflectance camera can closely represent the natural color of object 106. It is important that if digital reflectance camera 110 is used to provide a user with an image of the object, that image should be as close as possible to human perception of the object. Digital reflectance camera 110 may be a CCD, CMOS, or multispectral or hyperspectral camera.

[0078] Each of at least one digital fluorescence camera 111, 111a is configured to record a different digital fluorescence image 112 (i.e., digital input image 130), the digital fluorescence image 112 being recorded in one or more fluorescence spectra of at least one fluorophore 116, 118, 120. Each fluorescence camera 111, 111a may be configured to record the fluorescence of a different fluorophore.

[0079] The fluorescence camera 111 can be configured to record digital fluorescence images only in one or more narrow bands of light. These narrow bands should overlap with one or more fluorescence spectra of one or more fluorophores 116, 118, 120 to be recorded. Preferably, the fluorescence spectra of the different fluorophores 116, 118, 120 are at least partially separated, preferably completely separated (i.e., non-overlapping), so that the fluorescence camera 111 can record a digital color input image 130 representing two separate fluorescence bands spaced apart from each other.

[0080] Alternatively, if two fluorescence cameras 111 and 111a are provided, each fluorescence camera 111 and 111a preferably captures the fluorescence emission of different fluorophores.

[0081] At least one fluorescent camera 111, 111a can be a monochrome camera, CCD, CMOS, or multispectral or hyperspectral camera. Preferably, the white light color camera 110 and at least one fluorescent color camera 111 are of the same type, although this is not required.

[0082] As stated above, the digital reflection camera 110 and at least one fluorescence camera 111, 111a can be used to record the reflected input image of the object 106.

[0083] Any combination of cameras 110, 111, and 111a can be combined into a single multispectral camera or a hyperspectral camera, either virtually processing individual images recorded by cameras 110, 111, and 111a into a single multispectral image, or serving as a single real multispectral camera performing the functions of different cameras 110, 111, and 111a.

[0084] The respective fields of view 184 of cameras 110, 111, and (if present) 111a are preferably aligned, or even coincident and coaxial. Preferably, cameras 110 and 111 provide the same field of view 184 with the same angle of view and focal length. This produces the same presentation of object 106 in images 112 and 114 generated by different cameras 110 and 111. Both cameras 110 and 111 can use the same objective lens 174.

[0085] If the matching of viewpoint and field of view cannot be generated optically, the matching of viewpoint and field of view can be generated by applying a matching or registration routine to the digital input image 130 through image processing, as explained further below.

[0086] Preferably, cameras 110, 111, and (if present) 111a operate synchronously. Specifically, the exposure times can be synchronized. Thus, the medical observation device 100 can be configured to simultaneously generate digital input images 130.

[0087] Preferably, the gains of at least two cameras 110, 111, and 111a are synchronized, meaning the gains are adjusted simultaneously in at least two cameras 110, 111, and 111a. Furthermore, even if the gain is changed, the ratio of the gain applied in camera 110 to the gain applied in camera 111 (and, if present, the gain applied in camera 111a) can be constant. Gamma correction and color adjustment or white balance can be turned off or kept constant.

[0088] To separate the light recorded in the digital reflectance input image 114 from the spectrum recorded in at least one digital fluorescence input image 112, i.e., to separate the reflectance spectrum from the fluorescence spectrum, an optical dichroism assembly 176 may be provided. The dichroism assembly 176 may include optical elements, such as a dichroic beam splitter 192. The dichroism assembly 176 may further include, or optionally include, an optical observation filter set 188 and / or an optical fluorescence filter set 190.

[0089] The fluorescent filter set 190 is preferably configured to transmit light from one or more fluorescence spectra of one or more fluorophores 116, 118, 120 and block light from one or more fluorescence spectra outside the fluorescence spectrum.

[0090] The fluorescence filter set 190 may include one or more optical bandpass filters with one or more passbands. Each passband should overlap with the fluorescence emission spectrum of the corresponding fluorophore 116, 118, 120 whose fluorescence is to be recorded. Since the fluorescence filter set 190 is in the optical path between the beam splitter 192 and the fluorescence color camera 111, only wavelengths within the passbands of the fluorescence filter set 190 are transmitted to the fluorescence color camera 111.

[0091] If two fluorescence cameras 111 and 111a are used to capture different fluorescence emission spectra, the fluorescence filter set 190 may include different optical bandpass filters in front of each of the fluorescence color cameras 111 and 111a. The passband of one bandpass filter may be contained in the fluorescence emission spectrum of one fluorophore 116, while the passband of the other bandpass filter may be contained in the fluorescence emission spectrum of another fluorophore 116 or 118 in object 106.

[0092] The observation filter set 188 is preferably configured to block light in one or more fluorescence spectra of one or more fluorophores 116, 118. The observation filter set 188 may also be configured to block light in fluorescence excitation spectra.

[0093] The observation filter set 188 is preferably configured as a bandstop filter, the stopband of which corresponds to or at least includes the passband of the fluorescence filter set 190. The observation filter set 188 is located in the optical path between the beam splitter 192 and the white light camera 110. Therefore, the white light camera 110 only records wavelengths outside the stopband of the observation filter set 188, while hiding wavelengths outside the passband of the fluorescence filter set 190, which also records wavelengths outside the passband of the fluorescence filter set 190.

[0094] Either the observation filter set 188 or the fluorescence filter set 190 can be an adjustable filter.

[0095] If beam splitter 192 is a dichroic beam splitter, at least one of the filter sets 188 and 190 can be omitted, since the optical spectral filtering in this case is already integrated into the dichroic beam splitter. The above description of the passband and stopband should then be applied, with necessary modifications, to the dichroic beam splitter 192.

[0096] The medical observation device 100 may also include an illumination component 178, which is configured to preferably illuminate the object 106 through an objective lens 174 and record at least one digital image 112, 114 through an imaging system 102 of the objective lens 174.

[0097] The illumination assembly 178 can be configured to selectively generate white light (i.e., light uniformly distributed across the overall visible spectrum) and fluorescence excitation light, the fluorescence excitation light comprising light only at wavelengths that stimulate the fluorescence of at least one fluorophore 116, 118. The illumination light generated by the illumination assembly 178 can be fed into the objective lens 174 using an illumination beam splitter 180.

[0098] The illumination component 178 can be configured to generate illumination light simultaneously in multiple discrete (particularly narrowband) wavelength bands. These wavelength bands may include any one or any combination of the following wavelength bands.

[0099] One such discrete wavelength band may lie entirely within the fluorescence excitation spectrum of fluorophore 116. Another such wavelength band may lie entirely within the fluorescence emission spectrum of another fluorophore 118. Yet another such wavelength band may be confined to wavelengths greater than 700 nm and lie entirely within the NIR range.

[0100] Simultaneous illumination of an object using any of the discrete wavelength bands described above can be accomplished by a light source 199 (e.g., an adjustable light source, such as a light source comprising multiple LEDs of different colors (especially different primary colors)), which is configured to generate light simultaneously in these wavelength bands. Alternatively or additionally, wavelength bands can be generated by using an illumination filter 179 having multiple passbands, wherein the passbands preferably correspond to the wavelength bands described above. If such an illumination filter 179 is used, the light source 199 can generate white light, which is then filtered by the illumination filter 179 so that only light in the passbands illuminates the object 106.

[0101] The illumination filter 179 may be provided depending on at least one fluorophore whose fluorescence is to be triggered and its specific excitation spectrum. For example, if 5-ALA is used as the fluorophore, the illumination filter may have a transmittance of 90% to 98% at wavelengths up to 425 nm, a transmittance of 0.5% to 0.7% at wavelengths between 450 nm and 460 nm, a transmittance of no more than 0.1% between 460 nm and 535 nm, and virtually zero transmittance at wavelengths above 535 nm. The illumination filter 179 may be configured for the passage of NIR light. For example, the illumination filter 179 may include a passband in NIR. The illumination filter 178 may further include a passband preferably located entirely within the fluorescence excitation spectrum of another fluorophore.

[0102] By reconfiguring the dichroic assembly 176 (e.g., by replacing the optical elements of the dichroic assembly 176, such as filter set 190 and / or 192 or dichroic beam splitter 180), the medical observation device 100 can be adapted to different fluorophores or sets of fluorophores.

[0103] The input image data 122 is processed by data processing device 170. Data processing device 170 may be an integral part of medical observation device 100. In one example, the data processing device may be a processor embedded in the medical observation device, also serving as a controller for controlling the hardware of medical observation device 100 (such as the brightness and / or spectral emission of light source 199, and / or any objective lens of medical observation device 100, and / or any actuator of medical observation device 100). In another example, data processing device 170 is part of a general-purpose computer connected to the medical observation device via wired or wireless means for one-way or two-way data transfer.

[0104] Data processing device 170 may be a hardware module (such as a microprocessor) or a software module. Data processing device 170 may also be a combination of both hardware and software modules, for example, by using a software module configured to operate on a specific processor (such as a vector processor, floating-point graphics processor, parallel processor, and / or multiprocessor). Data processing device 170 may be part of a general-purpose computer 186 (such as a PC). In another embodiment, data processing device 170 is an embedded system or embedded processor of medical observation device 100.

[0105] Data processing device 170 is configured to acquire input image data 122 (e.g., in the form of one or more digital input images 130, such as digital white light color input image 114 and digital fluorescence image 112). Data processing device 170 may be configured to retrieve digital input images 130 from memory 194 and / or directly from cameras 110, 111 (and 111a, if present). Memory 194 may be part of data processing device 170 or located elsewhere in medical observation device 100.

[0106] The data processing device 170 is further configured to calculate a digital color output image 160 from the input image data 122.

[0107] The digital color output image 160 is a color image represented in a color space. The color space of the digital color output image may differ from the color space of any digital color input image contained in the input image data 122. However, preferably, the color space of the digital color output image 160 is the same as the color space of any digital color input image 130.

[0108] A color space includes at least three such color channels. In a color space, each color channel is represented by different color space coordinates. Color space transformations can be used to convert between different color spaces. In different color spaces, the same color is represented by different color space coordinates. Each pixel of the digital color input image 130 includes a set of color space coordinates that collectively represent the color of the corresponding pixel. Therefore, each color band can be considered to represent a color space axis, and each color can be considered to be a point in the color space defined by a vector pointing to that color (i.e., color space coordinates). Therefore, the addition of two colors corresponds to vector addition. If one color has color space coordinates {x1, y1, z1} and another color has color space coordinates {x2, y2, z2}, then the sum of these two colors corresponds to the color {x1 + x2, y1 + y2, z1 + z2}.

[0109] In one example, the digital color input image 130 (or generally, input image data 122) can be recorded in the RGB color space using three primary colors or color bands or color space coordinates R, G, B. Alternatively, the digital color input image 130 can be recorded in different color spaces, and / or represent multispectral or hyperspectral color input images. Digital input images 130 in a set of digital input images, such as digital white light color input image 114 and digital fluorescent color input image 112, do not need to be recorded in the same color space, although this is preferred.

[0110] In the RGB color space, each color is represented by a triplet of three color space coordinates in integer form, where each integer indicates the intensity of one of the primary colors R, G, and B. For example, the most intense red is indicated by the triplet [255, 0, 0]. The most intense green is indicated by [0, 255, 0], and the most intense blue by [0, 0, 255]. Therefore, the RGB color space is three-dimensional, and the CMYK color space would be four-dimensional. A color can be viewed as a point in the color space, with a vector (such as [0, 0, 255]) pointing to that point. A multispectral or hyperspectral color space with n color bands correspondingly produces an n-dimensional color space, where each color is represented by an n-tuple of color space coordinates.

[0111] Data processing device 170 may include routines 140 for determining the oxygenated and / or deoxygenated hemoglobin values ​​at pixels in at least one digital input image 130, as described by Hashimoto M et al. (1987) in “Color analysis method for estimating the oxygen saturation of hemoglobin using an image-input and processing system”, Analytical Biochemistry, 162(1), pp. 178-184. The accuracy of determining oxygenated and / or deoxygenated hemoglobin values ​​is improved if more than one digital (color) image, multispectral image, or hyperspectral image with different imaged spectra is used, resulting in more color bands available for spectral analysis.

[0112] The oxyhemoglobin value represents the concentration or amount of oxyhemoglobin at the location of the object, which is mapped onto or imaged within the input pixel. The deoxyhemoglobin value represents the concentration or amount of deoxyhemoglobin at the location of the object, which is mapped onto or imaged within the input pixel.

[0113] Furthermore, the data processing device 170 may include a routine 142 for determining the blood oxygenation value at an input pixel. The blood oxygenation value represents the degree of oxygen saturation in the blood. Therefore, it can represent the concentration or amount of deoxyhemoglobin and oxyhemoglobin at the location of an object mapped onto the input pixel. The routine 142 may be configured to obtain the blood oxygenation value from the oxyhemoglobin and deoxyhemoglobin values ​​at the input pixel. Specifically, the blood oxygenation value at a pixel may represent the difference between the deoxyhemoglobin and oxyhemoglobin values ​​at that input pixel (such as a ratio or difference between the oxyhemoglobin and deoxyhemoglobin values, or any combination of ratios and differences).

[0114] The data processing device 170 may include a routine 144 for determining a blood concentration value at a pixel, wherein the blood concentration value represents the absolute or relative amount of blood contained at a location of object 106 mapped onto the corresponding input pixel (preferably where the blood oxygenation value is obtained). The relative amount may be determined, for example, the density, mass, area, or volume of blood relative to the density, mass, area, or volume of non-blood substances. The absolute amount may be determined as the mass per unit volume or area.

[0115] Routine 144 can be configured to obtain the blood concentration value at the input pixel by calculating the sum of the oxyhemoglobin and deoxyhemoglobin values ​​at the input pixel. The sum can be a sum or a product.

[0116] The data processing device 170 may include a routine 148 for assigning colors, such as pseudo-color or false-color, to values ​​obtained through either routine 142 or 144. In the case of pseudo-color, different colors are assigned to each value. In the case of false-color, different intensities of the same hue are assigned to each value. Here, the term "color" is assumed to refer to any (combination) of color appearance values.

[0117] Routine 150 can be used to generate a digital color output image 160 from the blood concentration value and blood oxygenation value at each pixel. For example, the intensity at the output pixel can correspond to a combination of the blood oxygenation value and blood concentration value at the corresponding pixel.

[0118] Routine 150 may include a filtering step. For example, a blood concentration value may be used as a filter mask for filtering blood oxygenation value or the color of blood oxygenation value. According to one aspect, the blood oxygenation value intensity at a pixel is multiplied by the blood concentration value at that pixel.

[0119] To combine two digital input images into a single real or virtual multispectral input image, the data processing device may include a color image combination routine 152.

[0120] Finally, a routine 154 configured to combine two digital color images may be provided. Such a routine can be used to combine a digital output image 160 with at least one digital input image. The resulting digital image represents the location of blood vessels and their oxygenation levels, as well as background anatomical structures. The routine 154 may include alpha mixing, vector addition of color space coordinates, and / or linear transformations.

[0121] Any of routines 140 to 154 may be a software routine, a routine implemented in hardware, or a routine in which software and hardware components are combined. Any of routines 140 to 148 may be stored in the memory 194 of the data processing device 170 or the medical observation device 100.

[0122] The medical observation device 100 may include a user input device 162, which, when operated by a user, can generate a user selection signal 164 that can communicate with a digital processing device 170. The user input device 162 may be, for example, a physical button, dial, slider, or lever, or a widget representing a physical button, dial, slider, lever, or widget.

[0123] By operating the user input device 162, the user can determine which false colors or pseudo colors and at what intensity to assign to the blood oxygenation and / or blood concentration values, and / or determine which values ​​and images to combine in a single view.

[0124] The digital color output image 160 can be displayed on a monitor 132, which is integrated with the medical observation device 100. For example, the monitor 132 can be integrated into the ocular or eyepiece 104 of the medical observation device 100.

[0125] The digital color output image 160 is preferably generated in real time, that is, the digital color output image 160 is generated from the set of digital color input images 130 by at least two cameras 110, 111, 111a before the next set of digital color input images 130 is generated.

[0126] The medical observation device 100 may include a direct optical path 134 from the object 106 through the objective lens 174 to the eyepiece 104. In this case, the display may be a semi-transparent display 132 located in the direct optical path 134, or the display may be projected onto the direct optical path 134. A beam splitter 136 may be provided to split the light between the optical eyepiece 104 and the digital imaging system 102. In one embodiment, up to 80% of the light may be directed toward the eyepiece 104.

[0127] Alternatively, the medical observation device 100 may not have a direct optical path 134, but only display images from the integral display 132. As another alternative, the medical observation device may not have any display at all.

[0128] The medical observation device 100 may include an output interface 172, to which one or more (external) displays 182 may be connected. For this purpose, the output interface 172 may include standardized connectors and data transmission protocols (such as USB, HDMI, DVI, DisplayPort, Bluetooth, and / or others). The external display may be a monitor, 3D glasses, eyepieces, etc. Any combination of external displays may be connected to the output interface 172.

[0129] The computer 186 and / or data processing device 170 are connected to the digital imaging system 102 using one or more data transmission lines 196. The data transmission lines can be wired or wireless, or partially wired and partially wireless. The computer 186 and / or data processing device 170 do not need to be integrated into the medical observation device 100 and can be physically located away from the digital imaging system 102. For this purpose, the digital imaging system 102 and the computer 186 and / or data processing device 170 can be connected to a network (such as a LAN, WLAN, or WAN), and at least one monitor 182 is also connected to this network.

[0130] According to the modification, the medical observation device 100 can be stereoscopic, but includes only two cameras, one camera per stereo channel. In one stereo channel, a fluorescent color camera 111 is used and configured to selectively record white light reflection, while in the other stereo channel, a white light color camera 110 is used. If fluorescence is not used, this arrangement provides a stereoscopic white light color input image; if fluorescence is used, this arrangement provides a monocular white light color input image and a monocular fluorescent color input image. The above and following descriptions apply equivalently to this configuration.

[0131] The input image data 122 comprises multiple pixels. These pixels can be color pixels or monochrome pixels. Monochrome pixels represent only intensity (e.g., as in a grayscale image). Color pixels include information about at least some color appearance parameters, such as hue, lightness, brightness, chroma, saturation, and color intensity. Color pixels are recorded using color bands (or equivalently), color channels, or primary colors in a color space when using a digital color camera. Each color band is represented by different color space coordinates.

[0132] Figure 4This illustrates how a multispectral image can be generated from two digital input color images that can be represented in the RGB color space. For this purpose, the data processing device 170 can employ routine 152. As described above, the first digital color input image 130 can be a digital reflectance input image 114, and the second digital color input image 130 can be a digital fluorescence input image 112. Figure 4 As shown, both digital input images 112 and 114 represent objects 106 photographed under the same lighting conditions. Figure 1 The reflected image of the image is preferably recorded simultaneously.

[0133] The first input images 130 and 114 are recorded in a first imaging spectrum 420, which includes one or more passbands 430 and one or more stopbands 432. This is by way of example only. Figure 4 The first imaged spectrum 420 indicates that it may include two passbands 430 separated from each other by a stopband 432. The passbands 430 are preferably located within the visible light range 440, i.e., including or defined by wavelengths λ from about 380 nm to about 750 nm.

[0134] Quantitative examples of sensitivity curves 402, 404, and 406 for the RGB sensors are also shown. Sensitivity curve 402 represents the wavelength-dependent sensitivity of the blue (B) sensor, sensitivity curve 404 represents the wavelength-dependent sensitivity of the green (G) sensor, and sensitivity curve 406 represents the wavelength-dependent sensitivity of the red (R) sensor. The color space coordinates at each input pixel are generated by a set of such sensors. For a given wavelength range λ recorded at the input pixel... 0， Each sensor will record a different intensity I in its color band. Therefore, the wavelength range λ0 will be represented by a single set 460 of spatial coordinates {R1, G1, B1}.

[0135] The second digital input images 130, 112 are recorded in the second imaged spectrum 422. The second imaged spectrum 422 has at least one passband 430 and at least one stopband 432. The first imaged spectrum 420 and the second imaged spectrum 422 are preferably complementary to each other. The stopband 432 in one imaged spectrum 420, 422 will correspond to the passband in the other spectrum 422, 420.

[0136] The wavelength range λ0 recorded at the input pixels of the second digital images 130 and 112 will generate a set 460 of color space coordinates {R2, G2, B2}. Since the first imaged spectrum 420 and the second imaged spectrum 422 are different, even if the same range λ0 is recorded, the color space coordinates {R2, G2, B2} in the second digital input images 130 and 112 will be different from the color space coordinates {R1, G1, B1} in the first digital input images 130 and 114.

[0137] The second imaged spectrum 422 may include a passband 430 in the NIR (near-infrared) range 442. This passband may include or be limited to wavelengths from about 700 nm to about 2500 nm.

[0138] A multispectral image can be generated by forming a union 462 by merging two sets 460 of color space coordinates for each pair of corresponding input pixels in the first and second digital input images 112, 114. For example, the union 462 may include a set 460 of input pixels {R1, G1, B1} in the first digital input images 130, 114 and a set 460 of color space coordinates {R2, G2, B2} of corresponding input pixels in the second digital input images 130, 112. Therefore, the spectrum 450 of the multispectral image includes separate spectral bands 452 of the two digital input images 112, 114, thereby allowing for improved color resolution because the wavelength range λ0 is now resolved by six color space coordinates, instead of just three as in the case of using only a single digital input image. Each spectral band 452 corresponds to a passband 430 in either the first imaged spectrum 420 or the second imaged spectrum 422.

[0139] refer to Figure 2 This explains how to generate a digital output image 160, where output pixels 234 represent the blood oxygenation and blood concentration values ​​as described above.

[0140] First, one or more digital input images 130 are obtained, each representing an image of an object 106 formed by light reflected from the object. At least one digital input image 130 may be a simple RGB image, but is preferably a multispectral image (as shown in the reference image). Figure 4 The described multispectral image is generated from two or more color images. One or more digital input images can be retrieved directly from the camera or from temporary or permanent storage (such as computer memory or a disk drive).

[0141] At least one digital input image 130 is composed of a plurality of different color channels 204, which together form a digital reflective color input image 212. For example, the color channels 204 may correspond to Figure 4 The spectral band 452 in the image.

[0142] The digital reflective color input image 212 comprises a plurality of input pixels 232. At each input pixel 232, a routine 140 for determining the oxyhemoglobin value and / or the deoxyhemoglobin value is applied. This produces an intermediate image 218 in which the oxyhemoglobin value is represented at each pixel 230. Further, applying the routine 140 to the digital reflective color input image 212 produces an intermediate image 220 representing the deoxyhemoglobin value. Images 218 and 220 may be grayscale images, wherein the intensity at each pixel 230 corresponds to the concentration of oxyhemoglobin and the concentration of deoxyhemoglobin, respectively.

[0143] Using, for example, routine 142, the blood oxygenation value is determined from pixels 230 in image 218 and corresponding pixels 230 in image 220 by calculating the difference (e.g., the ratio of oxyhemoglobin value to deoxyhemoglobin value). This produces a blood oxygenation value for each pair of corresponding pixels 230 and images 218, 220. Different hues 222 can be assigned to different blood oxygenation values ​​using routine 148, preferably hues 222 having the same intensity. For example, a low blood oxygenation value can be assigned a blue hue, and a high blood oxygenation value can be assigned a red hue. The intensities of blue and red can be the same. This produces an intermediate color image 214, where the hue at each pixel 230 represents the blood oxygenation value.

[0144] Furthermore, blood concentration values ​​can be calculated from each pair of corresponding pixels 230 in images 218 and 220, thereby generating image 216, in which each pixel 230 represents a blood concentration value. The blood concentration value can be calculated by forming a total value (e.g., summing) for the oxyhemoglobin and deoxyhemoglobin values ​​of each pair of corresponding pixels in images 218 and 220.

[0145] The location at point 106 (where the total value of oxyhemoglobin and deoxyhemoglobin is high) indicates a higher blood concentration at that location. Simultaneously, the difference between the oxyhemoglobin and deoxyhemoglobin values ​​indicates the degree of hemoglobin oxygenation at that location.

[0146] Therefore, pixel 230 in image 216, which represents a high blood concentration value, most likely represents a blood vessel.

[0147] To generate a digital color output image 160 in which arteries and veins can be identified, image 216 is used in routine 150 to blend two images 214, 216. Routine 150 may, for example, use image 216 as a mask to cover image 214 (e.g., by using image 216 as an intensity mask for image 214): the blood oxygenation value or the color assigned to it is multiplied by a preferably normalized blood concentration value. If the result at output pixel 234 is below a predetermined threshold, the pixel may be set to black or any other color.

[0148] If routine 148 is not executed before image 214 is acquired, color 222 can be assigned after image 216 is used to mask image 214. In this case, routine 148 can be applied after routine 150 is applied.

[0149] The resulting digital output color image 160 represents the blood vessels and their oxygenation levels.

[0150] In another step, Figure 3 The digital output color image 160 shown can be combined with the digital input image 212 or any of its color channels 204. This is in Figure 4 As shown in the image.

[0151] For this purpose, routine 154 is applied to the digital color output image 160 and the digital input image 212. Images 160 and 212 are combined to produce a digital color output image 160a. The combination of images 160 and 212 may include blending by alpha mixing, for example by assigning transparency to a color based on the intensity of that color in the digital input image 212. Thus, the color at pixel 230 in image 160 is superimposed on the color of the corresponding input pixel 232 in image 212.

[0152] Alternatively, images 160 and 212 can be blended by vector addition of the color space coordinates of corresponding pixels in images 160 and 212 to 230 and 232, respectively.

[0153] As an alternative, a linear transformation can be applied simultaneously to the color space coordinates of pixel 230 in image 160 and the corresponding color space coordinates of pixel 232 in image 212 to produce the color space coordinates of output pixel 234 in image 160a. This linear transformation may include a color transformation matrix, with the color space coordinates of pixels 230 and 232 multiplied to produce the color space coordinates of image 160a and output pixel 234.

[0154] exist Figure 5 The diagram shows a schematic overview of a method for obtaining images representing blood oxygenation and saturation values.

[0155] First, in step 500, a digital input image 130 is obtained. The digital input image 130 may be a digital reflectance input image 114. In one example, the digital input image 130 may be a multispectral or hyperspectral image 212.

[0156] Optionally, in step 502 or in another optional step 504, an additional digital input image 130 (such as digital fluorescent input image 112) may be obtained.

[0157] In optional step 506, two or more digital input images 130 can be combined into a single multispectral input image 202. This can be done, for example, using routine 152. The digital input images 130 input to step 506 can be color images or monochrome images.

[0158] In step 508, if optional step 506 is performed, representative oxyhemoglobin and deoxyhemoglobin values ​​are calculated at various locations of the digital input image 202; if optional step 506 is not performed, representative oxyhemoglobin and deoxyhemoglobin values ​​are calculated at various locations of one or more digital input images 130. Preferably, oxyhemoglobin and deoxyhemoglobin values ​​are determined at each pixel of at least one digital input image 130, 212 input in step 508. For example, routine 140 may be performed in step 508. As a result of step 508, a set of oxyhemoglobin values ​​and a set of deoxyhemoglobin values ​​are obtained for each pixel or each set of corresponding input pixels. This set of values ​​can be considered as constituting an image 218 representing oxyhemoglobin values ​​and an image 220 representing deoxyhemoglobin values.

[0159] In step 510, for example, by using procedure 142, the blood oxygenation value is determined from the oxyhemoglobin and deoxyhemoglobin values ​​for each input pixel. The resulting set of blood oxygenation values ​​can be considered as constituting image 214.

[0160] In step 512, blood concentration values ​​are calculated using images 218 and 220. This can be accomplished using routine 144. As a result of step 512, a set of blood concentration values ​​is obtained, which can again be understood as image 216.

[0161] In step 514, a false color or pseudo-color can be assigned to the blood oxygenation value. Therefore, the color of a pixel in image 214 depends on its blood oxygenation value. Step 514 can utilize routine 148.

[0162] The selection of pseudo-color and / or false color and / or its intensity level may depend on the user-selected signal 164.

[0163] As a result of step 514, a colorized image 214 representing the blood oxygenation value is obtained.

[0164] In step 516, the image 214 representing blood oxygenation value and the image 216 representing blood concentration value are combined. This can be done using routine 150. As a result, a digital output image 160 is obtained. If step 514 was not performed previously, it can now be performed to assign a color value to each value in the digital output image 160.

[0165] In optional step 518, the digital output image 160 can be combined with the reflected images 130, 212. This can be done using routine 154.

[0166] In step 522, digital output image 160 may be displayed using, for example, display 182, or digital output image 160a may be displayed if step 518 has been performed.

[0167] Some implementations involve a microscope that includes... Figures 1 to 5 One or more related systems described in the text. Alternatively, the microscope can be... Figures 1 to 5 It is part of or connected to one or more related descriptions of the system. Figure 6 A schematic diagram of a system 600 configured to perform the methods described herein is shown. System 600 includes a microscope 610 and a computer system 620. The microscope 610 is configured to capture images and is connected to the computer system 620. The computer system 620 is configured to perform at least a portion of the methods described herein. The computer system 620 may be configured to execute machine learning algorithms. The computer system 620 and the microscope 610 may be separate entities, or they may be integrated together in a common housing. The computer system 620 may be part of the central processing system of the microscope 610 and / or the computer system 620 may be part of a sub-component of the microscope 610, such as a sensor, actuator, camera, or illumination unit of the microscope 610.

[0168] Computer system 620 may be a local computer device (e.g., a personal computer, laptop, tablet, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed in various locations, such as local clients and / or one or more remote server farms and / or data centers). Computer system 620 may include any circuitry or combination of circuitry. In one embodiment, computer system 620 may include one or more processors, which may be of any type. As used herein, a processor may refer to any type of computing circuitry, such as, but not limited to, microprocessors, microcontrollers, complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, graphics processors, digital signal processors (DSPs), multi-core processors, field-programmable gate arrays (FPGAs), computing circuitry for microscopes or microscope components (e.g., cameras), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 620 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuitry) used in wireless devices such as mobile phones, tablets, laptops, two-way radios, and similar electronic systems. Computer system 620 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives that process removable media (such as optical discs (CDs), flash memory cards, digital video discs (DVDs), etc.). Computer system 620 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input and receive information from computer system 620.

[0169] Some or all of the method steps can be performed by (or using) hardware devices, such as, for example, processors, microprocessors, programmable computers, or electronic circuits. In some embodiments, such devices can perform one or more of the most important method steps.

[0170] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be performed using a non-transient storage medium, such as a digital storage medium like a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, storing electronically readable control signals that cooperate (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage medium can be computer-readable.

[0171] Some embodiments of the invention include a data carrier having electronically readable control signals, which is capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0172] Typically, embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product is run on a computer. The program code may, for example, be stored on a machine-readable medium.

[0173] Other implementations include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0174] In other words, therefore, an embodiment of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is run on a computer.

[0175] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon for performing one of the methods described herein when executed by a processor. Data carriers, digital storage media, or recording media are typically tangible and / or non-transitional. Another embodiment of the invention is an apparatus as described herein, including a processor and a storage medium.

[0176] Therefore, another embodiment of the invention represents a data stream or signal sequence for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0177] Other embodiments include processing devices, such as computers or programmable logic devices, configured or adapted to perform one of the methods described herein.

[0178] Another implementation includes a computer on which a computer program for performing one of the methods described herein is installed.

[0179] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transmitting the computer program to the receiver.

[0180] In some embodiments, programmable logic devices (e.g., field-programmable gate arrays) may be used to perform some or all of the functions of the methods described herein. In some embodiments, field-programmable gate arrays may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0181] Figure Labels

[0182] 100 medical observation devices

[0183] 101L 3D component for the left channel

[0184] 101R 3D component for the right channel

[0185] 102 Digital Imaging System

[0186] 104 eyepieces

[0187] 106 objects

[0188] 107 biological tissues

[0189] 108 digital camera

[0190] 110 digital reflective color camera

[0191] 111 Digital Fluorescent Color Camera

[0192] 111a Second Digital Fluorescent Color Camera

[0193] 112 digital fluorescence input images

[0194] 114 digital reflection input images

[0195] Naturally present fluorophores in object 116

[0196] 118 Fluoresistants artificially added to the object

[0197] 120 Another artificially added fluorophore to the object

[0198] 130 digital input images

[0199] 132 Built-in Display

[0200] 134 direct optical path

[0201] 136 beam splitter

[0202] 140 Routine procedures for determining oxyhemoglobin / deoxyhemoglobin levels

[0203] 142 Routine procedures for measuring blood oxygenation

[0204] 144 Routine procedures for measuring blood concentration values

[0205] 148 Routine for assigning colors

[0206] 150 Routine for generating digital color output images

[0207] 152 Multispectral Image Generation Routine

[0208] 154 Color Image Combination Routine

[0209] 160, 160a digital output images

[0210] 162 mode selector

[0211] 164 users select signal

[0212] 166 arrows

[0213] 170 Data Processing Equipment

[0214] 172 output interface

[0215] 174 objective lens

[0216] 176 color separation components

[0217] 178 Lighting Components

[0218] 179 Illumination Filter

[0219] 180° illumination beam splitter

[0220] 182 monitor

[0221] 184-degree field of view

[0222] 186 computer

[0223] 188 Observation Filter Set (First Observation Filter Set)

[0224] 190 Fluorescence Filter Set (Second Observation Filter Set)

[0225] 192 Dichroic Beam Splitter

[0226] 194 memory

[0227] 196 data transmission lines

[0228] 199 light source

[0229] 202 digital multispectral input images

[0230] 204 color channels of input color image

[0231] 212 digital reflective color input image

[0232] 214 represents the image of blood oxygenation level.

[0233] 216 represents the image of blood concentration values.

[0234] 218 represents the image of oxyhemoglobin levels.

[0235] 220 represents the image of deoxyhemoglobin levels.

[0236] 222 pseudo-color

[0237] 230 pixels

[0238] 232 input pixels

[0239] 234 output pixels

[0240] 400 digital color input images

[0241] Sensitivity curve for the first color space coordinates (e.g., B) of 402

[0242] Sensitivity curves for 404 second color space coordinates (e.g., G).

[0243] Sensitivity curve (e.g., R) in 406 third color space coordinates.

[0244] 420 First Imaging Spectrum

[0245] 422 Second Imaging Spectrum

[0246] 430 passband

[0247] 432 stopband

[0248] 440 visible light spectrum

[0249] 442NIR spectrum

[0250] Spectra of 450 multispectral images

[0251] Spectral bands of 452 multispectral images

[0252] A set of 460 color space coordinates

[0253] Union of 462 color space coordinates

[0254] Steps for obtaining a digital input image (500, 502, 504)

[0255] 506 Steps to obtain a multispectral input image from a digital input image

[0256] 508 Steps for calculating oxyhemoglobin and deoxyhemoglobin levels

[0257] 510 Steps for determining blood oxygenation level

[0258] 512 Steps for determining blood concentration values

[0259] 514 Steps for assigning colors to blood oxygenation and / or blood concentration values

[0260] 516 Steps for combining blood oxygenation images and blood concentration images

[0261] 518 is the step of combining the result of step 516 with the reflection image.

[0262] 522 Steps to display digital output image

[0263] 600 system

[0264] 610 microscope

[0265] 620 Computer

[0266] B1, B2, color space coordinates

[0267] G1, G2 color space coordinates

[0268] I Intensity

[0269] Imaging modes I, II, III...

[0270] R1, R2 color space coordinates

[0271] λ wavelength

[0272] wavelength band recorded by λ0

Claims

1. A data processing device (170) for a medical observation device (100), such as a microscope or endoscope, for imaging biological objects (106), in, The data processing device (170) is configured to: - Acquire at least one digital input image (130), the at least one digital input image (130) representing an image of the object (106) formed by light reflected from the object (106), and the at least one digital input image (130) comprising a plurality of input pixels (230, 232). - Determine the blood concentration value at input pixel (230, 232) among the plurality of input pixels, the blood concentration value representing the amount of blood at the location of the object (106); - Determine the blood oxygenation value at the input pixel (230, 232), the blood oxygenation value representing the amount of deoxyhemoglobin and / or oxyhemoglobin at the location of the object (106) imaged in the input pixel (230, 232); - Generate a digital output color image (160) with multiple output pixels (230, 234). The output pixel (230, 234) among the plurality of output pixels is generated as follows: - Assign a color (222) to the output pixel (230, 234), the color (222) depending on the blood oxygenation value and the blood concentration value.

2. The data processing device (170) according to claim 1. in, The data processing device (170) is configured to: - Based on the blood oxygenation value of the input pixel, assign a pseudo-color hue to the output pixel; - Based on the blood concentration value of the input pixel, assign pseudo-color intensity to the output pixel (230, 234).

3. The data processing device (170) according to claim 1 or 2. in, The at least one digital input image (130) is a digital multispectral input image (202).

4. The data processing device (170) according to claim 3. in, The data processing device (170) is configured to: - The digital multispectral input image (202) is generated from at least two digital color input images (130, 112, 114, 400), each of the at least two digital color input images (130, 112, 114, 400) representing the reflectance image of the object (106) in different imaging spectra (420, 422).

5. The data processing apparatus (170) according to any one of claims 1 to 4. in, The data processing device (170) is configured to: - Determine the deoxyhemoglobin value at the input pixel (230, 232), the deoxyhemoglobin value representing the deoxyhemoglobin concentration at the location of the object (106) represented by the input pixel (230, 232).

6. The data processing apparatus according to any one of claims 1 to 5, in, The data processing device (170) is configured to - Determine the oxyhemoglobin value at the input pixel (230, 232), the oxyhemoglobin value representing the oxyhemoglobin concentration at the location of the object (106) represented by the input pixel (230, 232).

7. The data processing apparatus (170) according to claims 5 and 6. in, The data processing device (170) is configured to - The blood oxygenation value at the input pixel is determined by calculating the difference between the oxyhemoglobin value at the input pixel (230, 232) and the deoxyhemoglobin value at the input pixel (230, 232).

8. The data processing apparatus (170) according to claims 5 and 6 or claim 7. in, The data processing device (170) is configured to - The blood concentration value at the input pixel (230, 232) is calculated by summing the oxyhemoglobin value at the input pixel (230, 232) and the deoxyhemoglobin value at the input pixel (230, 232).

9. The data processing apparatus (170) according to any one of claims 1 to 8. in, The data processing device (170) is configured to - The output pixels (230, 234) are generated by mixing the pseudo-color (222) assigned to the output pixels (230, 234) with the colors of the corresponding input pixels (230, 232).

10. A medical observation device (100), such as an endoscope or microscope, for imaging biological objects (106), in, The medical observation equipment includes: - The data processing apparatus (170) according to any one of claims 1 to 9; and - At least one color camera, said at least one color camera being used to record digital reflective color input images.

11. A computer-based method for processing images from a medical observation device (100) such as a microscope or endoscope. The computer implementation method includes the following steps: - Acquire at least one digital input image (130), the at least one digital input image (130) representing an image of the object (106) formed by light reflected from the object (106), and the at least one digital input image (130) containing a plurality of input pixels (230, 232). - Determine the blood concentration value at one of the plurality of input pixels (230, 232), the blood concentration value representing the amount of blood at the location of the object (106) which is imaged in the input pixels (230, 232); - Determine the blood oxygenation value at the input pixel (230, 232), the blood oxygenation value representing the amount of deoxyhemoglobin and / or oxyhemoglobin at the location of the object (106); - Generate a digital output color image (160) with multiple output pixels (230, 234). - Output pixels (230, 234) are generated by assigning a color (222) to the output pixels (230, 234), the color (222) depending on the blood oxygenation value and the blood concentration value.

12. A method for operating a medical observation device (100), said medical observation device (100), such as a microscope or endoscope, for observing biological objects. The method includes the following steps: - Record at least one digital input image (130); and - Perform the computer implementation method according to claim 11.

13. A computer program product comprising instructions which, when executed by a computer (186), cause the computer (186) to perform the method as claimed in claim 12.

14. A computer-readable medium containing instructions that, when executed by a computer (186), cause the computer (186) to perform the method as claimed in claim 12.

Images