Apparatus for an optical imaging system, optical imaging system, method and computer program

Abstract

Examples relate to an apparatus (130) for an optical imaging system comprising one or more processors (134) and one or more storage devices (136). The apparatus is configured to obtain sensor data indicative of a reflectance measurement of a sample (110). The apparatus is further configured to convert the sensor data into reflectance spectral data indicative of a reflectance spectral signal. The reflectance spectral signal is a function of the concentration distribution of the sample. Further, the apparatus is configured to obtain fluorescence spectral data indicative of fluorescence spectral signal of a fluorescence measurement of the sample. The apparatus is further configured to combine the reflectance spectral data and the fluorescence spectral data to combined spectral data. Further, the apparatus is configured to determine the concentration distribution of the sample based on the combined spectral data.

Description

[0001] Apparatus for an Optical Imaging System, Optical Imaging System, Method and Computer Program

[0002] Technical field

[0003] Examples relate to an apparatus for an optical imaging system, an optical imaging system, a method, and a computer program.

[0004] Background

[0005] Microscopes of an optical imaging system may be used for imaging in various imaging modes. For example, a microscope may comprise two optical imaging sensors, of which one is used for fluorescence imaging, and one is used for reflectance imaging. The two optical imaging sensors might be used for imaging in different wavelength bands. Such microscopes usually comprise a light source which is used to emit light that can be picked up by the respective sensors, e.g., as reflection (i.e., at the same wavelength as emitted) or as fluorescence (i.e., at a wavelength that is different from the wavelength of the emitted light).

[0006] There may be a desire for an improved concept for a microscope, in which the various imaging modes are used in a more versatile fashion.

[0007] Summary

[0008] This desire is addressed by the subject-matter of the independent claims.

[0009] The concept proposed in the present disclosure is based on the insight, that a determination of the concentration distribution of the sample can be improved by combining measurement data of a reflectance measurement and measurement data of a fluorescence measurement. Combining the reflectance measurement with the fluorescence measurement may allow to improve the determination of the concentration distribution due to different spectral signals obtained by the different measurement modes.

[0010] Examples provide an apparatus for an optical imaging system comprising one or more processors and one or more storage devices. The apparatus is configured to obtain sensor data indicative of a reflectance measurement of a sample. The apparatus is further configured to convert the sensor data into reflectance spectral data indicative of a reflectance spectral signal. The reflectance spectral signal is a function of the concentration distribution of the sample. Further, the apparatus is configured to obtain fluorescence spectral data indicative of fluorescence spectral signal of a fluorescence measurement of the sample. The apparatus is further configured to combine the reflectance spectral data and the fluorescence spectral data to combined spectral data. Further, the apparatus is configured to determine the concentration distribution of the sample based on the combined spectral data. The reflectance spectral data and the fluorescence spectral data can be used to analyze the properties of materials, identify specific substances, distinguish between different classes of objects, or extract valuable information from the sample. Converting the sensor data into reflectance spectral data may allow to combine the reflectance spectral data and the fluorescence spectral data to determine the concentration distribution of the sample. By converting the sensor data into the reflectance spectral data, data depending on a concentration distribution of the sample can be generated for the reflectance measurement. Thus, the reflectance measurement can be combined with the fluorescence measurement for evaluation. In this way, the concentration distribution of the sample can be determined based on both, the reflectance measurement and the fluorescence measurement. This may improve an accuracy of the determined concentration distribution.

[0011] In an example, the apparatus may be further configured to determine the reflectance spectral data using a logarithmic function. For example, the apparatus may determine the reflectance spectral data by applying a logarithmic function to the sensor data or a signal derived from the sensor data. Using a logarithmic function may allow to determine the reflectance spectral data in a facilitated way.

[0012] In an example, the apparatus may be further configured to determine the combined spectral data by combining the reflectance spectral data and the fluorescence spectral data in a single vector. Using a single vector may allow to represent spectral data indicative of the concentration distribution of the sample in a facilitated way. For example, using a single vector may reduce a computational complexity and / or may simplify algorithms needed for further processing, e.g., spectral unmixing. In an example, the apparatus may be further configured to determine the concentration distribution using spectral unmixing. For example, spectral unmixing may allow to extract subpixel information revealing a presence of materials that occupy a fraction of the pixel, estimation of different materials within a mixed pixel and / or identification and characterization of different materials present within a mixed pixel.

[0013] In an example, the apparatus may be configured to perform the spectral unmixing using linear transformation, independent component analysis, orthogonal subspace projection, minimum volume simplex analysis, convex cone analysis and / or fully constrained least squares.

[0014] Examples provide an optical imaging system comprising an apparatus as described above.

[0015] In an example, the optical imaging system may further comprise a first sensor for acquisition of the (first) sensor data and a second sensor for acquisition of second sensor data. The apparatus may be configured to control the first sensor and the second sensor such that the (first) sensor data and the second sensor data may be acquired at the same time. By image acquisition of the first sensor and the second sensor at the same time a reliability of the combined spectral data can be improved.

[0016] Examples provide a method for an optical imaging system, comprising obtaining sensor data indicative of a reflectance measurement of a sample and converting the sensor data into reflectance spectral data indicative of a reflectance spectral signal. The reflectance spectral signal is a function of a concentration distribution of the sample. Further, the method comprises obtaining fluorescence spectral data indicative of a fluorescence spectral signal of a fluorescence measurement of the sample and combining the reflectance spectral data and the fluorescence spectral data to combined spectral data. The method further comprises determining, based on the combined spectral data, the concentration distribution of the sample.

[0017] Various examples of the present disclosure relate to a corresponding computer program with a program code for performing the above method when the computer program is executed on a processor.

[0018] Short description of the Figures Some examples of apparatuses and / or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

[0019] Fig. 1 shows a schematic diagram of an example of an apparatus for an optical imaging system;

[0020] Fig. 2 shows a schematic flow diagram of an example of a method;

[0021] Fig. 3 shows a schematic flow diagram of another example of a method for an optical imaging system; and

[0022] Fig. 4 shows a schematic illustration of a system.

[0023] Detailed Description

[0024] Various examples will now be described more fully with reference to the accompanying drawings in which some examples are illustrated. In the figures, the thicknesses of lines, layers and / or regions may be exaggerated for clarity.

[0025] Fig. 1 shows a schematic diagram of an example of an apparatus 130 for an optical imaging system. The apparatus 130 is tasked with controlling various aspects of a microscope 120 of the optical imaging system 100 and of the entire optical imaging system 100 and / or with processing various types of sensor data, e.g., the sensor data and / or the fluorescence spectral data, of the optical imaging system 100. Consequently, the apparatus 130 may be implemented as a computer system, which interfaces with the various components of the optical imaging system, e.g., the sensor 122.

[0026] The apparatus 130 comprises, as shown in Fig. 1, one or more processors 134 and one or more storage devices 136. Optionally, the apparatus 130 further comprises one or more interfaces 132. The one or more processors 134 are coupled to the one or more storage devices 136 and to the optional one or more interfaces 132. In general, the functionality of the apparatus 130 may be provided by the one or more processors 134 (e.g., for determining the concentration distribution), in conjunction with the one or more interfaces 132 (for exchanging information, e.g., with the sensor 122, e.g., to receive the sensor data and the second sensor data) and / or with the one or more storage devices 136 (for storing and / or retrieving information).

[0027] The apparatus 130 is configured to obtain sensor data indicative of a reflectance measurement of a sample 110. For example, the sensor data may be received from the sensor 122. Alternatively, the sensor 122 may be part of the apparatus 130. Thus, the apparatus 130 may measure the sensor data by controlling an image acquisition of the sensor 122. Alternatively, the sensor data may be retrieved from a storage device, e.g., the storage device 136 or a frame buffer (which can be separate from the apparatus 130). The sensor data may be raw data. That is, the apparatus 130 may post process the sensor data to determine the reflectance measurement data. Alternatively, the sensor data may be already post processed. For example, the sensor 122 may have post processed the sensor data and no further post processing by the apparatus 130 may be necessary.

[0028] Reflectance is a measure of the amount of light reflected by the sample 110 at a wavelength. It can be expressed as a ratio of the reflected light intensity to the incident light intensity. The incident light may be radiated by a light emission module part of the optical imaging system. The light imaging module may be for providing illumination of the sample 110 for a plurality of wavelength bands. For example, the wavelength bands for the reflectance measurement may differ from the wavelength bands of the fluorescence measurement. The light imaging module may be configured to radiate wavelength bands for both, the reflectance measurement and fluorescence measurement.

[0029] The reflectance measurement may provide information needed to determine or construct a spectral signal, e.g., the reflectance spectral signal. The reflectance measurement may comprise a measuring of the reflectance at different wavelengths, which can be used to generate a spectral curve of a spectrum that represents the reflectance properties of the sample 110. However, the reflectance measurement may be not proportional to a concentration distribution of the sample 110. Therefore, combining the reflectance measurement straightforward with the fluorescence measurement is not possible. To allow a combination of the reflectance measurement with the fluorescence measurement, the apparatus 130 is configured to convert the reflectance measurement (i.e., the sensor data). The apparatus is configured to convert the sensor data into reflectance spectral data indicative of a reflectance spectral signal. The reflectance spectral signal derived from the reflectance measurement may allow to combine the reflectance measurement and the fluorescence measurement to determine the concentration distribution of the sample 110. For example, the reflectance spectral signal derived from the reflectance measurement may allow the characterization and / or analyzes of materials based on the unique spectral signatures. This may enable to perform materially identification, quality control and / or determination of concentration distribution, for example.

[0030] A spectral signal, such like the reflectance spectral signal or the fluorescence spectra signal, may be refer to the representation of electromagnetic radiation or light as a function of wavelength or frequency. The spectral signal may provide information about the intensity and / or magnitude of the electromagnetic radiation or light at different wavelengths or frequency bands.

[0031] The spectral signal may refer to the measured or recorded spectral information for a specific location or pixel in an image of the sample 110. The spectral signal may represent the reflectance (for the reflectance measurement) or emission characteristics (for the fluorescence measurement) of the sample 110 in that pixel. The spectral signal may be represented as a spectrum, which is a plot of intensity or radiance versus wavelength or frequency.

[0032] The spectral signal can be used to analyze the properties of materials, identify specific substances, distinguish between different classes of objects, or extract valuable information from the scene. Spectral unmixing algorithms leverage the spectral signals to estimate the abundance of different materials or components within each pixel, allowing for the characterization and analysis of complex scenes, for example.

[0033] The reflectance spectral signal (and the fluorescence spectral signal) may be a function of the concentration distribution of the sample 110. Converting the sensor data to spectral data may allow to determine the concentration distribution of the sample 110 based on the reflectance measurement.

[0034] To combine the reflectance measurement with a fluorescence measurement the apparatus 130 is configured to obtain fluorescence spectral data indicative of fluorescence spectral signal of a fluorescence measurement of the sample 110. For example, the fluorescence spectral data may be acquired by the same sensor as the sensor data, e.g., the sensor 122. The sensor data and the fluorescence spectral data may be measured by an RGB sensor 122, for example. Thus, the sensor data and the fluorescence spectral data can be acquired by at most one optical imaging sensor. For example, the sensor data and the fluorescence spectral data may be acquired simultaneously by the sensor 122. Alternatively, the sensor 122 may be a multispectral sensor. A multispectral sensor, also known as an RGB infrared sensor, is an optical imaging sensor that combines the ability to capture visible light (RGB) and non-visible light, e.g., infrared radiation. Alternatively, the sensor data and the fluorescence spectral data may be acquired by two different sensors. In this way, a design of the sensor can be adjusted to match the reflectance and / or the fluorescence characteristic of the sample 110, e.g., of fluorescent tracers or dyes used for the fluorescent measurement.

[0035] For example, the fluorescence spectral data may be received from the sensor 122. Alternatively, the sensor 122 may be part of the apparatus 130. Thus, the apparatus 130 may measure the fluorescence spectral data by controlling an image acquisition of the sensor 122. Alternatively, the fluorescence spectral data may be retrieved from a storage device, e.g., the storage device 136 or a frame buffer (which can be separate from the apparatus 130). The fluorescence spectral data may be raw data. That is, the apparatus 130 may post process the sensor data to determine the reflectance measurement data. Alternatively, the sensor data may be already post processed. For example, the sensor 122 may have post processed the fluorescence spectral data and no further post processing by the apparatus 130 may be necessary.

[0036] The fluorescent measurement can be performed without the use of specific fluorescent tracers or dyes. Intrinsic fluorescence, also known as autofluorescence, refers to the natural fluorescence emitted by certain molecules or structures without the need for external fluorescent dyes or tracers. Many endogenous molecules in biological samples, such as proteins, nucleic acids (DNA, RNA), and certain metabolites, can exhibit autofluorescence. These molecules have inherent fluorescence properties that allow them to absorb light at specific wavelengths and emit fluorescence at longer wavelengths. However, autofluorescence signals can vary and be more challenging to distinguish from background noise compared to fluorescence measurements from specific fluorescence tracers or dyes. Therefore, additionally or alternatively, the fluorescence measurement can be performed with fluorescence tracers or dyes. By using fluorescent tracers or dyes that are specifically designed to bind to or interact with the target material of the sample 110, a measurable fluorescent spectral signal that correlates with the concentration of the material of the sample 110 can be measured.

[0037] Since the fluorescence measurement and thus the fluorescence spectral signal depends on the autofluorescence of the sample 110 and / or a number of fluorescent tracers or dyes, the concentration distribution of the sample 110 is in principle proportional to the fluorescence spectral signal. Therefore, no conversion of the fluorescence measurement of the sample 110 may be required to determine a concentration distribution of the sample 110 based on the fluorescence measurement.

[0038] The reflectance spectral signal and the fluorescence spectral signal may comprise information about the same material of the sample 110. However, a determination of the material concentration based on the reflectance spectral signal or the fluorescence spectral signal may result in different concentration distributions of the material for each spectral signal. In addition, a material that is not detected by the reflectance spectral signal (and is only detected by the fluorescence spectral signal) may affect a concentration of a material determined based on the reflectance spectral signal. For example, the reflectance measurement and fluorescence measurement may comprise complementary information about the concentration distribution of the sample 110. For example, oxygen of globin appears different in reflectance and fluorescence measurements. Thus, by combining the reflectance spectral signal and the fluorescence spectral signal a determination of the concentration distribution of the sample 110 can be improved.

[0039] Therefore, the apparatus 130 is further configured to combine the reflectance spectral data and the fluorescence spectral data to combined spectral data. The combined spectral data may comprise information of both, the reflectance measurement and the fluorescence measurement. Thus, a determination of a concentration distribution of the sample 110 can be improved. Interactions between different materials can be considered an improved way. The apparatus 130 is configured to determine the concentration distribution of the sample 110 based on the combined spectral data. In this way, the determination of the concentration distribution of the sample 110 can be improved by combining measurements of a reflectance mode and a fluorescence mode of the optical imaging system. The reflectance spectral data and the fluorescence spectral data can be used to analyze the properties of materials, identify specific substances, distinguish between different classes of objects, or extract valuable information from the sample. Converting the sensor data into reflectance spectral data may allow to combine the reflectance spectral data and the fluorescence spectral data to determine the concentration distribution of the sample. By converting the sensor data into the reflectance spectral data, data depending on a concentration distribution of the sample 110 can be generated for the reflectance measurement. Thus, the reflectance measurement can be combined with fluorescence measurement for evaluation. In this way, the concentration distribution of the sample can be determined based on both, the reflectance measurement and the fluorescence measurement. This may improve an accuracy of the determined concentration distribution.

[0040] Multispectral imaging of both reflectance measurement and fluorescence measurement is being used in microsurgical microscopes for simultaneous white light reflectance imaging and fluorescence imaging of clinically used fluorescent dyes, for example. The fluorescence information is combined with the white light image to create a pseudocolor image. This application utilizes simultaneous acquisition of multispectral imaging of reflectance and fluorescence but then treats the two modes (i.e., reflectance mode and fluorescence mode) separately. However, today each measurement mode (i.e., reflectance mode and fluorescence mode) is used independently. Although both measurement modes can be used to analyze in vivo biological tissues, e.g., reflectance multispectral imaging for blood oxygenation, and multispectral fluorescence imaging for tissue type classification, they are not used combined.

[0041] Converting the sensor data into reflectance spectral data and combining the reflectance spectral signal with the fluorescence spectral signal may allow to use not only the reflectance mode or fluorescence mode of the optical imaging system. In contrast, both the reflectance mode (reflectance measurement) and the fluorescence mode (fluorescence measurement) can be combined to determine the concentration distribution of the sample 110. In this way, the determination of the concentration distribution of the sample 110 can be improved.

[0042] The apparatus 130 is configured to provide hybrid or combined reflectance and fluorescence multispectral imaging aiming to increase the generated insights by combining two fundamentally different modes, reflectance mode and fluorescence mode of the optical imaging system. Optionally, as described below spectral unmixing may be utilized to determine the concentration distribution of the sample 110.

[0043] A finding of the inventors is to convert the reflectance measurement into an absorption-proportional signal, i.e., convert the sensor data into reflectance spectral data. The physics principle behind it is that reflectance (or transmittance) is not proportional to the concentration of the absorber, while absorption is proportional. Thus, converting the reflectance measurement into an absorption-proportional signal may allow to combine absorption measurement (i.e., fluorescence measurement) with reflectance measurement. In this way, the reflectance mode and the fluorescence mode of the optical imaging system can be combined to determine a concentration distribution of the sample 110.

[0044] Therefore, an ability of multispectral imaging technology to extract insights for biological tissues by combining evaluation of absorption and fluorescence properties of the molecular components of the sample 110 can be improved. For example, for clinical applications tissue recognition and classification, e.g., healthy versus pathologic tissue, of different tissue types (e.g., muscle, nerve, fat, bone) can be improved.

[0045] The proposed concept may be built around two main components - the microscope, which comprises the optical components being used to view the sample 110, and the apparatus 130, which may be used to control the optical imaging system 100, process sensor data of the microscope, e.g., the sensor 122, and / or to determine the concentration distribution of the sample 110.

[0046] In general, a microscope is an optical instrument that is suitable for examining objects that are too small to be examined by the human eye (alone). For example, a microscope may provide an optical magnification of a sample, such as a sample 110 shown in Fig. 1. In modern microscopes, the optical magnification is often provided for a camera or an imaging sensor, such as the optical imaging sensors 122 of the microscope.

[0047] In an example, the apparatus 130 may be further configured to determine the reflectance spectral data using a logarithmic function. For example, the apparatus 130 may determine the reflectance spectral data by applying a logarithmic function to the sensor data or a signal derived from the sensor data. A reflectance measurement with a high reflectance may mean that an absorption of the sample 110 is low (e.g., caused by a low concentration of a material). Thus, a high reflectance of the sample 110 may result in an image being acquired with a high intensity for a low concentration of a material. Accordingly, a low reflectance of the sample 110 may result in an image being acquired with a low intensity for a high concentration of a material.

[0048] In contrast, a fluorescence measurement of the sample 110 may result in an image being acquired with a high intensity for a high concentration of a material. The fluorescence measurement of the sample 110 may result in an image being acquired with a low intensity for low concentration of a material.

[0049] Thus, a negative logarithmic function may be used to align the reflectance measurement and the fluorescence measurement. For example, the sensor data (comprising the reflectance measurement r of the sample 110) may be converted to an absorption-proportional signal a (the reflectance spectral signal) using the Beer-Lambert law: a = -log(r). For example, using the Beer-Lambert law may be justified by a fully reflectance layer at the end of the optical path of the microscope. It is assumed that a highly reflectance layer may reflect all incident light. Further, scattering or reflection effects may be neglected, such that no reference spectrum may be needed. If scattering or reflection are neglected and it is assumed that the only factor affecting the measured absorbance is the concentration of the absorbing species, then the Beer-Lambert law can be used without a reference spectrum. Additionally or alternatively, a reference spectrum can be used to improve a reliability of the conversion of the sensor data into the spectral reflectance data. Using a logarithmic function may allow to determine the reflectance spectral data with reduced computational effort.

[0050] Additionally or alternatively, alternative approaches to the Beer-Lambert law can be used to establish a relationship between reflectance and concentration of the material of the sample 110 without using a logarithmic function. For example, the Kubelka-Munk theory, which provides a mathematical model for relating reflectance to the concentration of a material, can be used.

[0051] In an example, the apparatus 130 may be configured to determine the combined spectral data by combining the reflectance spectral data and the fluorescence spectral data in a single vector. A single vector may reduce a computational complexity and / or may simplify an algorithm needed for further processing, e.g., spectral unmixing. Alternatively, a collection or a set of vectors can be used by the apparatus 130 for combining the reflectance spectral data and the fluorescence spectral data. For example, the apparatus 130 may be configured to determine the combined spectral data by combining the reflectance spectral data and the fluorescence spectral data in a matrix, vector field and / or a list of array.

[0052] In an example, the apparatus 130 may be further configured to determine the concentration distribution using spectral unmixing. Spectral unmixing is a computational process used to extract and separate individual spectral signatures or components present in a multispectral or hyperspectral image. In these multispectral or hyperspectral images, each pixel contains a spectrum of data, representing the reflectance and / or emission characteristics of the materials within the sample 110. Spectral unmixing algorithms aim to identify the contributions of different materials or substances present in each pixel, effectively "unmixing" the spectral information and providing estimates of the abundance of each component. The combined spectral data may comprise reflectance characteristics (reflectance spectral signal) and emission characteristics (fluorescence spectral signal).

[0053] Instead of segmenting the information of the reflectance measurement and the fluorescence measurement the combined spectral data depends on parameters of the reflectance measurement and the fluorescence measurement. For example, if spectral signal 1 (reflectance measurement, reflectance spectral signal) comprises information about cl (concentration of material 1) and c2 (concentration of material 2) and c5 (concentration of material 5) and spectral signal 2 (fluorescence measurement, fluorescence spectral signal) comprises information about c3 (concentration of material 3) and c4 (concentration of material 4) and c5, unmixing of c5 for two different spectral signals can be avoided. By combining the reflectance spectral data and fluorescence spectral data before unmixing, unmixing is only performed once for c5.

[0054] Further, c5 may depend, for example, on c3, and since spectral signal 1 is blind for c3, the result of c5 determined based on spectral signal 1 may be wrong. If the separated approach is used to determine c5 from spectral signal 1 and spectral signal 2 independently, the average of the two individual values of c5 for the spectral signal 1 and the spectral signal 2 would be taken as total value. This averaging may be incorrect for the determination of c5. By combining the information about all concentrations cl-c5 by combining the spectral signal 1 and the spectral signal 2 before unmixing, the determination of the concentrations, e.g., c5, can be improved.

[0055] Spectral unmixing can be done with different methods. For example, Independent Component Analysis (ICA) can be used. ICA is a method used to separate a multivariate signal into independent non-Gaussian components. It is often used to separate mixed signals in hyperspectral imagery. For example, Orthogonal Subspace Projection (OSP) can be used. OSP is a method that projects data onto a subspace that is orthogonal to the subspace spanned by the endmembers. It is often used to estimate the abundance fractions of mixed pixels in hyperspectral imagery. For example, Minimum Volume Simplex Analysis (MVSA) can be used. MVSA is a method that uses a simplex volume minimization algorithm to estimate the abundance fractions of mixed pixels in hyperspectral imagery. For example, Convex Cone Analysis (CCA) can be used. CCA is a method that uses a convex cone algorithm to estimate the abundance fractions of mixed pixels in hyperspectral imagery. For example, Fully Constrained Least Squares (FCLS) can be used. FCLS is a method that uses at least squares algorithm to estimate the abundance fractions of mixed pixels in hyperspectral imagery while constraining the abundance fractions to be non-negative and sum to 1. In an example, the apparatus 130 may be configured to perform the spectral unmixing using linear transformation, ICA, OSP, MVSA, CCA and / or FCLS.

[0056] Alternatively, to spectral unmixing spectral classification or spectral clustering can be used. Instead of decomposing the spectral information into individual components as in spectral unmixing, spectral classification focuses on grouping pixels with similar spectral signatures into distinct classes.

[0057] As shown in Fig. 1 the optional one or more interfaces 132 is coupled to the respective one or more processors 134 at the apparatus 130. In examples the one or more processors 134 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. Similar, the described functions of the one or more processors 134 may as well be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc. The one or more processors 134 is capable of controlling the one or more interfaces 132, so that any data transfer that occurs over the one or more interfaces 132 and / or any interaction in which the one or more interfaces 132 may be involved may be controlled by the one or more processors 134.

[0058] In an embodiment the apparatus 130 may comprise a memory, e.g., the one or more storage devices 136 and at least one or more processors 134 operably coupled to the memory and configured to perform the method described below.

[0059] In examples the one or more interfaces 132 may correspond to any means for obtaining, receiving, transmitting or providing analog or digital signals or information, e.g., any connector, contact, pin, register, input port, output port, conductor, lane, etc. which allows providing or obtaining a signal or information. The one or more interfaces 132 may be wireless or wireline and it may be configured to communicate, e.g., transmit or receive signals, information with further internal or external components.

[0060] The apparatus 130 may be a computer, processor, control unit, (field) programmable logic array ((F)PLA), (field) programmable gate array ((F)PGA), graphics processor unit (GPU), application-specific integrated circuit (ASICs), integrated circuits (IC) or system-on-a-chip (SoCs) system.

[0061] More details and aspects are mentioned in connection with the examples described below. The example shown in Fig. 1 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described below (e.g., Fig. 2 - 4).

[0062] Fig. 2 shows a schematic flow diagram of an example of a method 200. The method 200 comprises obtaining 210 first sensor data and second sensor data. The first sensor data may be the sensor data as described with reference to Fig. 1. The second sensor data may be the fluorescence spectral data as described with reference to Fig. 1.

[0063] The first sensor data may comprise the measurement data n of a reflectance multi spectral imaging. For example, an image may be acquired for n spectral channels. Thus, the first sensor data may be indicative of a reflectance measurement of a sample for n channels (n, 7 . . . , rn). The second sensor data comprise the measurement data of a fluorescence multispectral imaging. For example, an image may be acquired for m spectral channels. Thus, the second sensor data may be indicative of a fluorescence measurement of the sample for m channels

[0064] As described above the first sensor data and the second sensor may be received from at most one optical imaging sensor or multiple optical imaging sensors. The first sensor data (e.g., reflectance multispectral cube (n, 7 . . ., rn)) and the second sensor data (fluorescence multi- spectral cube ( / i, / 2, . . . ,fm)) may be starting data for the method 200.

[0065] In 212 the first sensor data may be converted into an absorption-proportional signal (e.g., reflectance spectral data), e.g., using a negative logarithmic function. For example, the Beer- Lambert law may be used a = -log(r). Since the second sensor data may be already absorptionproportional measurement data, no further post processing of the second sensor data may be necessary.

[0066] Thus, at 220 spectral signals proportional to a concentration of a material of the sample for both, reflectance and fluorescence measurement may be given. For example, the reflectance spectral signal may be defined by a reflectance vector (ai, «2,..., an). For example, the fluorescence spectral signal may be defined by a fluorescence vector ( / i, / 2, . . . ,fm).

[0067] In 222 the reflectance vector and the fluorescence vector may be combined into a single vector. Thus, at 230 a hybrid signal (i.e., the combined spectral data) proportional to a concentration of a material of the sample may be given. The single vector comprising the reflectance vector and the fluorescence vector may be defined by (ai, r / 2, . . . , an, / i,^, . . ., n).

[0068] In 232 spectral unmixing may be performed. The spectral unmixing may comprise a linear transformation. For example, the spectral unmixing in 232 on the hybrid vector may be performed to calculate the concentration c of the biological components of the sample. Thus, at 240 the concentration of biological components of the sample, e.g., for different materials of the sample (ci, ci,.. ., Ck), may be certain.

[0069] More details and aspects are mentioned in connection with the examples described above and / or below. The example shown in Fig. 2 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1) and / or below (e.g., Fig. 3 - 4).

[0070] Fig. 3 shows a schematic flow diagram of another example of a method 300 for an optical imaging system. The method 300 may be performed by an apparatus as described with reference to Fig. 1. The method 300 comprises obtaining 310 sensor data indicative of a reflectance measurement of a sample and converting 320 the sensor data into reflectance spectral data indicative of a reflectance spectral signal. The reflectance spectral signal is a function of a concentration distribution of the sample. Further, the method 300 comprises obtaining 330 fluorescence spectral data indicative of a fluorescence spectral signal of a fluorescence measurement of the sample and combining 340 the reflectance spectral data and the fluorescence spectral data to combined spectral data. The method 300 further comprises determining 350, based on the combined spectral data, the concentration distribution of the sample.

[0071] More details and aspects are mentioned in connection with the examples described above and / or below. The example shown in Fig. 3 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1 - 2) and / or below (e.g., Fig. 4).

[0072] Fig. 4 shows a schematic illustration of a system 400, e.g., an optical imaging system 400. The optical imaging system 400 may comprise an apparatus as described with reference to Fig. 1 and a microscope 410. For example, the microscope 410 may comprise or can be communicatively coupled to the apparatus. Alternatively, the computer system 420 may comprise the apparatus. The apparatus can be used to determine a concentration distribution of a sample, for example.

[0073] In an example, the optical imaging system 400 may further comprise the first sensor for acquisition of the (first) sensor data and a second sensor for acquisition of second sensor data. The apparatus may be configured to control the first sensor and the second sensor such that the (first) sensor data and the second sensor data may be acquired at the same time. By image acquisition of the first sensor and the second sensor at the same time a reliability of the combined spectral data can be improved. There are a variety of different types of optical imaging systems. If the optical imaging system 400 is used in the medical or biological fields, the sample may be a sample of organic tissue, e.g., arranged within a petri dish or present in a part of a body of a patient. In some examples of the present disclosure, e.g., as shown in Fig. lb, the optical imaging system 400 may be a surgical optical imaging system, e.g., an optical imaging system that is to be used during a surgical procedure, such as an oncological surgical procedure or during tumor surgery. However, the proposed concept may also be applied to other types of microscopy, e.g., microscopy in a laboratory or microscopy for the purpose of material inspection.

[0074] Fig. 4 shows a schematic illustration of a system 400, e.g., an optical imaging system, configured to perform a method described herein, e.g., with reference to Figs. 2 or 3. The system 400 comprises a microscope 410 and a computer system 420. The microscope may comprise the apparatus as described above, e.g., with reference to Fig. 1. The microscope 410 is configured to take images and is connected to the computer system 420. The computer system 420 is configured to execute at least a part of a method described herein. The computer system 420 may be configured to execute a machine learning algorithm. The computer system 420 and microscope 410 may be separate entities but can also be integrated together in one common housing. The computer system 420 may be part of a central processing system of the microscope 410 and / or the computer system 420 may be part of a subcomponent of the microscope 410, such as a sensor, an actor, a camera or an illumination unit, etc. of the microscope 410.

[0075] The computer system 420 may be a local computer device (e.g., personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 420 may comprise any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g., camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 420 may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 420 may include one or more storage devices, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system 420.

[0076] More details and aspects are mentioned in connection with the examples described above. The example shown in Fig. 4 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1 - 3).

[0077] Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0078] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non- transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable. Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0079] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0080] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0081] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0082] A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.

[0083] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

[0084] A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

[0085] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein. A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0086] In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[0087] If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method and vice versa. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system.

[0088] The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Furthermore, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

[0089] The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example. List of reference Signs

[0090] 110 sample

[0091] 122 sensor

[0092] 130 apparatus

[0093] 132 interface

[0094] 134 processor

[0095] 136 storage device

[0096] 200 method for an optical imaging system

[0097] 210 obtaining first sensor data and second sensor data

[0098] 212 converting first sensor data

[0099] 220 spectral signals proportional to a concentration of a material

[0100] 222 combine reflectance vector and the fluorescence vector

[0101] 230 hybrid signal proportional to a concentration of a material

[0102] 232 spectral unmixing

[0103] 240 concentration of materials

[0104] 300 method for an optical imaging system

[0105] 310 obtaining sensor data

[0106] 320 converting the sensor data

[0107] 330 obtaining fluorescence spectral data

[0108] 340 combining the reflectance spectral data

[0109] 350 determining the concentration distribution of the sample

[0110] 400 system

[0111] 410 microscope

[0112] 420 computer system

Claims

Claims1. An apparatus (130) for an optical imaging system, comprising one or more processors (134) and one or more storage devices (136), wherein the apparatus (130) is configured to: obtain sensor data indicative of a reflectance measurement of a sample (110); convert the sensor data into reflectance spectral data indicative of a reflectance spectral signal, wherein the reflectance spectral signal is a function of a concentration distribution of the sample (110); obtain fluorescence spectral data indicative of a fluorescence spectral signal of a fluorescence measurement of the sample (110); combine the reflectance spectral data and the fluorescence spectral data to combined spectral data; and determine, based on the combined spectral data, the concentration distribution of the sample (110).

2. The apparatus (130) according to claim 1, wherein the apparatus (130) is configured to determine the reflectance spectral data using a logarithmic function.

3. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured to determine the combined spectral data by combining the reflectance spectral data and the fluorescence spectral data in a single vector.

4. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured to determine the concentration distribution using spectral unmixing.

5. The apparatus (130) according to claim 4, wherein the apparatus (130) is configured to perform spectral unmixing using at least one of linear transformation, independent component analysis, orthogonal subspace projection, minimum volume simplex analysis, convex cone analysis or fully constrained least squares.

6. An optical imaging system, comprising an apparatus (130) according to any one of the preceding claims.

7. The optical imaging system, further comprising a first sensor for acquisition of the sensor data; a second sensor for acquisition of second sensor data; and wherein the apparatus (130) is configured to control the first sensor and the second sensor such that the sensor data and the second sensor data are acquired at the same time.

8. A method (300) for an optical imaging system, comprising obtaining (310) sensor data indicative of a reflectance measurement of a sample; converting (320) the sensor data into reflectance spectral data indicative of a reflectance spectral signal, wherein the reflectance spectral signal is a function of a concentration distribution of the sample; obtaining (330) fluorescence spectral data indicative of a fluorescence spectral signal of a fluorescence measurement of the sample; combining (340) the reflectance spectral data and the fluorescence spectral data to combined spectral data; and determining (350), based on the combined spectral data, the concentration distribution of the sample.

9. A computer program with a program code for performing the method (300) according to claim 8 when the computer program is executed on a processor.

Images