Apparatus for optical imaging system, optical imaging system, method and computer program

Abstract

Examples relate to an apparatus (130) for an optical imaging system, the apparatus (130) comprising one or more processors (134) and one or more storage devices (136). The apparatus is configured to obtain sensor data indicative of a reflection measurement of the sample (110). The apparatus is further configured to convert the sensor data into reflectance spectral data indicative of the reflectance spectral signal. The reflected spectral signal is a function of a concentration profile of the sample. The apparatus is further configured to obtain fluorescence spectrum data indicative of a fluorescence spectrum signal of a fluorescence measurement of the sample. The apparatus is further configured to combine the reflectance spectral data and the fluorescence spectral data into combined spectral data. The apparatus is further configured to determine a concentration profile of the sample based on the combined spectral data.

Description

Technical Field

[0001] Examples relate to apparatus, optical imaging systems, methods, and computer programs used in optical imaging systems. Background Technology

[0002] Microscopes with optical imaging systems can be used for imaging in various imaging modes. For example, a microscope may include two optical imaging sensors, one for fluorescence imaging and the other for reflection imaging. These two optical imaging sensors can be used for imaging at different wavelengths. Such microscopes typically include a light source that emits light that can be picked up by the respective sensors, for example, in reflected form (i.e., at the same wavelength as the emitted light) or in fluorescent form (i.e., at a wavelength different from the emitted light).

[0003] There may be expectations for improved designs for microscopes, in which various imaging modes are used in a more versatile manner. Summary of the Invention

[0004] This expectation is addressed by the subject matter of the independent claims.

[0005] The concept presented in this disclosure is based on the insight that combining reflectance measurement data with fluorescence measurement data can improve the determination of sample concentration distribution. Since different spectral signals are obtained through different measurement modes, combining reflectance and fluorescence measurement results can improve the determination of concentration distribution.

[0006] An example provides an apparatus for an optical imaging system, comprising one or more processors and one or more storage devices. The apparatus is configured to acquire sensor data indicating reflectance measurements of a sample. The apparatus is further configured to convert the sensor data into reflectance spectral data indicating a reflectance spectral signal, which is a function of the concentration distribution of the sample. Furthermore, the apparatus is configured to acquire fluorescence spectral data indicating fluorescence measurements of the sample. The apparatus is further configured to combine the reflectance spectral data and the fluorescence spectral data into combined spectral data. The apparatus is further configured to determine the concentration distribution of the sample based on the combined spectral data. The reflectance spectral data and the fluorescence spectral data can be used to analyze the properties of materials, identify specific substances, distinguish different categories of objects, or extract valuable information from a sample. Converting sensor data into reflectance spectral data allows for the combination of reflectance spectral data and fluorescence spectral data to determine the concentration distribution of the sample. By converting sensor data into reflectance spectral data, data dependent on the concentration distribution of the sample based on the reflectance measurement results can be generated. Therefore, the reflectance measurement results can be combined with the fluorescence measurement results for evaluation. In this way, the concentration distribution of the sample can be determined based on both the reflectance measurement results and the fluorescence measurement results. This can improve the accuracy of the determined concentration distribution.

[0007] In the example, the device can be further configured to determine reflectance spectral data using a logarithmic function. For example, the device can determine reflectance spectral data by applying a logarithmic function to sensor data or a signal derived from the sensor data. Using a logarithmic function makes it possible to determine reflectance spectral data in a convenient manner.

[0008] In the example, the device can be further configured to determine combined spectral data by combining reflectance spectral data and fluorescence spectral data into a single vector. Using a single vector makes it possible to represent spectral data indicating the concentration distribution of a sample in a convenient manner. For example, using a single vector can reduce computational complexity and / or simplify the algorithms required for further processing (e.g., spectral demixing).

[0009] In the example, the device can be further configured to use spectral unmixing to determine the concentration distribution. For example, spectral unmixing can enable the extraction of sub-pixel information revealing the presence of material occupying a portion of a pixel, estimation of different materials within a mixed pixel, and / or identification and characterization of different materials present within a mixed pixel.

[0010] In the example, the apparatus can be configured to perform spectral unmixing using linear transformation, independent component analysis, orthogonal subspace projection, minimum volume simplex analysis, convex cone analysis, and / or fully constrained least squares.

[0011] The example provides an optical imaging system that includes the means as described above.

[0012] In the example, the optical imaging system may further include a first sensor for acquiring data from a (first) sensor and a second sensor for acquiring data from a second sensor. The device can be configured to control the first and second sensors such that data from both sensors can be acquired simultaneously. By simultaneously acquiring images from the first and second sensors, the reliability of the combined spectral data can be improved.

[0013] The example provides a method for an optical imaging system, comprising acquiring sensor data indicating a reflectance measurement of a sample, and converting the sensor data into reflectance spectral data indicating a reflectance spectral signal. The reflectance spectral signal is a function of the concentration distribution of the sample. The method further comprises acquiring fluorescence spectral data indicating a fluorescence measurement of the sample, and combining the reflectance spectral data and the fluorescence spectral data into combined spectral data. The method further comprises determining the concentration distribution of the sample based on the combined spectral data.

[0014] The various examples disclosed herein relate to corresponding computer programs having program code that, when executed on a processor, performs the methods described above. Attached Figure Description

[0015] The following will describe some examples of apparatus and / or methods by way of example and with reference to the accompanying drawings, wherein:

[0016] Figure 1 A schematic diagram of an example device for an optical imaging system is shown;

[0017] Figure 2 A schematic flowchart illustrating an example of the method is shown;

[0018] Figure 3 A schematic flowchart illustrating another example of a method for an optical imaging system; and

[0019] Figure 4 A schematic diagram of the system is shown. Detailed Implementation

[0020] The various examples will now be described more fully with reference to the accompanying drawings, some of which illustrate certain aspects. In the drawings, the thickness of lines, layers, and / or regions may be exaggerated for clarity.

[0021] Figure 1A schematic diagram of an example of a device 130 for an optical imaging system is shown. The task of device 130 is to control the microscope 120 and various aspects of the optical imaging system 100 and / or process various types of sensor data from the optical imaging system 100, such as sensor data and / or fluorescence spectral data. Therefore, device 130 can be implemented as a computer system that interfaces with various components of the optical imaging system (e.g., sensor 122).

[0022] like Figure 1 As shown, device 130 includes one or more processors 134 and one or more storage devices 136. Optionally, device 130 further includes one or more interfaces 132. The one or more processors 134 are coupled to one or more storage devices 136 and optionally to one or more interfaces 132. Typically, the functionality of device 130 may be provided by one or more processors 134 (e.g., for determining concentration distribution) in combination with one or more interfaces 132 (for exchanging information, e.g., with sensor 122, e.g., to receive sensor data and second sensor data) and / or in combination with one or more storage devices 136 (for storing and / or retrieving information).

[0023] Apparatus 130 is configured to acquire sensor data indicating the reflection measurement result of sample 110. For example, the sensor data may be received from sensor 122. Alternatively, sensor 122 may be part of apparatus 130. Therefore, apparatus 130 can measure the sensor data by controlling image acquisition from sensor 122. Alternatively, the sensor data may be retrieved from a storage device (e.g., storage device 136) or a frame buffer (which may be separate from apparatus 130). The sensor data may be raw data. That is, apparatus 130 may post-process the sensor data to determine the reflection measurement data. Alternatively, the sensor data may have already been post-processed. For example, sensor 122 may have already post-processed the sensor data, and further post-processing by apparatus 130 may not be required.

[0024] Reflectance is a measure of the amount of light reflected by sample 110 at a given wavelength. It can be expressed as the ratio of reflected light intensity to incident light intensity. The incident light can be emitted by the light-emitting module of the optical imaging system. The light-emitting module can be used to provide illumination to sample 110 in multiple wavelengths. For example, the wavelength used for reflectance measurements can be different from the wavelength used for fluorescence measurements. The light-emitting module can be configured to emit wavelengths used for both reflectance and fluorescence measurements.

[0025] Reflectance measurements can provide the information needed to determine or construct a spectral signal (e.g., a reflectance spectral signal). Reflectance measurements can include measurements of reflectance at different wavelengths, which can be used to generate a spectral curve representing the reflectance characteristics of sample 110. However, reflectance measurements may not be proportional to the concentration distribution of sample 110. Therefore, it is not possible to directly combine reflectance measurements with fluorescence measurements. To enable the combination of reflectance measurements with fluorescence measurements, device 130 is configured to convert reflectance measurements (i.e., sensor data).

[0026] The device is configured to convert sensor data into reflectance spectral data indicating a reflectance spectral signal. The reflectance spectral signal derived from the reflectance measurement results allows for the combination of reflectance and fluorescence measurement results to determine the concentration distribution of sample 110. For example, the reflectance spectral signal derived from the reflectance measurement results allows for the characterization and / or analysis of materials based on unique spectral characteristics. This enables, for example, material identification, quality control, and / or concentration distribution determination.

[0027] Spectral signals (such as reflectance or fluorescence spectra) can refer to electromagnetic radiation or light as a function of wavelength or frequency. Spectral signals can provide information about the intensity and / or amplitude of electromagnetic radiation or light at different wavelengths or frequency bands.

[0028] A spectral signal can refer to measured or recorded spectral information about a specific location or pixel in an image of sample 110. A spectral signal can represent the reflectance (for reflectance measurements) or emission (for fluorescence measurements) characteristics of sample 110 at that pixel. A spectral signal can be represented as a spectrum, which is a graph of intensity or radiance relative to wavelength or frequency.

[0029] Spectral signals can be used to analyze material properties, identify specific substances, distinguish different categories of objects, or extract valuable information from a scene. Spectral unmixing algorithms fully utilize spectral signals to estimate the abundance of different materials or components within each pixel, enabling characterization and analysis of complex scenes, for example.

[0030] The reflectance spectral signal (and fluorescence spectral signal) can be a function of the concentration distribution of sample 110. Converting sensor data into spectral data makes it possible to determine the concentration distribution of sample 110 based on reflectance measurements.

[0031] To combine reflectance measurements with fluorescence measurements, device 130 is configured to acquire fluorescence spectral data indicating the fluorescence spectral signal of the fluorescence measurement results for sample 110. For example, the fluorescence spectral data can be acquired by the same sensor as the sensor data (e.g., sensor 122). The sensor data and fluorescence spectral data can be measured, for example, by RGB sensor 122. Therefore, the sensor data and fluorescence spectral data can be acquired by at most one optical imaging sensor. For example, the sensor data and fluorescence spectral data can be acquired simultaneously by sensor 122. Alternatively, sensor 122 can be a multispectral sensor. A multispectral sensor (also known as an RGB infrared sensor) is an optical imaging sensor that combines the ability to capture visible (RGB) and invisible light (e.g., infrared radiation). Alternatively, the sensor data and fluorescence spectral data can be acquired by two different sensors. In this way, the sensor design can be adjusted to match the reflectance and / or fluorescence characteristics of sample 110 (e.g., a fluorescent tracer or dye used for fluorescence measurements).

[0032] For example, fluorescence spectral data can be received from sensor 122. Alternatively, sensor 122 can be part of device 130. Therefore, device 130 can measure fluorescence spectral data by controlling image acquisition from sensor 122. Alternatively, fluorescence spectral data can be retrieved from a storage device (e.g., storage device 136) or a frame buffer (which may be separate from device 130). The fluorescence spectral data can be raw data. That is, device 130 can post-process the sensor data to determine reflectance measurement data. Alternatively, the sensor data may have already been post-processed. For example, sensor 122 may have already post-processed the fluorescence spectral data, and further post-processing by device 130 may not be required.

[0033] Fluorescence measurements can be performed without the use of specific fluorescent tracers or dyes. Intrinsic fluorescence (also known as autofluorescence) refers to the natural fluorescence emitted by certain molecules or structures without the need for external fluorescent dyes or tracers. Many endogenous molecules in biological samples, such as proteins, nucleic acids (DNA, RNA), and certain metabolites, can exhibit autofluorescence. These molecules have inherent fluorescent properties that allow them to absorb light of specific wavelengths and emit fluorescence of longer wavelengths. However, autofluorescence signals vary compared to fluorescence measurements from specific fluorescent tracers or dyes, and distinguishing them from background noise is extremely challenging. Therefore, additionally or alternatively, fluorescence measurements can be performed using fluorescent tracers or dyes. By using fluorescent tracers or dyes specifically designed to bind to or interact with the target material of sample 110, measurable fluorescence spectral signals related to the material concentration of sample 110 can be measured.

[0034] Since the fluorescence measurement results and the resulting fluorescence spectral signal depend on the autofluorescence of sample 110 and / or some fluorescent tracer or dye, the concentration distribution of sample 110 is, in principle, proportional to the fluorescence spectral signal. Therefore, the concentration distribution of sample 110 can be determined based on the fluorescence measurement results without conversion.

[0035] Reflectance and fluorescence spectra can both contain information about the same material in sample 110. However, determining material concentration based on either reflectance or fluorescence spectra may result in different concentration distributions for each spectral signal. Furthermore, materials not detected by reflectance (but only by fluorescence) can affect the concentration determined based on reflectance. For example, reflectance and fluorescence measurements may contain complementary information about the concentration distribution of sample 110. For instance, oxygen in globin may exhibit different characteristics in reflectance and fluorescence measurements. Therefore, combining reflectance and fluorescence spectra can improve the determination of the concentration distribution of sample 110.

[0036] Therefore, device 130 is further configured to combine reflectance spectral data and fluorescence spectral data into combined spectral data. The combined spectral data may include information from both reflectance and fluorescence measurements. Therefore, the determination of the concentration distribution of sample 110 can be improved. Interactions between different materials can be considered as a means of improvement. Device 130 is configured to determine the concentration distribution of sample 110 based on the combined spectral data. In this way, the determination of the concentration distribution of sample 110 can be improved by combining the measurement results of the reflectance and fluorescence modes of the combined optical imaging system.

[0037] Reflectance and fluorescence spectral data can be used to analyze material properties, identify specific substances, distinguish different categories of objects, or extract valuable information from samples. Converting sensor data into reflectance spectral data allows for the combination of reflectance and fluorescence spectral data to determine the concentration distribution of a sample. By converting sensor data into reflectance spectral data, concentration distribution data dependent on sample 110 can be generated based on the reflectance measurement results. Therefore, reflectance measurement results can be combined with fluorescence measurement results for evaluation. In this way, the concentration distribution of the sample can be determined based on both reflectance and fluorescence measurement results. This can improve the accuracy of the determined concentration distribution.

[0038] Multispectral imaging for both reflectance and fluorescence measurements is being used in microsurgical microscopes for simultaneous white light reflectance and fluorescence imaging, such as for clinically used fluorescent dyes. Fluorescence information is combined with a white light image to produce a pseudo-color image. This application utilizes the simultaneous acquisition of multispectral imaging of reflectance and fluorescence, but then processes the two modes (i.e., reflectance mode and fluorescence mode) separately. However, currently, each measurement mode (i.e., reflectance mode and fluorescence mode) is used independently. Although both measurement modes can be used to analyze biological tissues in vivo (e.g., reflectance multispectral imaging for blood oxygenation, multispectral fluorescence imaging for tissue type classification), they are not used in combination.

[0039] Converting sensor data into reflectance spectral data and combining the reflectance spectral signal with the fluorescence spectral signal allows for the use of more than just the reflectance or fluorescence mode of the optical imaging system. Instead, it enables the combination of both the reflectance mode (reflectance measurement result) and the fluorescence mode (fluorescence measurement result) to determine the concentration distribution of sample 110. In this way, the determination of the concentration distribution of sample 110 can be improved.

[0040] Device 130 is configured to provide mixed or combined reflectance and fluorescence multispectral imaging, designed to increase the resulting insights by combining two completely different modes (reflectance mode and fluorescence mode) of the optical imaging system. Optionally, as described below, spectral unmixing can be used to determine the concentration distribution of sample 110.

[0041] The inventors discovered that converting reflectance measurements into an absorption-proportional signal—that is, converting sensor data into reflectance spectral data—is a key technological advancement. The underlying physics is that reflection (or transmission) is not proportional to the concentration of the absorbant, while absorption is. Therefore, converting reflectance measurements into an absorption-proportional signal allows for the combination of absorption measurements (i.e., fluorescence measurements) with reflectance measurements. In this way, the reflection and fluorescence modes of an optical imaging system can be combined to determine the concentration distribution of sample 110.

[0042] Therefore, by combining the absorption and fluorescence properties of the molecular components of sample 110, the ability of multispectral imaging technology to extract insights into biological tissues can be improved. For example, in clinical applications, it can improve the identification and classification of different tissue types (e.g., muscle, nerve, fat, bone), such as healthy tissue relative to pathological tissue.

[0043] The proposed concept can be built around two main components—a microscope and a device 130, the microscope including optical components for viewing sample 110, and the device 130 for controlling the optical imaging system 100, processing sensor data from the microscope (e.g., sensor 122), and / or for determining the concentration distribution of sample 110.

[0044] Typically, a microscope is an optical instrument suitable for examining objects too small to be examined (by the human eye alone). For example, a microscope can provide a view of samples (such as...) Figure 1 The optical magnification of sample 110 shown is typical of modern microscopes, where optical magnification is usually provided for cameras or imaging sensors, such as the optical imaging sensor 122 of the microscope.

[0045] In the example, device 130 can be further configured to use a logarithmic function to determine reflectance spectral data. For example, device 130 can determine reflectance spectral data by applying a logarithmic function to sensor data or a signal derived from sensor data.

[0046] High reflectance in a reflection measurement may indicate low absorption of sample 110 (e.g., due to a low concentration of the material). Therefore, high reflectance of sample 110 can result in high-intensity imaging for low-concentration materials. Conversely, low reflectance of sample 110 can result in low-intensity imaging for high-concentration materials.

[0047] Conversely, fluorescence measurements of sample 110 can result in acquiring images at high intensity, as is often the case with high-concentration materials. Fluorescence measurements of sample 110 can also result in acquiring images at low intensity, as is often the case with low-concentration materials.

[0048] Therefore, a negative logarithmic function can be used to align reflectance and fluorescence measurements. For example, the Beer-Lambert law: a = -log(r) can be used to convert sensor data (including the reflectance measurement r of sample 110) into a signal a (reflectance spectral signal) proportional to absorption. The use of the Beer-Lambert law can be justified, for example, by the presence of a total reflection layer at the end of the microscope's optical path. It is assumed that the highly reflective layer reflects all incident light. Furthermore, scattering or reflection effects can be ignored, making a reference spectrum unnecessary. If scattering or reflection is ignored, and it is assumed that the only factor affecting the absorbance measurement is the concentration of the absorbing substance, the Beer-Lambert law can be used without a reference spectrum. Additionally or alternatively, a reference spectrum can be used to improve the reliability of converting sensor data into spectral reflectance data. Using a logarithmic function allows for the determination of reflectance spectral data with a reduced computational burden.

[0049] Additionally or alternatively, the relationship between the reflectance of sample 110 and the material concentration can be established using alternative methods of the Beer-Lambert law without the use of a logarithmic function. For example, the Kubelka-Munk theory can be used, which provides a mathematical model relating reflectance to material concentration.

[0050] In the example, device 130 can be configured to determine combined spectral data by combining reflectance and fluorescence spectral data into a single vector. A single vector can reduce computational complexity and / or simplify algorithms required for further processing (e.g., spectral demixing). Alternatively, device 130 can use a set of vectors or a collection of vectors to combine the reflectance and fluorescence spectral data. For example, device 130 can be configured to determine combined spectral data by combining reflectance and fluorescence spectral data into matrices, vector fields, and / or lists of arrays.

[0051] In the example, device 130 can be further configured to use spectral unmixing to determine the concentration distribution. Spectral unmixing is a computational process used to extract and separate individual spectral features or components present in a multispectral or hyperspectral image. In these multispectral or hyperspectral images, each pixel contains spectral data representing the reflection and / or emission properties of the material within sample 110. Spectral unmixing algorithms are designed to identify the contributions of different materials or substances present in each pixel, effectively “unmixing” the spectral information and providing an estimate of the abundance of each component. Combined spectral data can include reflectance properties (reflectance spectral signals) and emission properties (fluorescence spectral signals).

[0052] The combined spectral data depends on the parameters of the reflectance and fluorescence measurements, rather than on the information from the separate reflectance and fluorescence measurements. For example, if spectral signal 1 (reflectance measurement, reflectance spectrum signal) includes information about c1 (concentration of material 1), c2 (concentration of material 2), and c5 (concentration of material 5), and spectral signal 2 (fluorescence measurement, fluorescence spectrum signal) includes information about c3 (concentration of material 3), c4 (concentration of material 4), and c5, then demixing of c5 for the two different spectral signals can be avoided. By combining the reflectance and fluorescence spectral data before demixing, only one demixing occurs for c5.

[0053] Furthermore, c5 can depend on, for example, c3, and since c3 is not detected in spectral signal 1, the result of determining c5 based on spectral signal 1 may be incorrect. If a separation method is used to determine c5 independently from spectral signals 1 and 2, the average of the two individual values ​​of c5 for spectral signals 1 and 2 will be used as the total value. This averaging may be incorrect for determining c5. By combining spectral signals 1 and 2 before demixing to combine information about all concentrations c1 to c5, the determination of concentration (e.g., c5) can be improved.

[0054] Spectral unmixing can be performed using various methods. For example, Independent Component Analysis (ICA) can be used. ICA is a method for separating multivariate signals into independent non-Gaussian components. It is commonly used to separate mixed signals in hyperspectral images. Orthogonal Subspace Projection (OSP) can be used. OSP is a method that projects data onto a subspace orthogonal to the subspace spanned by the endmembers. It is commonly used to estimate the abundance fractions of mixed pixels in hyperspectral images. Minimum Volume Simplex Analysis (MVSA) can be used. MVSA is a method that uses the simplex volume minimization algorithm to estimate the abundance fractions of mixed pixels in hyperspectral images. Convex Cone Analysis (CCA) can be used. CCA is a method that uses the convex cone algorithm to estimate the abundance fractions of mixed pixels in hyperspectral images. Fully Constrained Least Squares (FCLS) can be used. FCLS is a method that uses the least squares algorithm to estimate the abundance fractions of mixed pixels in hyperspectral images while constraining the abundance fractions to be non-negative and totaling 1. In the example, device 130 can be configured to perform spectral demixing using linear transformation, ICA, OSP, MVSA, CCA, and / or FCLS.

[0055] Alternatively, in contrast to spectral unmixing, spectral classification or spectral clustering can be used. Spectral classification focuses on grouping pixels with similar spectral features into different categories, rather than breaking down spectral information into individual components as in spectral unmixing.

[0056] like Figure 1As shown, one or more optional interfaces 132 are coupled to corresponding one or more processors 134 at device 130. In the example, the one or more processors 134 may be implemented using one or more processing units, one or more processing devices, or any apparatus for processing, such as a processor, computer, or programmable hardware component that can operate with correspondingly adapted software. Similarly, the described functionality of the one or more processors 134 may also be implemented in software, which then executes on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. The one or more processors 134 are capable of controlling one or more interfaces 132 such that any data transfer occurring through the one or more interfaces 132 and / or any interaction that the one or more interfaces 132 may participate in can be controlled by the one or more processors 134.

[0057] In an embodiment, device 130 may include a memory (e.g., one or more storage devices 136) and at least one or more processors 134 operatively coupled to the memory and configured to perform the methods described below.

[0058] In the example, one or more interfaces 132 may correspond to any apparatus for acquiring, receiving, transmitting, or providing analog or digital signals or information, such as any connector, contact, pin, register, input port, output port, conductor, channel, etc., that enables the provision or acquisition of signals or information. One or more interfaces 132 may be wireless or wired, and may be configured to communicate with other internal or external components, such as transmitting or receiving signals or information.

[0059] Device 130 may be a computer, processor, control unit, (Field) Programmable Logic Array ((F) PLA), (Field) Programmable Gate Array ((F) PGA), Graphics Processing Unit (GPU), Application-Specific Integrated Circuit (ASIC), Integrated Circuit (IC), or System-on-Chip (SoC) system.

[0060] Further details and aspects will be described in conjunction with the examples described below. Figure 1 The examples shown may include one or more optional additional features, which correspond to one or more aspects mentioned in conjunction with the proposed concept or one or more examples described below (e.g., Figures 2 to 4 ).

[0061] Figure 2 A schematic flowchart illustrating an example of method 200 is shown. Method 200 includes obtaining 210 first sensor data and second sensor data. The first sensor data may be a reference... Figure 1The described sensor data. Second sensor data may be used as a reference. Figure 1 The described fluorescence spectral data.

[0062] The first sensor data may include measurement data from reflectance multispectral imaging. i For example, images of n spectral channels can be acquired. Therefore, the first sensor data can indicate the sample in n channels (r1, r2, ..., r...). n The reflection measurement results of the second sensor. The second sensor data includes measurement data from fluorescence multispectral imaging. i For example, images of m spectral channels can be acquired. Therefore, the second sensor data can indicate the sample in m channels (f1, f2, ..., f...). m The fluorescence measurement results of ).

[0063] As described above, the first sensor data and the second sensor data can be received from at most one optical imaging sensor or from multiple optical imaging sensors. The first sensor data (e.g., a reflectance multispectral cube (r1, r2, ..., r...)) n )) and second sensor data (fluorescence multispectral cubes (f1, f2, ..., f m )) can be the starting data for method 200.

[0064] In section 212, the first sensor data can be converted into a signal proportional to absorption (e.g., reflectance spectral data), for example, using a negative logarithmic function. For instance, the Beer-Lambert law a = -log(r) can be used. Since the second sensor data can already be a measurement proportional to absorption, no further post-processing of the second sensor data is necessary.

[0065] Therefore, at 220, a spectral signal proportional to the concentration of the sample material can be given for both the reflectance measurement result and the fluorescence measurement result. For example, the reflectance spectral signal can be obtained from the reflectance vector (a1, a2, ..., a...). n The fluorescence spectrum signal can be defined by fluorescence vectors (f1, f2, ..., f). m Defined by ).

[0066] At 222, the reflection vector and fluorescence vector can be combined into a single vector. Therefore, at 230, a mixed signal proportional to the concentration of the sample material can be given (i.e., combined spectral data). The single vector including the reflection vector and fluorescence vector can be derived from (a1, a2, ..., a...). n f1, f2, ..., f m Defined by ).

[0067] In section 232, spectral unmixing can be performed. Spectral unmixing can include linear transformations. For example, spectral unmixing in section 232 can be performed on the mixing vector to calculate the concentration c of the biological components of the sample. Therefore, at section 240, the concentration of the biological components of the sample can be determined, for example, for the concentrations of different materials in the sample (c1, c2, ..., c...). k ).

[0068] Further details and aspects will be described in conjunction with the examples described above and / or below. Figure 2 The examples shown may include one or more optional additional features, which correspond to the combination of the proposed concept or the preceding text (e.g., Figure 1 ) and / or the following (e.g., Figures 3 to 4 The one or more examples described herein refer to one or more aspects.

[0069] Figure 3 A schematic flowchart illustrating another example of method 300 for an optical imaging system is shown. Method 300 can be derived from references. Figure 1 The described apparatus is used to perform the method. Method 300 includes obtaining sensor data indicating a reflectance measurement result of the sample at 310, and converting the sensor data at 320 into reflectance spectral data indicating a reflectance spectral signal. The reflectance spectral signal is a function of the concentration distribution of the sample. Method 300 further includes obtaining fluorescence spectral data indicating a fluorescence measurement result of the sample at 330, and combining the reflectance spectral data and the fluorescence spectral data into combined spectral data at 340. Method 300 further includes determining the concentration distribution of the sample based on the combined spectral data at 350.

[0070] Further details and aspects will be described in conjunction with the examples described above and / or below. Figure 3 The examples shown may include one or more optional additional features, which correspond to the combination of the proposed concept or the preceding text (e.g., Figures 1 to 2 ) and / or the following (e.g., Figure 4 The one or more examples described herein refer to one or more aspects.

[0071] Figure 4 A schematic diagram of a system 400 (e.g., an optical imaging system 400) is shown. The optical imaging system 400 may include, as shown in the reference diagram... Figure 1 The described apparatus and microscope 410. For example, microscope 410 may include or be communicatively coupled to the apparatus. Alternatively, computer system 420 may include the apparatus. The apparatus may be used, for example, to determine the concentration distribution of a sample.

[0072] In the example, the optical imaging system 400 may further include a first sensor for acquiring (first) sensor data and a second sensor for acquiring second sensor data. The device can be configured to control the first and second sensors such that (first) sensor data and second sensor data can be acquired simultaneously. By simultaneously acquiring images from the first and second sensors, the reliability of the combined spectral data can be improved.

[0073] There are many different types of optical imaging systems. If the optical imaging system 400 is used in the medical or biological fields, the sample can be a sample of organic tissue, such as that arranged in a Petri dish or present in a part of a patient's body. In some examples of this disclosure, for example, such as... Figure 1 As shown in b, the optical imaging system 400 can be a surgical optical imaging system, for example, an optical imaging system used during surgical procedures (such as oncology surgery) or tumor surgery. However, the proposed concept can also be applied to other types of microscopy, such as laboratory microscopy or microscopy for the purpose of material testing.

[0074] Figure 4 This illustrates a configuration to perform the methods described herein (e.g., references). Figure 2 Or 3) A schematic diagram of system 400 (e.g., an optical imaging system). System 400 includes a microscope 410 and a computer system 420. The microscope may include devices as described above, such as a reference. Figure 1 Microscope 410 is configured to capture images and is connected to computer system 420. Computer system 420 is configured to perform at least a portion of the methods described herein. Computer system 420 may be configured to execute machine learning algorithms. Computer system 420 and microscope 410 may be separate entities, but may also be integrated together in a common housing. Computer system 420 may be part of the central processing system of microscope 410 and / or computer system 420 may be part of sub-components of microscope 410, such as sensors, actuators, cameras, or illumination units of microscope 410.

[0075] Computer system 420 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in different locations (e.g., local clients and / or one or more remote server clusters and / or data centers)). Computer system 420 may include any circuitry or combination of circuitry. In one embodiment, computer system 420 may include one or more processors of any type. As used herein, a processor may refer to any type of computing circuitry, such as, but not limited to, microprocessors, microcontrollers, complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, graphics processors, digital signal processors (DSPs), multi-core processors, field-programmable gate arrays (FPGAs), such as any of the above-described types of computing circuitry in a microscope or microscope component (e.g., a camera), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 420 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuitry) used in wireless devices for mobile phones, tablet computers, portable computers, two-way radios, and similar electronic systems. Computer system 420 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for processing removable media such as optical discs (CDs), flash memory cards, digital video discs (DVDs), etc. Computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that enables a system user to input and receive information from computer system 420.

[0076] Further details and aspects will be described in conjunction with the examples described above. Figure 4 The examples shown may include one or more optional additional features, which correspond to the combination of the proposed concept or the combination of one or more examples described above (e.g., Figures 1 to 3 One or more aspects as described in the document.

[0077] Some or all of the method steps can be performed by (or using) hardware devices, such as, for example, a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most important method steps can be performed by such devices.

[0078] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be performed using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM, or flash memory) that stores electronically readable control signals thereon, which cooperates with (or is capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage medium can be computer-readable.

[0079] Some embodiments of the invention include a data carrier having electronically readable control signals, which is capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0080] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operatively used to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0081] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0082] In other words, therefore, embodiments of the present invention are computer programs having program code that, when run on a computer, performs one of the methods described herein.

[0083] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon for performing one of the methods described herein when executed by a processor. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, including a processor and a storage medium.

[0084] Therefore, another embodiment of the invention represents a data stream or signal sequence for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection, such as via the Internet.

[0085] Another embodiment includes processing equipment, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0086] Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0087] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, mobile device, memory device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0088] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0089] If aspects have been described for a device or system, these aspects should also be understood as descriptions of the corresponding methods, and vice versa. For example, a block, device, or functional aspect of a device or system may correspond to a feature of the corresponding method, such as method steps. Accordingly, aspects described for a method should also be understood as descriptions of corresponding blocks, components, attributes, or functional features of the corresponding device or system.

[0090] The following claims are incorporated herein by reference, each of which may be considered an independent example. It should also be noted that, although in the claims, a dependent claim refers to a specific combination with one or more other claims, other examples may also include combinations of dependent claims with the subject matter of any other dependent or independent claim. Such combinations are expressly stated herein unless otherwise stated. Furthermore, features of a claim should also be included in any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

[0091] The aspects and features described for a specific example in the previous examples can also be combined with one or more other examples to replace the same or similar features in those other examples, or to additionally introduce those features into other examples.

[0092] Reference Symbol List

[0093] 110 samples

[0094] 122 sensors

[0095] 130 device

[0096] 132 interface

[0097] 134 processor

[0098] 136 storage devices

[0099] 200 Methods for Optical Imaging Systems

[0100] 210 Obtain data from the first sensor and data from the second sensor.

[0101] 212 Convert the first sensor data

[0102] 220 spectral signal proportional to material concentration

[0103] 222 Combined reflection vector and fluorescence vector

[0104] 230 Mixed signal proportional to material concentration

[0105] 232 Spectral Demixing

[0106] Concentration of 240 materials

[0107] 300 Methods for Optical Imaging Systems

[0108] 310 acquires sensor data

[0109] 320 Converter Sensor Data

[0110] 330 fluorescence spectral data were obtained

[0111] 340 combined reflectance spectral data

[0112] 350 Determine the concentration distribution of the sample

[0113] 400 system

[0114] 410 microscope

[0115] 420 Computer System

Claims

1. An apparatus (130) for an optical imaging system, the apparatus (130) comprising one or more processors (134) and one or more storage devices (136), wherein the apparatus (130) is configured to: obtain sensor data indicative of reflectance measurements of a sample (110); convert the sensor data into reflectance spectral data indicative of a reflectance spectral signal, wherein the reflectance spectral signal is a function of a concentration distribution of the sample (110); obtain fluorescence spectral data of a fluorescence spectral signal indicative of fluorescence measurements of the sample (110); combine the reflectance spectral data and the fluorescence spectral data into combined spectral data; and determine the concentration distribution of the sample (110) based on the combined spectral data.

2. The apparatus (130) of claim 1, wherein the apparatus (130) is configured to: determine the reflectance spectral data using a logarithmic function.

3. The apparatus (130) of any one of the preceding claims, wherein the apparatus (130) is configured to: determine the combined spectral data by combining the reflectance spectral data and the fluorescence spectral data into a single vector.

4. The apparatus (130) of any one of the preceding claims, wherein the apparatus (130) is configured to: determine the concentration distribution using spectral unmixing.

5. The apparatus (130) of claim 4, wherein the apparatus (130) is configured to: perform spectral unmixing using at least one of a linear transformation, independent component analysis, orthogonal subspace projection, least volume simplex analysis, convex cone analysis, or total least squares.

6. An optical imaging system, comprising: the apparatus (130) of any one of the preceding claims.

7. The optical imaging system, further comprising: a first sensor for acquiring the sensor data; a second sensor for acquiring second sensor data; and wherein the apparatus (130) is configured to: control the first sensor and the second sensor such that the sensor data and the second sensor data are acquired simultaneously.

8. A method (300) for an optical imaging system, comprising: obtaining (310) sensor data indicative of reflectance measurements of a sample; converting (320) the sensor data into reflectance spectral data indicative of a reflectance spectral signal, wherein the reflectance spectral signal is a function of a concentration distribution of the sample; obtaining (330) fluorescence spectral data of a fluorescence spectral signal indicative of fluorescence measurements of the sample; combining (340) the reflectance spectral data and the fluorescence spectral data into combined spectral data; and determining (350) the concentration distribution of the sample based on the combined spectral data.

9. A computer program having a program code for performing the method (300) of claim 8 when the computer program is executed on a processor.

Images