Computer-implemented method and data processing device for extracting at least one signal in region of image using crosstalk quantity

Abstract

To provide a data processing device that accurately separates at least one signal from at least one other signal contained in a digital color input image.SOLUTION: A computer-implemented method provided herein comprises: extracting an appropriate subset 118 from a plurality of pixels 102 of a digital color input image 100; extracting, from the subset, a preliminary estimate of the first signal 104 as a first unmixed signal by spectral unmixing, and a preliminary estimate of at least one further signal 106, 108 of a plurality of signals as at least one further unmixed signal; and computing an estimate of a dependency of the first unmixed signal on the at least one further signal as a crosstalk quantity for the subset 118.SELECTED DRAWING: Figure 1B

Description

[Technical Field]

[0001] The present invention generally relates to image processing using computer-implemented methods and / or data processing devices. The present invention further relates to optical observation devices, such as microscopes or endoscopes, that include data processing devices and methods of using such optical observation devices, including computer-implemented methods. [Background technology]

[0002] Image processing often requires the extraction of one or more signals from an image, particularly a digital color image. These signals can correspond, for example, to the fluorescence of different fluorophores in an object, such as a biological tissue, e.g., a cell, a portion of a cell, or a portion of the body of an organism, and / or to the fluorescence of a fluorophore in a different chemical environment. Fluorophores are often used as markers that accumulate in areas of biological tissue with specific properties. Other fluorophores are transported in a container and are used to mark passages. Some fluorophores are naturally present in the object under observation. The type of biological tissue labeled depends on the chemical properties of the fluorophore. Other signals can be generated by the transmission of light through or reflection from the object, rather than by fluorescence.

[0003] More than one fluorophore may be present in a given area of ​​biological tissue, and different fluorophores may be contained in distinct areas that overlap along the optical axis and therefore appear to be in the same part of the object.

[0004] In each of these cases, the signals may overlap not only spatially but, more importantly, spectrally and must be separated from one another as accurately as possible. Different fluorophores may have overlapping fluorescence emission spectra, or the fluorescence emission spectrum of a fluorophore may overlap with the fluorescence excitation spectrum of this or another fluorophore. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, there is a need to provide a computer-implemented method and data processing device that accurately separates at least one signal from at least one other signal contained in a digital color input image. [Means for solving the problem]

[0006] The above need is addressed by providing a computer-implemented method for calculating an estimate of a first signal of a plurality of signals, the plurality of signals being included in a digital color input image, each signal of the plurality of signals having a different ground truth spectrum, the digital color input image comprising a plurality of pixels, the computer-implemented method comprising: extracting an appropriate subset of the plurality of pixels from the digital color input image; extracting from the subset by spectral decomposition a preliminary estimate of the first signal as a first decomposed signal and a preliminary estimate of at least one further signal of the plurality of signals as at least one further decomposed signal; calculating an estimate of the dependence of the first decomposed signal on the at least one further signal from the subset as a crosstalk amount for the subset; and removing the crosstalk amount from the first decomposed signal in the subset to obtain an estimate of the first signal in the subset.

[0007] The above needs are also addressed by a data processing device for calculating an estimate of a first signal of a plurality of signals, the plurality of signals being comprised in a digital color input image, each signal of the plurality of signals having a different ground truth spectrum, the digital color input image comprising a plurality of pixels, the data processing device being configured to: retrieve a suitable subset of the plurality of pixels of the digital color input image; extract from the subset by spectral decomposition a preliminary estimate of the first signal as a first decomposed signal and a preliminary estimate of at least one further signal of the plurality of signals as at least one further decomposed signal; calculate an estimate of the dependence of the first decomposed signal on the at least one further signal as a crosstalk amount for the subset; and remove the crosstalk amount from the first decomposed signal to obtain an estimate of the first signal in the subset.

[0008] In a digital color input image, signals are superimposed on one another and can only be separated by calculation. The computer-implemented method and data processing device described above improve the accuracy of the estimate of the first signal by using a crosstalk amount, i.e., the crosstalk amount of the decomposition signals. The crosstalk amount reflects the dependency of the first decomposition signal on at least one further decomposition signal. This is an improvement over known spectral decomposition methods. Furthermore, dividing the digital color input image into appropriate subsets and calculating the crosstalk amount only for the subsets rather than for the entire image constitutes a further improvement.

[0009] The above method can be further improved by the following features, which are mutually independent, can be combined with each other in any combination, and have their own technical effect: These features always apply to both the method and the device, regardless of whether they are described in the context of the method or the device.

[0010] In one example, the digital color input includes or can be generated using multiple color channels, or equivalently, color space coordinates. The digital color input image can be represented in any color space, and may be, for example, a tristimulus image, a multispectral image, or a hyperspectral image. Each color channel can be considered a (monochrome) image in itself. The same applies to the pixels of the digital color input image; the pixels can be color pixels represented in any color space and / or multispectral or hyperspectral pixels. Thus, each pixel can include multiple color channels. In the subset, at least two signals of the plurality of signals can include at least one identical color space coordinate.

[0011] Each signal is represented by a color space coordinate, or more precisely, a tuple of color space coordinates. For example, in the RGB color space, each signal is represented by three color space coordinates that represent the intensity of the signal in each of the red, green, and blue color channels. When more than one signal is present at a subset, or more specifically, at an image location such as a pixel, the color space coordinates reflect the superposition of these signals.

[0012] Preferably, the number of color channels or color space coordinates of the digital color input image is at least as large as the number of signals to be extracted or estimated.

[0013] The appropriate subset extracted from the digital color input image can correspond to a connected, but not necessarily simply connected, region of the digital input image. The subset may also correspond to a non-connected region of the digital color input image. The subset may consist of one or more pixels. The pixels of the subset may be contiguous and / or non-contiguous.

[0014] In one embodiment, all pixels of a subset may have at least one color appearance parameter that is within the same predetermined, preferably user-controllable range. Pixels of different subsets may have at least one color appearance parameter that is within a different predetermined, preferably user-controllable range. The color appearance parameter may include any of hue, lightness, luminance, chroma, colorfulness, and saturation.

[0015] Alternatively, or cumulatively, the pixels of the subset may be located in a predetermined neighborhood, i.e., within a certain maximum distance from each other. The accuracy of the estimation can be improved by collecting the pixels in the subset according to location and / or at least one color appearance parameter. For example, the subset can include pixel blocks of a predetermined, but preferably user-controllable size, e.g., 2x2, 3x3, 2x3, 4x4, etc.

[0016] The digital color input image can be considered as a digital multiplexed color input image that includes, in addition to the first signal, one or more additional signals, each of which has a respective ground truth spectrum that is different from the ground truth spectrum of all other signals, including in particular the first signal. The term "ground truth" refers to a signal (or its spectrum) that would be recorded under an ideal, artifact-free recording process. Thus, although the signals in the digital color input image represent the respective ground truth signals, they are only approximations of the respective ground truth signals.

[0017] One application of the data processing device and / or computer-implemented method is to process fluorescence images. For example, the first signal can correspond to a first fluorescence signal having a first spectrum. The first signal can be generated by or represent the fluorescence emission of a first fluorophore. The first signal can have a first spectrum representing the first fluorescence emission spectrum, i.e., the fluorescence emission spectrum of the first fluorophore.

[0018] The multiple signals in the digital color input image can include two or at least two signals, for example, three or four signals. Some or all of the signals can represent the fluorescence emission of different fluorophores, each with a different fluorescence spectrum. Furthermore, some or all of the signals can represent different fluorescence spectra of the same fluorophore in different chemical environments. The fluorescence spectra of some fluorophores change significantly depending on the chemical environment.

[0019] According to another embodiment, the plurality of signals can include one or more signals representing an illumination spectrum of the object under observation. The illumination spectrum can consist of or include a fluorescence excitation spectrum of a fluorophore whose fluorescence is represented by one or more signals. According to another embodiment, the illumination spectrum can include, be contained within, or overlap with a fluorescence emission spectrum of a fluorophore represented by one of the plurality of signals. According to yet another embodiment, the illumination spectrum can include, be contained within, or overlap with a fluorescence excitation spectrum of a fluorophore represented by one of the plurality of signals. One, some, or all of the signals can represent a transmittance spectrum, and one, some, or all of the signals can represent a reflectance spectrum.

[0020] The extraction of the subset, subsequent extraction of the first decomposition signal and at least one further decomposition signal, calculation of the crosstalk amount for the subset from the subset, and removal of the crosstalk amount from the first decomposition signal to obtain an estimate of the first signal in the subset may be performed repeatedly for different subsets until the combination of extracted subsets covers a predetermined portion, in particular all, of the digital color input image.

[0021] In such an embodiment, it is preferable that the different subsets do not overlap, however, if the subsets do overlap, an average amount of crosstalk can be calculated for the overlapping subsets.

[0022] In a preferred embodiment, each subset contains only a single pixel. The amount of crosstalk is then calculated for each pixel individually as described above and / or below and removed from the preliminary estimate of the decomposed signal. In this way, all pixels of the digital color input image, or all pixels of a predetermined, preferably user-selectable area of ​​the digital color input image, can be processed, and these pixels do not need to be connected.

[0023] Because the separation signals are represented by color space coordinates in the color space of the digital color input image, the amount of crosstalk can also be represented in color space coordinates, i.e., as a tuple of color space coordinates. Each color space coordinate of the crosstalk amount represents the dependence of a color space coordinate of a preliminary estimate of one signal on the same color space coordinate of a preliminary estimate of another signal. If the color space of the digital color input image is not appropriate for calculating the estimate, a conversion to another color space can be performed and all calculations can be performed in that color space.

[0024] The estimates of the first signal (and any further signals) do not necessarily have to be the best estimates in the mathematical sense. From an application point of view, it may be preferable to set the estimates to zero if the amount of crosstalk exceeds a certain threshold, since in this case the estimates may be too unreliable to serve as a basis for further post-processing.

[0025] The amount of crosstalk depends on the subset and may vary from one subset to another.

[0026] In the following, a crosstalk amount representing an estimate of the dependence of the first decomposed signal on at least one further signal is referred to as a "first crosstalk amount" if the additional crosstalk amount is calculated for at least one further signal.

[0027] As mentioned above, a crosstalk measure represents the influence of one or more other signals on another signal. Accordingly, multiple different crosstalk measures may be calculated for the subset. Each of these crosstalk measures preferably represents the dependency of a preliminary estimate of one signal on a preliminary estimate of another signal.

[0028] According to a further aspect, extracting a preliminary estimate of at least one further signal of the plurality of signals from the subset by spectral decomposition as at least one further decomposed signal may include extracting a preliminary estimate of a second signal of the plurality of signals from the subset by spectral decomposition as a second decomposed signal, wherein the at least one further signal corresponds to the second signal. An estimate of the dependency of the second decomposed signal on the first decomposed signal may be calculated as a second crosstalk amount. The second crosstalk amount may be removed from the second decomposed signal to obtain an estimate of the second signal in the subset.

[0029] This aspect allows both estimates of the first signal and the second signal to be extracted with improved accuracy. The second signal can correspond to a second fluorescent signal having a second spectrum, which represents a second fluorescent emission spectrum. However, it is not necessary for the first and second signals to correspond to fluorescent signals; instead, they may correspond to reflectance and / or fluorescent excitation signals of different fluorophores.

[0030] For example, the first signal can represent a first fluorescence emission spectrum of a first fluorophore, and the second signal can represent a second fluorescence emission spectrum of a second fluorophore, where the first fluorophore is different from the second fluorophore. Alternatively, the first signal can represent a first fluorescence emission spectrum from the first fluorophore in a first chemical environment, and the second signal can represent a second fluorescence emission spectrum from the first fluorophore in a second chemical environment, where the second chemical environment is different from the first chemical environment. Alternatively, the first signal can represent a fluorescence emission spectrum, and the second signal can represent a fluorescence excitation spectrum. The object of interest can be illuminated with light consisting of or including a fluorescence excitation spectrum to trigger fluorescence emission.

[0031] Similar to the above step, an estimate of the third decomposed signal in the subset can be calculated using the further crosstalk amount.

[0032] For example, extracting a preliminary estimate of at least one further signal of the plurality of signals by spectral decomposition from the subset as at least one further decomposed signal may include extracting a preliminary estimate of a third signal of the plurality of signals by spectral decomposition from the subset as a third decomposed signal. A third crosstalk amount may be calculated, which is an estimate of the dependency of the third decomposed signal on the first decomposed signal in the subset, and / or a fourth crosstalk amount may be calculated, which is an estimate of the dependency of the third decomposed signal on the second decomposed signal in the subset. The third crosstalk amount and / or the fourth crosstalk amount may be removed from the third decomposed signal to obtain an estimate of the third signal in the subset. Furthermore, a fifth crosstalk amount may be calculated, which is an estimate of the dependency of the first decomposed signal on the third decomposed signal. The first crosstalk amount and the fifth crosstalk amount may be removed from the first decomposed signal to obtain an estimate of the first signal in the subset. This further improves the accuracy of the estimate of the first signal.

[0033] To improve the accuracy of the second signal, a sixth crosstalk amount can be calculated, which is an estimate of the dependency of the second decomposed signal from the third decomposed signal. The second crosstalk amount and the sixth crosstalk amount can be removed from the second decomposed signal to obtain an estimate of the second signal in the subset.

[0034] The third signal can represent a third fluorescence signal of a third fluorophore different from the first and second fluorophores. Like the second signal, the third signal can represent the fluorescence emission spectrum or fluorescence excitation spectrum, or reflected or transmitted light, of the first, second, or third fluorophore. According to another embodiment, the third signal can represent the third fluorescence emission spectrum of the third fluorophore or the first fluorophore in a third chemical environment different from the first or second chemical environment.

[0035] The digital color input image can represent a color image of the fluorescent object under observation, which can be primarily composed of biological material or tissue. The colors represented in the digital color input image can extend beyond the human visible spectrum, i.e., can include wavelengths below and / or above the visible spectrum, i.e., wavelengths below 380 nm and / or above 750 nm. The digital color input image preferably represents (or is synonymously recorded as) an imaging spectrum that overlaps with, is included in, or includes the fluorescence spectrum of at least one fluorophore contained in the object under observation. This ensures that the fluorescence of interest is captured.

[0036] In one embodiment, the step of extracting by spectral decomposition may include one of extracting by linear spectral decomposition and extracting by nonnegative tensor factorization. Both linear spectral decomposition and nonnegative tensor factorization often provide accurate results while not imposing a heavy computational burden. In this context, a "matrix" is considered to be a special two-dimensional case of a tensor. In other words, a tensor is an n-dimensional matrix, where n>1.

[0037] In particular, the step of extracting by spectral decomposition may include solving a set of equations that represent a linear superposition of multiple signals in the subset, which may be solved using, for example, a minimization routine from a software library.

[0038] The ground truth spectra of the first signal, if applicable, the second signal, the third signal, and any additional signals are preferably used as end members in the spectral decomposition. The contribution of each end member to the subset or to the total spectrum recorded in the digital color input image, respectively, is referred to as the abundance. The end members can be determined experimentally, for example, by measuring with very high precision the fluorescence emission spectra of fluorophores in the color space of the digital color input image.

[0039] In spectral decomposition, particularly linear spectral decomposition, extracting preliminary estimates of the first, if applicable, second, third, and / or additional signals included in the subset may include calculating an estimate of a mixing tensor for the subset. The mixing tensor maps the ground truth of each first, if applicable, second, third, and / or additional signal to each first, if applicable, second, third, and / or additional signal in the subset. As described above, signals in the subset are represented by color channel coordinates. Therefore, strictly speaking, the mixing tensor maps the ground truth of a signal to its (recorded) representation in color space. The mixing tensor is calculated separately for each subset.

[0040] In a further embodiment, extracting a preliminary estimate of the first signal, and if applicable the second signal, the third signal, or further signals from the subset may comprise calculating an estimate of the mixture tensor using minimization. In one example, the estimate of the mixture tensor is n Furthermore, the calculated estimate of the mixing tensor can minimize the deviation of one, some, or all signals contained in the subset from the linear transformation of their end members by the estimated mixing tensor. The deviation is n It can be expressed by a norm, where n is a natural number.

[0041] The diagonal elements of the crosstalk tensor represent the contribution of each ground truth signal in the subset. For example, the first diagonal element of the crosstalk tensor may correspond to the contribution of a first signal in the subset, and the second diagonal element of the crosstalk tensor may correspond to the contribution of a second signal in the subset. The diagonal elements of the crosstalk tensor corresponding to signals representing the fluorescence emissions of fluorophores indicate the concentration of each fluorophore at the location of the object under observation, which location is mapped to the subset.

[0042] In another embodiment, the step of calculating the crosstalk amount may include calculating a probability that the first decomposed signal, the second decomposed signal, the third decomposed signal and / or the further decomposed signal corresponds to the respective first signal, the second signal, the third signal and / or the further signal. In a particular embodiment, the crosstalk amount may be calculated from a mixture tensor, in particular from off-diagonal elements of the mixture tensor.

[0043] Alternatively or cumulatively, the step of calculating the amount of crosstalk may include calculating the correlation or covariance between one signal and another signal. Covariance can be calculated from correlation.

[0044] In particular, the crosstalk amount may be calculated as the correlation or covariance between one signal of the plurality of signals in the subset and another different signal of the plurality of signals in the subset. That is, the crosstalk amount is calculated from the correlation or covariance between a signal of the plurality of signals in the subset from which the crosstalk amount has been removed and another signal of the plurality of signals. In another approach, the covariance may be calculated using a Bayesian estimator. For example, the first crosstalk amount may be calculated from the correlation or covariance between the first signal and the third signal. This applies mutatis mutandis to the second through sixth crosstalk amounts and any further crosstalk amounts.

[0045] Because matrix calculations are relatively computationally efficient, it may be preferred that the step of calculating the crosstalk amounts comprises calculating a crosstalk tensor whose elements, particularly the off-diagonal elements, are the crosstalk amounts.

[0046] The dimension of the crosstalk tensor may be determined by the number of decomposition signals from which the crosstalk tensor is calculated. For example, if the crosstalk tensor is calculated from two decomposition signals, the crosstalk tensor may have dimensions of 2×2. If the crosstalk tensor is calculated from three decomposition signals, the crosstalk tensor may have dimensions of 3×3, and so on. Here, each element of the crosstalk tensor may include all color space coordinates in the color space of the digital input image.

[0047] Calculating the crosstalk amount may include calculating the crosstalk amount from off-diagonal elements of the crosstalk tensor. In certain cases, the crosstalk amount may correspond to the off-diagonal elements of the crosstalk tensor.

[0048] In a variant, the step of calculating the crosstalk tensor and / or crosstalk amount may include calculating at least one of calculating a covariance tensor and calculating a Fisher information tensor from the decomposed signals.

[0049] Calculating the covariance tensor may include calculating the covariance tensor from weighted estimates of the mixture tensor. The weights may depend on the subset, i.e., may be different for different subsets. For example, the weights may depend on preliminary estimates of the first signal, the second signal, the third signal, and / or any additional signals. The weights may consist of the average of the preliminary estimates of the signals in the subset, i.e., a vector that separately weights each column of the mixture tensor. The weights may also consist of the median of the preliminary estimates of the signals in the subset. The weights may be a scalar calculated from the preliminary estimates of the signals in the subset, e.g., a global average of all preliminary estimates of the signals in the subset.

[0050] To improve the decomposition signals in the subset, the crosstalk amount can be removed in various ways. For example, removing the crosstalk amount from the decomposition signals can include one of subtracting the crosstalk amount from the decomposition signals and multiplying the decomposition signals by the crosstalk amount. The crosstalk amount is preferably expressed in a decomposition signal space, i.e., independent of any color space. The signal space is formed by using end members as "coordinates."

[0051] The amount of crosstalk can be weighted additively and / or multiplicatively by a preferably user-controllable weighting parameter, which may be a scalar and in particular may be constant for all signals in the subset.

[0052] The amount of crosstalk may depend on or correspond to the off-diagonal elements of the covariance tensor. In a particular example, the amount of crosstalk for a decomposed signal may be calculated using the sum of the off-diagonal elements of the covariance tensor that represent the correlation between this decomposed signal and other decomposed signals. Weighting parameters may be applied to this sum.

[0053] The crosstalk amount can be subtracted directly from each decomposed signal. If the result of the subtraction is negative, all pixels in the subset can be set to zero. If a single signal is negative in signal space after subtraction of the crosstalk amount, all pixels in the subset can be set to zero.

[0054] Another form of crosstalk measure can be derived, for example, by using an optionally weighted sum of the off-diagonal elements of the covariance matrix as the negative exponent of a number. Such a crosstalk measure represents a confidence value that decreases exponentially with the sum of the off-diagonal elements of the covariance tensor. Such a crosstalk measure can be used to multiply each signal in the subset. Thus, a decomposed signal becomes increasingly attenuated if it is highly correlated with other decomposed signals.

[0055] The digital color input image can be retrieved from storage, such as memory, of the data processing device and / or the optical observation device. The storage can be in the cloud, on disk, or in any other non-volatile memory. Alternatively, the digital color input image can be retrieved directly from a camera of the optical observation device.

[0056] According to another embodiment, separate digital color output images may be generated from estimates of the first signal and, if applicable, the second signal and / or any further signals, or alternatively, all estimates may be combined into a single digital color output image.

[0057] The subset of the digital color input image used to calculate the estimates of the first signal and, if applicable, any further signals is preferably co-located in the digital color output image as the digital color input image, and the digital color input image and the digital color output image are preferably aligned with respect to each other.

[0058] The estimates of the first signal and / or any further signals, and / or the digital color output image may be presented, for example in a memory of the data processing device, in particular for external access to the digital processing device, which may be used, for example, to display the estimates and / or the digital color output image.

[0059] The invention also relates to a computer program comprising instructions for causing a computer to carry out the method described in any of the above embodiments when the program is executed by a computer.

[0060] Furthermore, the invention relates to a computer readable medium having stored thereon such a computer program.

[0061] As already mentioned above, the data processing device can be configured to perform any of the steps or embodiments described above.

[0062] The computer-implemented method of any of the above embodiments may be part of a method of using an optical observation device, such as a microscope or an endoscope. The microscope may be a laboratory microscope or a surgical microscope. Further, the microscope may be a fluorescent microscope, such as a fluorescent surgical microscope or a fluorescent laboratory microscope. The method of using the optical observation device may further include recording at least one color image of the object under investigation and generating a digital color input image from the at least one color image.

[0063] The step of generating the color input image from the at least one color image may include computing the digital color input image from a plurality of different color images, such that the different color images can be combined into a single digital color input image.

[0064] In its simplest form, the digital color input image can come directly from a camera.

[0065] The optical observation device may comprise at least one camera, in particular at least one fluorescence camera, for recording the first signal and any further signals. The at least one camera of the optical observation device may be configured to record at least one color image, and the optical observation device may be configured to generate a digital color input image from the at least one color image, as described above.

[0066] In an advantageous embodiment, the data processing device may comprise or consist of an embedded system of the optical observation device.

[0067] The embedded system may be configured to control an optical observation device, e.g., control an actuator, for example to move an object under observation, for autofocus, for autotracking, to shift a lens to change focus and / or focal length, to shift a microscope table, to control illumination of an object under observation, such as illumination spectrum and / or illumination brightness.

[0068] The optical observation device can include a hardware and / or software graphical user interface. The graphical user interface can include an interaction area designed for user input. For example, user-controllable parameters used in the data processing device and / or computer-implemented method in any of the above embodiments can be changed in response to user interaction with the interaction area.

[0069] The interaction area may alternatively or cumulatively be used to control an optical observation device.

[0070] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0071] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0072] In the following, a proposed solution to the above problem will be exemplarily described using one embodiment of the present invention with reference to the drawings, in which like reference numerals are used for elements that correspond to one another in terms of function and / or physical structure.

[0073] Furthermore, in accordance with the above discussion, an element described below in the context of an embodiment may be omitted from that embodiment if its technical effect is not required for a particular application. Conversely, an element described above but not in the context of the following embodiment may be added to an embodiment if the technical effect associated with that element is advantageous in a particular application. [Brief explanation of the drawings]

[0074] [Figure 1A] 1 is a schematic diagram of an object under observation including a signal source; [Figure 1B] 1B is a schematic diagram of a digital color input image of the object shown in FIG. 1A. [Figure 2] FIG. 2 is a schematic diagram of a method for calculating the signals shown in FIGS. 1A and 1B. [Figure 3] FIG. 1 is a schematic diagram of the effect of crosstalk. [Figure 4] FIG. 2 is a schematic diagram of an optical observation device. DETAILED DESCRIPTION OF THE INVENTION

[0075] FIG. 1A shows an object 120 under investigation by an optical viewing device 134, such as a laboratory or surgical microscope or endoscope.

[0076] The object 120 preferably comprises or consists of living tissue. For example, the object 120 may be a cell, a part of a cell such as an organelle, or a part of the body of a living organism such as an animal or plant.

[0077] The object 120 includes one or more sources 122, 124, 126 of ground truth signals 128, 130, 132 that an optical observation device 134 is configured to record. The ground truth signals 128, 130, 132 are generated by at least one source 122, 124, 126 in the form of light that includes, overlaps, or can be limited to the visible spectrum. The light sources 122, 124, 126 may be, for example, fluorophores that fluoresce with a fluorescence emission spectrum. The fluorescence emission spectrum may vary for different fluorophores and / or depend on the chemical environment in which the fluorophore is located. The fluorophores may be artificially added to the object 120 or may be naturally present within the object 120.

[0078] Alternatively or cumulatively, light sources 122, 124, and 126 may be light-reflecting or light-transmitting materials, such that object 120 is illuminated with a particular transmittance spectrum and / or reflectance spectrum, respectively. Alternatively or cumulatively, light sources 122, 124, and 126 may be areas illuminated with a transmitted illumination spectrum, a reflected illumination spectrum, and / or an illumination spectrum that triggers fluorescence. Any number of sources may be included in object 120. Three sources are selected as an example only for ease of understanding.

[0079] Ground truth signals 128, 130, 132 generated by at least one source 122, 124, 126 are recorded by optical observation device 134 as signals 104, 106, 108 in digital color input image 100, as shown in FIG. 1B. Any number of signals may be represented in digital color input image 100. However, the number of color channels, or equivalently, the number of color space coordinates, is preferably at least as large as the number of signals or sources recorded and distinguished from one another in digital color input image 100.

[0080] The digital color input image 100 is a color image, such as a tristimulus image, a multispectral image, or a hyperspectral image. The digital color input image 100 contains multiple color channels, or, equivalently, color space coordinates. The digital color input image 100 can be represented in any color space. Each color space defines a tuple of color space coordinates that define a color. Different color space coordinates can be transformed between each other; however, the transformations are not necessarily bijective.

[0081] The digital color input image 100 may be generated from multiple individual color images that may have been recorded sequentially by a single camera of the optical observation device 134 and / or in parallel by two or more cameras.

[0082] A digital color input image 100 includes a number of pixels 102. Each pixel 102 includes a tuple of color space coordinates that determine the color of the pixel 102. Each area of ​​an object 120 is mapped to a different region.

[0083] The digital color input image 100 may be divided into suitable subsets 118, each containing at least one pixel. Each subset 118 represents an area of ​​the digital color input image 100, and thus a region of the object 120. The subsets 118 may be connected, but not necessarily simply connected, regions of the digital color input image 100, or may be unconnected regions. Each subset 118 contains at least one pixel. In the simplest case, each subset 118 contains only a single pixel. The subsets 118 can be processed individually when processing the digital color input image 100. Reference numerals 118a and 118b denote different subsets 118, each containing at least one pixel 102.

[0084] The at least one signal 104, 106, 108 recorded in the digital color input image 100 is only an approximation of the respective at least one ground truth signal 128, 130, 132 generated by the at least one source 122, 124, 126. This is because the recording process always introduces artifacts such as noise, distortion, and other linear and / or non-linear errors.

[0085] 1B , a first signal 104 is recorded in the digital color input image 100 as representing a ground truth signal 128 generated by a source 122. Additionally, a second signal 106 is now recorded in the digital color input image 100 as representing a ground truth signal 130 generated by a second source 124. A further signal, here a third signal 108, may be recorded in the digital color input image 100 as representing a ground truth signal 132 generated by a third source 126.

[0086] In some or all areas of the object 120, two or more sources 122, 124, 126 may be present. In such areas, two or more ground truth signals 128, 130, 132 are generated. As a result, the region of the digital color input image 100 to which this area of ​​the object 120 is mapped contains two or more signals 104, 106, 108. For example, the first signal 104 and the second signal 106 overlap in region 110 of the digital color input image 100, the first signal 104 and the third signal 108 overlap in at least one region 112 of the digital color input image 100, the second signal 106 and the third signal 108 overlap in at least one region 116 of the digital color input image 100, and all three signals 104, 106, 108 overlap in region 114 of the digital color input image 100. In regions 112, 114, 116, each color space coordinate results from the superposition of at least two signals 104, 106, 108 and artifacts.

[0087] Due to physical processes occurring in the object 120 and / or the recording process, the signals 104, 106, 108 in the digital color input image 100 may be affected by at least one other signal 104, 106, 108. For example, the signals 104, 106 may exhibit linear or nonlinear dependencies on one another that may be unknown a priori. The same may be true for any other combination of the signals 104, 106, 108.

[0088] To separate the signals 104, 106, and 108 from one another, linear or nonlinear decomposition is applied, depending on whether linear or nonlinear dependencies of the signals 104, 106, and 108 are assumed. Spectral decomposition is applied to each subset 118 separately, i.e., each subset 118 is processed individually. Spectral decomposition can be performed on multiple subsets 118 until a desired portion of the digital color input image 100 has been processed. The portion of the digital color input image 100 that is processed may be user-controllable. The processed subsets 118 may or may not overlap. Due to reduced numerical effort, it may be preferable for the subsets 118 to not overlap.

[0089] When spectral decomposition is applied to a digital color input image 100, preliminary estimates 210, 212, 214 of the decomposed signals, ie, signals 104, 106, 108, are obtained.

[0090] The subset 118 can be defined by pixels 102 that share a predetermined range, preferably a user-controllable range, of at least one color appearance parameter, such as hue, lightness, brightness, chroma, colorfulness, and saturation. For example, the subset 118 can be defined by pixels 102 whose hue falls within a predetermined range of values ​​and, at the same time, whose brightness also falls within a predetermined range of values.

[0091] Additionally or cumulatively, the pixels 102 in a subset 102 may satisfy geometric constraints, such as not being located closer to each other than a predetermined, preferably user-controllable, distance, or not covering an area larger than a predetermined, preferably user-controllable, area of ​​the digital color input image 100. In a very simple embodiment, the subset may comprise a square arrangement of pixels, such as 2x2 or 3x3.

[0092] In the following, the determination of improved estimates of the signals 104, 106, and 108 is exemplarily described using linear spectral decomposition. In this model, the signals 104, 106, and 108 recorded in the digital color input image 100 are assumed to arise from ground truth signals 128, 130, and 132 generated by sources 122, 124, and 126 by linear transformations. The linear transformations are then transformed into the unknown mixture tensor M cd By

number

[0093] where y cp denotes the color space coordinate c of the recorded signal y in the particular subset 118 with index p. Index d represents the index of the ground truth signals 128, 130, 132. d is the total number of ground truth signals. d represents the ground truth signals 128, 130, 132 with index d. The index d can be considered to represent the different sources 122, 124, 126. The mixture tensor is of dimension N c ×N d N c is the number of color space coordinates of the color space in which the spectral decomposition is performed.

[0094] Ground truth signal x dp represents the spatial distribution of fluorophore concentrations, for example, of fluorophore d across the various subsets p.

[0095] The above equations can be solved for the unknown mixture tensor M, for example by maximum likelihood estimation, in particular by blind decomposition, or by linear decomposition, or by non-negative tensor factorization.

[0096] For example, as described in Zimmermann, Timo: "Spectral Imaging and Lieanr Unmixing in Light Microscopy", Adv Biochem Engin / Biotechnol (2005) 95:245-265, the mixture tensor can be calculated entirely from a priori knowledge. The a priori knowledge can include the spectral signature of each ground truth signal, which can be experimentally measured. Such a spectral signature can be, for example, the fluorescence emission spectrum of a fluorophore or the illumination spectrum of a particular light source.

[0097] Using minimization, estimate the unknown mixture tensor M

number

number

[0098] an estimate of the unknown mixture tensor M

number

number

number

number

number

[0099] Decomposition Signal

number

[0100] The effect of crosstalk, or equivalently the dependency of one decomposed signal on another, is explained with reference to FIG. 3 (and FIG. 5).

[0101] The effect of crosstalk is shown diagrammatically in Figure 3. Along the vertical axis, the intensity I 104 Along the horizontal axis, the intensity I of the second signal 106 in the digital input image 100 increases. 106 Each measurement point 300 in this diagram represents the measured signal intensity in a different subset 118 of the digital color input image 100.

[0102] Reference numeral 302 indicates the range of intensities in subset 118 where only signal 104 is present, and therefore the intensity can be attributed to signal 104. Reference numeral 304 indicates the range intensity in subset 118 where only signal 106 is present, and the intensity can be attributed to signal 106. The ranges of intensities 302 and 304 indicate a spread that can be represented by statistical metrics 306 and 308, respectively. One such metric can be the standard deviation.

[0103] In a subset of the digital color input image 100 where both signals 104 and 106 are present, such as subset 118 within area 110 in FIG. 1B , the decomposed signals can exhibit a range of intensities 304. It can be seen that metric 310 of the spread of signal 104 in the presence of signal 106 is greater than metric 306 of signal 104 in the absence of the other signal. The difference between metric 306 and metric 310 is due to crosstalk or bleed from signal 106 to signal 104. The same applies mutatis mutandis to signal 106. Metric 312 of the spread of signal 106 in the presence of signal 104 is greater than metric 306 of signal 106 in the absence of the other signal. The difference between metrics 308 and 312 is due to crosstalk or bleed from signal 104 to signal 106. These differences are captured by the crosstalk quantity, which describes the dependence of one signal on another.

[0104] Crosstalk is not taken into account in the spectral decomposition. Therefore, when there is spectral overlap of the signals 104, 106, and 108, especially in the subset 118, the decomposed signal

number

number

number

[0105] The first decomposed signal in the subset 118 with index p

number

number

number

number

[0106] At least one crosstalk quantity may be determined using a statistical method, e.g., by comparing non-identical resolved signals.

number

[0107] In one example, a covariance tensor or matrix, or a Fisher information tensor or matrix, is used to represent the decomposed signal

number

[0108] The amount of crosstalk can also be calculated using a Bayesian estimator.

[0109] In another more specific example, the crosstalk matrix is 2The element C can be calculated from the covariance matrix through minimization. mn It can consist of or contain the element.

[0110] In addition to other known methods for calculating covariance, we consider the elements of the covariance tensor as

number

number

number

[0111] The crosstalk amount C of the subset 118 having the index p p To calculate σ, the covariance tensor may be weighted, preferably with a user-controllable weighting factor α, and its off-diagonal elements may be combined, for example by adding, i.e.

number

[0112] The crosstalk amount can be calculated, for example, by improving the estimate of the ground truth signals 128, 130, 132 in each subset 118.

number

number

[0113] If the above subtraction results in a negative value,

number

[0114] Alternatively, a different amount of crosstalk

number

number

[0115] The confidence values ​​represent a scalar normalized metric for the share of crosstalk in the overall intensity of the decomposed signal in each subset 118. The confidence values ​​cannot be negative, which is convenient for further calculations.

[0116] Using the confidence values ​​as the crosstalk measure, an improved estimate of the ground truth signal is obtained from the decomposed signal:

number

[0117] In the above formula, the amount of crosstalk is multiplied and removed from the decomposed signal. The decomposed signal is weighted by a confidence value. The larger the crosstalk, represented by the off-diagonal elements of the crosstalk tensor, the smaller the estimate.

[0118] FIG. 2 provides a schematic diagram of how a method 200 for enhancing decomposed signals 210, 212, 214 can be performed.

[0119] In a first step 202, a digital color input image 100 can be generated, for example, by at least one camera of the optical viewing device 134. Alternatively, an already generated digital color input image 100 can be retrieved directly from a camera or from a storage device.

[0120] In step 204, a spectral decomposition is performed on a subset 118 (FIG. 1B) of the digital color input image 100.

[0121] In step 206, the end members 208 may be retrieved, for example, from a storage device, and provided to the spectral decomposition process or step 204. As a result of the spectral decomposition, several decomposed signals 210, 212, ..., 214 are extracted from the subset 118. The decomposed signals 210, 212, 214 are

number

number

number

number

[0122] In step 216, a first crosstalk measure 218 and any further crosstalk measures 220 are calculated as described above. The number of crosstalk measures calculated in step 216 may depend on the number of decomposed signals extracted from each subset 118 in step 204. At least one crosstalk measure 218 may be calculated in response to an estimate of the mixture tensor 230 determined in step 204.

[0123] In step 222, the crosstalk measure 218 is calculated as an improved signal estimate 224, 226, i.e.,

number

[0124] In step 232, a weighting factor 234, α, used to calculate the amount of crosstalk is retrieved. For example, the weighting factor may be input by a user.

[0125] Post-processing of the signal estimates, such as displaying the signal estimates 224, 226 or combining the signal estimates 224, 226 into a single digital output image, may occur in step 228.

[0126] Some embodiments relate to a microscope including a system such as that described in connection with one or more of Figures 1 to 3 (and Figure 5). Alternatively, the microscope may be part of or connected to a system such as that described in connection with one or more of Figures 1 to 3 (and Figure 5). Figure 4 shows a schematic diagram of a system 400 configured to perform the methods described herein. The system 400 includes a microscope 410 and a computer system 420. The microscope 410 is configured to capture images and is connected to the computer system 420. The microscope 410 may include a data processing device 430, which may be an embedded processor of the microscope 410 or the system 400.

[0127] The computer system 420 is configured to perform at least some of the methods described herein. The computer system 420 may be configured to execute machine learning algorithms. The computer system 420 and the microscope 410 may be separate entities or may be integrated into a common housing. The computer system 420 may be part of a central processing system of the microscope 410 and / or the computer system 420 may be part of a subordinate component of the microscope 410, such as a sensor, actor, camera, or lighting unit of the microscope 410.

[0128] The computer system 420 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed across various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuitry that may be included in computer system 420 may be custom circuitry, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 420 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.Computer system 420 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touch screen, voice recognition device, or any other device that allows a user of the system to input information to and receive information from computer system 420.

[0129] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry. In some embodiments, any one or more of the critical steps may be performed by such an apparatus.

[0130] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed by a non-transitory storage medium, such as a digital storage medium, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Therefore, the digital storage medium may be computer-readable.

[0131] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

[0132] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code that is operative to perform any of the methods when the computer program product is run on a computer, and that may be stored, for example, on a machine-readable carrier.

[0133] Further embodiments comprise the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0134] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein when the computer program runs on a computer.

[0135] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a recording medium.

[0136] A further embodiment of the present invention is, therefore, a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0137] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0138] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0139] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0140] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus. [Explanation of symbols]

[0141] 100 digital color input images 102 pixels 104 First Signal 106 Further / Second Signal 108 Further Signal / Third Signal 110 Overlapping Area 112 overlapping areas 114 Overlapping Area 116 Overlapping Area 118 subset 120 Objects to be observed 122 First (ground truth) signal source 124 Second (ground truth) signal source 126 Third (ground truth) signal source 128 ground truth signals 130 Ground Truth Signals 132 Ground Truth Signals 134 Optical Observation Device 200 ways 202 Generate / Generate a digital color input image 204 Spectral Decomposition 206 Remove the end member 208 End Member 210 First decomposition signal / Preliminary estimate of first signal 212 Second decomposition signal / Preliminary estimate of second signal 214 Further decomposition signal / preliminary estimate of further signal 216 Calculating the amount of crosstalk 218 (First) Crosstalk Amount 220 More crosstalk 222 Removing Crosstalk 224 First signal estimate 226 Further Signal Estimates 228 Post-processing 230 Mixed Tensor / Matrix 232 Extracting weight coefficients 234 Weighting Factor 300 measurement points 302 Intensity distribution when only the first signal is present 302 Intensity distribution when only the second signal is present 304 Intensity distribution when both are present 306 Distribution Spread Metrics300 308 Distribution Spread Metrics302 310 Metric of spread of first signal when both signals are present 312 Metric of the spread of the second signal when both signals are present 400 Optical Observation Devices / Systems 410 Microscope 420 Data Processing Devices / (General-Purpose) Computers 430 Embedded Systems I Signal Strength λ wavelength

Claims

1. A computer-implemented method (200) for calculating an estimate (224) of a first signal (104) of a plurality of signals, comprising: the plurality of signals are included in a digital color input image (100), each signal of the plurality of signals having a different ground truth spectrum (302, 308), the digital color input image (100) comprising a plurality of pixels (102); The computer-implemented method (200) comprises: - extracting a suitable subset (118) from said plurality of pixels (102); from said subset (118), by spectral decomposition, extracting a preliminary estimate of said first signal (104) as a first decomposed signal (210); extracting a preliminary estimate of at least one further signal (106, 108) of said plurality of signals as at least one further decomposed signal (212, 214); - calculating (216) from said subset (118) an estimate of the dependency of said first decomposed signal (210) on said at least one further signal (212, 214) as a crosstalk amount (218, 220) of said subset; - removing (222) said crosstalk quantities (218, 220) from said first decomposed signal (210) to obtain an estimate (224) of said first signal (104) for said subset (118); A computer-implemented method (200) comprising:

2. Extracting a preliminary estimate of at least one further signal (106, 108) of the plurality of signals from the subset (118) by spectral decomposition (204) as at least one further decomposed signal (212, 214) comprises: extracting from said subset (118) by spectral decomposition (204) a preliminary estimate of a second signal (106) of said plurality of signals as a second decomposed signal (212), The computer-implemented method (200) comprises: - calculating an estimate of the dependency of said second decomposed signal (212) on said first decomposed signal (210) as a second crosstalk amount (220) within said subset (118); - removing said second crosstalk amount (220) from said second decomposed signal (212) to obtain an estimate (226) of said second signal (106) in said subset (118); further comprising: The computer-implemented method (200) of claim 1.

3. The step of extracting by spectral decomposition (204) comprises: - extraction by linear spectral decomposition; - Extraction by non-negative tensor factorization; including either one of the The computer-implemented method (200) of claim 1 or 2.

4. The step of calculating (216) the crosstalk amount (218, 220) includes: - calculating the probability that said first decomposed signal (210) corresponds to said first signal (104), The computer-implemented method (200) of any one of claims 1 to 3.

5. The step of calculating the crosstalk amount (218, 220) includes calculating a correlation. The computer-implemented method (200) of any one of claims 1 to 4.

6. The step (216) of calculating the crosstalk amounts (218, 220) includes calculating a crosstalk tensor having the crosstalk amounts (218, 220) as elements. The computer-implemented method (200) of any one of claims 1 to 5.

7. The step of calculating the crosstalk tensor (216) comprises: - calculating the covariance tensor; - calculating the Fisher information tensor; at least one of: The computer-implemented method (200) of claim 6.

8. The step (222) of removing the crosstalk amounts (218, 220) from the decomposed signals (210, 212) comprises: - subtracting said crosstalk amounts (218, 220) from said decomposed signals (210, 212); multiplying said decomposed signals (210, 212) by said crosstalk quantities (218, 220); including either one of the The computer-implemented method (200) of any one of claims 1 to 7.

9. the subset (118) corresponds to a single pixel (102) of the plurality of pixels of the digital color input image (100); The computer-implemented method (200) of any one of claims 1 to 8.

10. A computer program comprising: A program comprising instructions for causing a computer to carry out a method according to any one of claims 1 to 9 when the program is executed by the computer. Computer program.

11. A computer-readable medium storing the computer program of claim 10.

12. 1. A method of using a medical optical observation device, such as a microscope or endoscope, said method comprising: - recording at least one color image and generating said digital color input image from said at least one color image; - executing a computer-implemented method according to any one of claims 1 to 9; A method comprising:

13. A data processing device (420, 430) for calculating an estimate (224) of a first signal (104) of a plurality of signals, comprising: the plurality of signals are contained in a digital color input image (100), each signal of the plurality of signals having a different ground truth spectrum (302, 308); The data processing device (420, 430) Taking a suitable subset (118) of the plurality of pixels (102) of the digital color input image (100); from said subset (118), by spectral decomposition (204), extracting a preliminary estimate of said first signal (104) as a first decomposed signal (210); extracting a preliminary estimate of at least one further signal (106, 108) of said plurality of signals as at least one further decomposed signal (212, 214); calculating (216) an estimate of the dependency of the first decomposed signal (210) on the at least one further signal (212, 214) as a crosstalk amount (218, 220) for the subset (118); - removing (222) the crosstalk quantities (218, 220) from the first decomposed signals (210) to obtain estimates (224) of the first signals (104) in the subset (118); A data processing device (420, 430) configured to:

14. An optical observation device (400, 410), such as a microscope or endoscope, comprising: said optical observation device being configured to record a digital color input image (100); The optical observation device (400, 410) - further comprising a data processing device (420, 430) according to claim 13, An optical observation device (400, 410).

15. The data processing device (420) includes or consists of an embedded system (430) of the optical observation device (400, 410). The optical observation device (400, 410) of claim 14.

Images