Signal processing apparatus and method for enhancing a digital input signal

Abstract

The present invention relates to a signal processing device (1) for enhancing a digital input signal (I(xi)) recorded by a recording system (4) having a system response (H(xi)), wherein the device is configured to retrieve the digital input signal; calculate a baseline estimate (f(xi)) of the digital input signal, the baseline estimate including spatial features of the digital input signal greater than a characteristic length (fl); remove the baseline estimate from the digital input signal to obtain an output signal O(xi)) including spatial features less than the characteristic length, wherein the device is configured to retrieve a characteristic length (cl) of the system response and calculate the baseline estimate (f(xi)) using a characteristic length less than the characteristic length of the system response.

Description

Technical Field

[0001] The present invention relates to a signal processing apparatus and method for enhancing a digital input signal. Background Art

[0002] When a recording system records a signal, the non-ideal response of the recording system to a point source generates additional noise, as described by its impulse or system response. This is independent of whether the signal is a time-dependent signal and / or a position-dependent signal, such as in radar or sound signals. It is also independent of the dimension of the signal. Noise is introduced into one-dimensional signals and two-dimensional signals, such as images, or multi-dimensional signals, such as three-dimensional data, for example in tomography or in an image with multiple color channels. Considerable efforts have been made to provide apparatuses and methods for removing any artifacts from signals. Summary of the Invention

[0003] Therefore, an object of the present invention is to provide an apparatus and method capable of reducing noise in a signal and thus improving the signal quality.

[0004] This object is solved by a signal processing apparatus for enhancing a digital input signal recorded by a recording system having a system response, wherein the apparatus is configured to retrieve the digital input signal; calculate a baseline estimate of the digital input signal, the baseline estimate representing a baseline and including spatial features of the digital input signal that are greater than or longer than a characteristic length; remove the baseline estimate from the digital input signal to obtain an output signal including spatial features that are less than the characteristic length; wherein the apparatus is configured to retrieve a characteristic length of the system response; and calculate the baseline estimate using a characteristic length that is less than the characteristic length of the system response.

[0005] Furthermore, this object is solved by a signal processing method for enhancing a digital input signal recorded by a recording system having a system response, the system response having at least one characteristic length in at least one dimension or direction, wherein spatial features having a characteristic length that is greater than or longer than at least one characteristic length of the system response are removed from the digital input signal to obtain an output signal.

[0006] This object is further solved by a non-transitory computer-readable medium storing a program that causes a computer to execute the claimed method; by a computer program having program code that, when run on a processor, is for executing the claimed method; by an output signal as a result of executing the claimed method; and / or by a neural network device trained with input and output signal data, wherein the output signal data is created from the input signal data by the claimed method.

[0007] Surprisingly, if the baseline estimate is calculated - and subsequently removed - using a length scale that is less than the characteristic length of the system response, i.e., less than the length of the smallest feature that can be expected in the input signal, noise reduction is improved. The characteristic length can be represented in terms of spatial coordinates or time. For example, when a time signal such as a microphone signal is recorded, the characteristic length of the system response, here the impulse response, can be measured in a time dimension such as seconds. In a recording system such as an imaging system that records digital images, where the system response is, for example, a point spread function, the characteristic length can be measured in spatial coordinates.

[0008] As practical examples of input signals, the input signal can comprise or consist of one of the following: preferably digital input image data, input sonar, sound and / or ultrasonic data, input radar data, input spectroscopy and / or spectral data including cepstrum, input microwave data, input vibration data such as seismic data, input tomography data of any type, and statistical data such as securities trading data, and any combination thereof, all of which can be integer values, real-valued arrays of digital data, or complex-valued arrays of digital data. The input signal can be one of one-dimensional, two-dimensional, three-dimensional, and N-dimensional, where N≥1.

[0009] The output signal can comprise or consist of the following: preferably digital output image data, output sonar, sound or ultrasonic data, output radar data, output spectroscopy and / or spectral data including cepstrum, output microwave data, output vibration data such as seismic data, and statistical data such as securities trading data, and any combination thereof. The output signal can be real-valued or integer-valued. The output signal data can be one of one-dimensional, two-dimensional, three-dimensional, and N-dimensional. The output signal data can be output for further processing.

[0010] In the input signal, the baseline estimate as described herein can be used to remove introduced artifacts from the system response.

[0011] More generally, the baseline estimation method described herein can not only be used to remove the baseline estimate from the input signal, but also to separate the baseline-noise-component I2(x i ) from the content component I1(x i ). Then these two components can be processed and ultimately analyzed separately. For example, in spectral data, especially hyperspectral data, large-scale baseline spectral features can be separated from small-scale spectral features and studied independently.

[0012] The term x i is a shorthand notation for the tuple {x1; …; x N} that contains N position values and represents the discrete position x i - or the position vector at that position. The position x iIt can be represented by the data in an array representing input signal data or preferably a coherent data set. Discrete position x i represents, for example, a pair of discrete position variables {x1; x2} in the case of two-dimensional input signal data and a triple of discrete position variables {x1; x2; x3} in the case of three-dimensional input signal data. In the i-th dimension, the array may contain M i positions, that is I(x i ) may contain a total of (M1×…×M N ) elements. Since specific positions or specific dimensions will not be involved hereinafter, the position is only indicated by x i . The symbol x i can represent spatial and / or temporal dimensions. A combination of temporal and spatial dimensions can occur, for example, in a series of image time frames or a sequence of input signal data.

[0013] I(x i ) can be any value or combination of values at position x i , such as values representing the intensity and / or amplitude of electromagnetic radiation or sound. I(x i ) can represent a color space, for example, the intensity of color R in the RGB space, or the combined intensity of more than one color, for example, in the RGB color space The input signal, such as a multi-spectral or hyperspectral camera or microscope, especially a scanning microscope, that has been recorded as an input image may contain more than three channels. The same is true for other types of input signals.

[0014] A two-dimensional input signal or input image in the three-color RGB format can be regarded as three sets of independent two-dimensional input signal data I(x i ) = {I R (x i ) ; I G (x i ) ; I B (x i )}, where I R (x i ) represents an intensity value such as that of color R, I G (x i ) represents an intensity value such as that of color G, and I B (x i) represents, for example, the intensity value of color B. Alternatively, each color can be considered as constituting a separate input signal and thus separate input signal data. If the input signal data has been recorded as an input image using a multispectral camera or a hyperspectral camera, the input signal data can represent more than three channels. Each channel can represent a different spectrum or spectral range of the spectrum. For example, more than three channels can be used to represent the visible spectrum. If the imaging object contains fluorescent materials, such as at least one fluorophore or at least one autofluorescent substance, each channel can represent a different fluorescence spectrum. For example, if there are multiple fluorescent fluorophores in the input signal, each fluorescence spectrum of a fluorophore can be represented by a different channel of the input signal. In addition, different channels can be used for fluorescence, on the one hand selectively triggered by illumination, and on the other hand autofluorescence generated as a by-product or as a secondary effect of triggering fluorescence. Additional channels may cover the NIR and IR ranges. The channels do not necessarily contain intensity data, but can represent other types of data related to the object image, such as phase. In another example, the channel can contain fluorescence lifetime data representing the fluorescence lifetime after triggering at a specific position in the image. Generally, the input signal data can thus have the following form

[0015] I(x i )={I1(x i );I2(x i );…;I C (x i )},

[0016] where C is the total number of channels in the input signal data. Preferably, all channels have the same dimension.

[0017] The above-described apparatus and method can be further improved by adding one or more of the features described below. Each of the following features can be added to the method and / or apparatus independently of the other features. In particular, a person skilled in the art - with knowledge of the inventive apparatus - is able to configure the inventive method such that the inventive method can operate the inventive apparatus. In addition, each feature has its own advantageous technical effects, as described below.

[0018] In one embodiment, the input signal can be an input image, particularly a digital input image. The output signal can be an output image, particularly a digital output image.

[0019] According to one embodiment, the device can be an observation device, in particular a medical observation device such as an endoscope or a microscope, and the digital input signal can be a digital input image. The device can be configured to at least temporarily capture and / or store the input image; output the output signal as an output image; remove, in particular subtract the baseline estimate from the input signal to calculate the output signal; use a least squares minimization criterion including a penalty term to calculate a baseline estimate representing the noise component by fitting at least one subset of the input signal.

[0020] Furthermore, the method can be a method for - in particular automatically - enhancing or improving the resolution of the input signal, the method including the steps of - in particular automatically - estimating a baseline component in the input signal; obtaining a baseline estimate representing the baseline in the image signal; removing, in particular subtracting the baseline estimate from the input signal to obtain an output signal, wherein the step of estimating the noise component by minimizing the least squares minimization criterion preferably includes the step of calculating the baseline estimate as a fit to at least one subset of the input signal, and wherein the least squares minimization criterion preferably includes a penalty term.

[0021] For example, it can be assumed that the content component in the input signal, that is, the component that should be isolated for further processing, has a high spatial or temporal frequency, for example responsible for the changes in the input signal occurring within a short distance or time period. It is assumed that the noise component has a low frequency, that is, it causes mainly gradual intensity changes and extends over a larger area of the input signal. Thus, the noise component is reflected in the baseline of the input signal.

[0022] Starting from this assumption, the changes in the input signal can be additionally divided into a high-frequency content component I1(x i ) and a low-frequency noise component I2(x i ) as follows

[0023] I(x i ) = I1(x i ) + I2(x i ).

[0024] Due to its low temporal or spatial frequency, the noise component I2(x i ) can be regarded as a more or less smooth baseline, and the content component I1(x i ) is superimposed on it as a high-frequency feature.

[0025] In particular for images, the frequency to be considered for separating the baseline component from the content component can be the spatial frequency. If the temporal frequency is considered instead of the spatial frequency, the same considerations of course apply. In this case, the input signal can for example represent a spectrum, cepstrum or multiple spectra or cepstra.

[0026] Thus, baseline estimation can be equally used to extract and / or eliminate small-scale or large-scale (baseline) signal content in the spatial domain or the frequency domain.

[0027] The input signal I(x i ) is recorded with a recording system that is assumed to have a system response H(x i ). The recorded input signal I(x i ) is obtained from the true image I T (x i ) by convolution with the system response:

[0028] I(x i ) = I T (x i ) * H(x i ).

[0029] If the recording system is an imaging system, the system response corresponds to the point spread function of the imaging system. In other systems, such as a system that records the time trace of a signal, e.g., a sound recording system, the system response corresponds to the impulse response.

[0030] The system response has a characteristic length cl, which can be, for example, the half-width at half-maximum (HWHM), the full-width at half-maximum (FWHM), or any other fraction of width and height. In the time domain, the length corresponds to the duration. The system response and / or its characteristic length can be pre-determined, i.e., known prior to the device described herein, e.g., by experiment, numerical simulation, analysis, or any combination thereof, and stored in the device. Alternatively or additionally, the system response or its characteristic length can be calculated, e.g., by the Lucy-Richardson algorithm.

[0031] Once the baseline estimation is determined and thus the baseline estimate f(x i ) of I2(x i ) is obtained, the output signal O(x i ) can be obtained from the baseline estimate and the input signal. In particular, the output signal can be calculated by subtracting the baseline estimate from the input signal:

[0032] O(x i ) = I(x i ) - f(x i ).

[0033] The output signal O(x i ) is preferably also represented by a discrete digital data array having dimensions N and M1×…×M N elements and thus preferably has the same dimensions as the input signal and / or the baseline estimate. The baseline estimate can also be a hypercube array having N dimensions and (M1×…×M N ) elements and thus has the same dimensions as the input signal.

[0034] The fitting of the input signal can be used to estimate the baseline. Computationally, the fitting, i.e., the baseline estimate, is represented by the discrete baseline estimate f(x i ).

[0035] In one embodiment, the device may be configured to calculate the baseline estimate using a regularization length scale, as further described below, for example. For example, regularization such as Tikhonov regularization may be used to calculate the baseline estimate, where the regularization length scale may be used as a regularization parameter. The regularization length scale preferably depends only on the characteristic length fl.

[0036] The device may be configured to calculate the baseline estimate using a least squares minimization criterion. The minimization criterion preferably includes the baseline estimate and the characteristic length. In particular, the penalty function of the least squares minimization criterion may include the characteristic length.

[0037] In another embodiment, the least squares minimization criterion may include a preferred dimensionless combination of the characteristic length and at least one derivative of the baseline estimate. The derivative is preferably the derivative with respect to the dimension x i or a combination of derivatives.

[0038] In a particular instance, the fitting may be a polynomial fitting of the input signal. In particular, the baseline estimate may be represented by a polynomial of order K in any one of the N dimensions i:

[0039]

[0040] where a i,k are the coefficients of the polynomial in the i-th dimension. A separate polynomial can be calculated for each dimension i = 1, …, N. According to one embodiment, the polynomial fitting can be performed simultaneously in multiple dimensions, depending on the dimension of the input signal.

[0041] The optimal value of the maximum polynomial order K depends on the smoothness required for the baseline estimate. For a smooth baseline, the polynomial order must be set as low as possible, while fitting a highly irregular background may require a higher order.

[0042] In the case of polynomial fitting, the baseline estimate data may consist only of the polynomial coefficients a i,k . However, polynomial fitting can be difficult to control and imprecise because the only parameter that allows adjustment of the input signal data is the maximum polynomial order. The polynomial order can only take integer values. Therefore, it may not always be possible to find the optimal baseline estimate. A non-optimal polynomial fitting may exhibit local minima in the baseline estimate, which may result in annoying artifacts.

[0043] Thus, according to another advantageous embodiment, the fitting of the input signal data can be a spline fitting, in particular a smoothing spline fitting. Spline fitting generally provides more reliable results than polynomial fitting because it is easier to control, for example in terms of smoothness, more robust to noise and produces fewer artifacts. On the other hand, spline fitting is computationally more complex than polynomial fitting.

[0044] For calculating the baseline estimate, it is preferable to use the least squares minimization criterion, which will be minimized for the fitting. The exact formula of the least squares minimization criterion determines the characteristics of the fitting and thus the characteristics of the baseline estimate data. An inappropriate choice of the least squares minimization criterion may result in the baseline estimate not representing the noise component with sufficient accuracy.

[0045] To ensure that the baseline estimate data is an accurate representation of the noise or baseline component in the input signal data and to avoid fitting the baseline estimate to the content component, the least squares minimization criterion can include a penalty term. The penalty term is used to penalize the undesired behavior of the baseline estimate, such as representing a component of the input signal data that has high-frequency content and is thus considered to belong to the content component.

[0046] According to one embodiment, the least squares minimization criterion M(f(x i )) can have the following form:

[0047] M(f(x i )) = C(f(x i )) + P(f(x i ))

[0048] where C(f(x i )) is the cost function and P(f(x i )) is the penalty term. The least squares minimization criterion, the cost function, and the penalty term are preferably scalar values.

[0049] In a particular instance, the cost function represents the difference between the input signal I(x i ) and the baseline estimate f(x i ). For example, if ε(x i ) represents the difference term between the input signal and the baseline estimate as

[0050] ε(x i ) = I(x i ) - f(x i )

[0051] the cost function C(f(x i )) may include the L2 norm ‖ε(x i )‖ 2, here used as a shorthand notation for the sum of the root mean square values across all dimensions of the sum of the squared differences between the input signal and the baseline estimate in the i-th dimension, i.e.,

[0052]

[0053] The L2-norm ||ε(x i )|| 2 is a scalar value. An example of a cost function is the following quadratic difference term:

[0054] C(f(x i )) = ||ε(x i )|| 2

[0055] To improve the accuracy of the baseline estimate, truncation can be beneficial if the difference between the input signal and the baseline estimate is truncated. For example, by using a truncated difference term. The truncated difference term reduces the impact of peaks in the input signal data on the baseline estimate data. This reduction is beneficial if it is assumed that the content component is located in the peaks of I(x i ). Due to the truncated difference term, peaks in the input signal data that deviate from the baseline estimate by more than a predetermined constant threshold s will be "ignored" in the cost function by truncating their penalty for the fit to the threshold, especially for spline fits. Thus, the baseline estimate will only follow such peaks within a limited amount. The truncated quadratic can be symmetric or asymmetric. It is represented below by the truncated difference term.

[0056] In some applications, the content component can consist only or at least mainly in the peaks of the input signal, such as the bright spots in an image. This can be reflected by choosing an asymmetric truncated quadratic term and allowing the fit, especially a spline fit, to follow the valleys rather than the peaks in the input signal data. For example, an asymmetric truncated quadratic can be of the following form

[0057]

[0058] If in another specific application, valleys, i.e., dark areas or regions with low values in the input signal, are also considered content components, a symmetric truncated quadratic can be used instead of an asymmetric truncated quadratic. For example, a symmetric truncated quadratic can have the following form:

[0059]

[0060] Using the truncated quadratic function, the cost function C(f(x i )) can preferably be expressed as

[0061]

[0062] The penalty term P(f(x i )) in the least squares minimization criterion M(f(x i )) can take any form, and a penalty is introduced if the baseline estimate fits data that is considered to belong to the content component I1(x i ). If the content component in the input signal is represented in the baseline estimate, the value of the penalty term increases, thus creating a penalty.

[0063] If, for example, it is assumed that the noise component I2(x i ) is considered to have a low spatial frequency, the penalty term can include a term that increases if the spatial frequency of the baseline estimate increases.

[0064] In one embodiment, such a penalty term can be a roughness penalty term that penalizes non-smooth baseline estimate data that deviates from a smooth baseline and thus effectively penalizes the fitting of data with a high spatial frequency.

[0065] In particular, a penalty term including a characteristic length fl can be used to calculate the baseline estimate, where the characteristic length represents a length scale, features in the input signal above which are considered noise and features below which are considered content. For example, if the characteristic length fl is set to 0.01 millimeters, all features smaller than 0.01 millimeters will be considered content and will thus be penalized if included in the baseline estimate. The characteristic length can be used as the regularization length scale for regularization, such as Tikhonov regularization, for calculating the baseline estimate.

[0066] According to another aspect, a deviation from a smooth baseline can result in large values in at least one of the first derivative of the baseline estimate, i.e., the steepness or gradient, and the second derivative, i.e., the curvature. Thus, the roughness penalty term can include at least one of the first spatial derivative of the baseline estimate, in particular the sum of squares and / or the absolute value of the first spatial derivative, and the second derivative of the baseline estimate, in particular the sum of squares and / or the absolute value of the second spatial derivative. More generally, the penalty term can include any arbitrary order spatial derivative of the baseline estimate, or any linear combination of the spatial derivatives of the baseline estimate. Different penalty terms can be used in different dimensions.

[0067] In one embodiment, the penalty term P(f(x i )) can include a dimensionless combination, such as a quotient, of the combination of the derivative of the baseline estimate f(x i ) with respect to its variables and the characteristic length fl. Different characteristic lengths can be used for different dimensions

[0068] For example, the roughness penalty term P(f(x i )) can be formed as

[0069]

[0070] This roughness penalty term penalizes large rates of change of the baseline estimated gradient, or equivalently, high curvature, thus favoring smooth estimates. Here, γ j is the regularization parameter, or equivalently, the regularization length scale, while is a discrete operator for computing the second derivative in the j-th dimension. Since the unit of the second derivative is (the unit of x i ) -2 , i.e., length -2 or time -2 , the regularization length scale γ j is set to the fourth power of the characteristic length, so that the penalty term is scalar.

[0071] In the penalty term based on the first derivative , such as

[0072]

[0073] the regularization length scale γ j can be equal to the square of the characteristic length, since the unit of the first derivative is (the unit of x i ) -1 .

[0074] For combinations of various derivatives, such as

[0075]

[0076] each regularization length scale is set according to the corresponding derivative. In the above example, and

[0077] In the combined derivatives, such as the corresponding combination of characteristic lengths, for example, fl i ·fl j can be used.

[0078] The regularization length scale γ j depends on the structure of the input signal, i.e., the largest length scale considered to be part of the content component. Structures in the input signal with length scales greater than the characteristic length represented by the regularization length scale are considered to belong to the noise component. The regularization length scale is preferably greater than zero. However, for better resolution, it is preferable that the regularization length scale γ j is chosen such that the characteristic length fl it represents is less than the characteristic length cl of the system response H(x i ), fl j < cl j , where cl jis the characteristic length in the direction of dimension j of the system response. If the feature length is equal to or less than approximately 40%-50% of the characteristic length of the system response, or equivalently fl j ≤0.4…0.5l j , then better resolution results can be obtained. It should be noted that both the characteristic length and the feature length can be constant in all dimensions.

[0079] In the discrete case, differentiation can be effectively computed using convolution. For example,

[0080]

[0081] where the second derivative matrix

[0082]

[0083] However, preferably, the roughness penalty term P(f(x i )) is formed as

[0084]

[0085] This is the roughness penalty term, which penalizes small-scale features and large gradients in the baseline estimate. The sum over j allows for different penalty terms to be used in different dimensions. It should be noted that since x j and f(x i ) are both discrete, differentiation can be performed by convolving with the derivative array . The operator represents the discrete first derivative or gradient operator in the j dimension, which can be represented by an array.

[0086] As an alternative or supplement to the derivative or linear combination of derivatives of the baseline estimate, the penalty term can include feature extraction, particularly linear filters or linear combinations of such filters. Feature extraction filters can be Sobel filters, Laplace filters, and / or FIR filters, such as high-pass or band-pass spatial filters with high spatial frequency passbands.

[0087] In such a general formula, the penalty term for the j dimension may include the general operator ζ (j) and is expressed as

[0088]

[0089] Similarly, γ j includes the feature length fl of any length scale involved in ζ (j) . In particular, γ j can be a function of the feature length fl, γ j= g(fl). More specifically, the regularization length scale can be a linear, quadratic, or more general polynomial function of the characteristic length.

[0090] The least squares minimization criterion M(f(x i )) can be minimized using known methods. In one example, a preferred iterative quadratic or semi - quadratic minimization scheme can be used. To perform the minimization, the baseline estimator engine can include a minimization engine. The minimization scheme can include an iterative mechanism with two levels of iteration.

[0091] The minimization scheme can include, for example, at least a portion of the LEGEND algorithm, which is computationally efficient. The LEGEND algorithm is described in Idier, J. (2001): Convex Half - Quadratic Criteria and Interacting Variables for Image Restoration, IEEE Transactions on Image Processing, 10(7), pp. 1001 - 1009 and Mazet, V., Carteret, C., Bire, D, Idier, J. and Humbert, B. (2005): Background Removal from Spectra by Designing and Minimizing a Non - Quadratic Cost Function, Chemometrics and Intelligent Laboratory Systems, 76, pp. 121 - 133. Both articles are incorporated herein by reference in their entirety.

[0092] The LEGEND algorithm introduces discrete auxiliary data d(x i ) which preferably has the same dimension as the input signal data. The auxiliary data is updated at each iteration based on the latest initial baseline estimate, the truncated quadratic term, and the input signal data.

[0093] In the LEGEND algorithm, two iterative steps are used to minimize the least squares minimization criterion that contains only the cost function and no penalty term until a convergence criterion is met.

[0094] A suitable convergence criterion can be, for example, that the sum of the differences between the current baseline estimate data and the previous baseline estimate data at all positions x i is less than a predetermined threshold.

[0095] In a further refinement, the convergence criterion can be expressed as

[0096]

[0097] where t is a scalar convergence value that can be set by the user.

[0098] As a starting step of the LEGEND algorithm, a set of initial baseline estimate data is defined.

[0099] If polynomial fitting is used, the LEGEND algorithm can be initiated by estimating a set of starting coefficients a for the first baseline for each i = 1,..., N polynomials k to start.

[0100] If spline fitting is used, the initial conditions for starting the LEGEND algorithm may be d l (x i ) = 0, f(x i ) = I(x i ) and start the iteration by entering the second iteration step.

[0101] In the first iteration step, the auxiliary data may be updated as follows:

[0102]

[0103] where l = 1…L is the index of the current iteration and α is a constant that can be chosen. Preferably, α is close to but not equal to 0.5. A suitable value of α is 0.493.

[0104] In the second iteration step, the baseline estimate f l (x i ) is updated based on the previously computed auxiliary data d l (x i ), the baseline estimate f l-1 (x i ) from the previous iteration l - 1, and the penalty term P(x i ).

[0105] The baseline estimate f l (x i ) can minimize the minimization criterion M(f(x i )) which has been corrected for the LEGEND algorithm including the auxiliary data.

[0106] Specifically, the updated baseline estimate data can be calculated using the following formula in the second iteration LEGEND step:

[0107]

[0108] Here, [‖I(x i ) - f l-1 (x i ) + d l (x i )‖ 2 + P(f(x i )] represents the corrected minimization criterion.

[0109] The second iterative step can calculate and update the baseline estimation data using the following matrix:

[0110]

[0111] Here is an (M1 × … × M N ) 2 -dimensional array. In the two-dimensional case, A i is an (M x -1)(M y -1) × M x M y array and is given by

[0112]

[0113] where

[0114]

[0115] Repeat the two iterative steps of updating d l (x i ) and f l (x i ) until the convergence criterion is met.

[0116] According to another aspect, the second step of the LEGEND algorithm is corrected using convolution instead of matrix calculation. This greatly reduces the computational workload.

[0117] More specifically, preferably, the updated baseline estimate f l (x i ) is directly calculated by convolving the Green's function with the sum of the input signal and the updated auxiliary data.

[0118] According to a more specific aspect, the second iterative step of the LEGEND algorithm can be replaced by the following iterative step, where the updated baseline estimation data f l (x i ) in the l-th iteration is calculated using the Green's function G(x i ) as follows

[0119] f l (x i ) = G(x i)*(I(x i )+d l (x i ))。

[0120] Compared with the traditional LEGEND algorithm, this step significantly reduces the computational burden.

[0121] The reduced computational burden stems from the fact that the convolution is calculated according to the second iterative step described above. This calculation can be efficiently performed using the FFT algorithm. Moreover, due to the FFT algorithm, the second iterative step can make full use of array processors, such as graphics processing units or FPGAs. If the input signal data and all other arrays are two-dimensional, the computational problem is reduced from (M x ×M y ) 2 to M x ×M y . For the general N-dimensional case, the computational burden is reduced from the (M1×…×M N ) 2 dimensional matrix calculation to the FFT calculation with an array of (M1×…×M N ) dimensions.

[0122] Therefore, for two-dimensional input signal data, the removal of the baseline estimate can be performed very quickly, preferably in real time. In a (2k×2k) data array, the baseline estimate can be removed in 50 milliseconds or less.

[0123] In a specific embodiment, the Green's function can have the following form

[0124]

[0125] where F[…] is the discrete N-dimensional Fourier transform, F -1 […] is the discrete N-dimensional inverse Fourier transform, γ j is the regularization length scale of the roughness penalty term, is the discrete penalty array at the position m in the i-th dimension, and N is the total number of dimensions. The above index D (j) indicates that there may be different penalty arrays for each dimension j.

[0126] Preferably, the discrete penalty array corresponds to the discrete representation of the functional derivative of the penalty term P (j) (f(x i )) Since all functions are represented by discrete arrays, numerical differentiation can be performed by convolution

[0127]

[0128] where is a discrete array for calculating the derivative of a function .

[0129] One great advantage of the above Green's function is that any form of penalty term P(f(x i )) can benefit from the fast calculation in the second iterative step of the minimization engine. Thus, in embodiments using the Green's function, any penalty term for obtaining a good baseline estimate can be used.

[0130] For the general expression of the penalty term

[0131]

[0132] The array is defined as

[0133]

[0134] where ζ (j) is the general operator of the penalty term, * represents an N-dimensional convolution, corresponds to the discrete first-order functional derivative in the function f(x i,m ), which can represent intensity, for example. This equation can be solved by the least squares method.

[0135] For example, if the penalty term is

[0136]

[0137] The derivative array in the convolution can be expressed as:

[0138]

[0139] The device may include a storage part. The storage part may be configured to store the input signal at least temporarily.

[0140] The device may include a signal input part, which may include, for example, one or more standard connectors and / or one or more standardized data exchange protocols, such as HDMI, USB, RGB, DVI. The signal input part may be adapted to receive the output signal via one or more standard connectors and / or one or more data exchange protocols. For example, a storage device and / or a sensor, such as a camera, may be connected to the signal input part.

[0141] The device may include a signal output section, which includes, for example, one or more standard connectors and / or one or more standardized data exchange protocols, such as HDMI, USB, RGB, DVI. The signal output section may be adapted to output an output signal via one or more standard connectors and / or one or more data exchange protocols. For example, another computer, network, and / or display may be connected to the signal output section.

[0142] The device may further include a signal processor, which may be configured to calculate a baseline estimate.

[0143] The signal processor may include a baseline removal section. The baseline removal section may be adapted to subtract a baseline component, for example, subtract the baseline estimate of the input signal to calculate the output signal. In certain applications, assuming that the content component exists in the low-frequency component of the input signal, the baseline estimate already represents the content component. In this case, it is not necessary to remove the baseline estimate from the input signal. Instead, the baseline estimate may be output for display or further processing.

[0144] The signal processor may include a baseline estimation engine. The baseline estimation engine may be configured to calculate a baseline estimate by fitting at least one subset of the input signal. The baseline estimation engine may include a discrete representation of the least squares minimization criterion (M(x i ))

[0145] The signal processor, the baseline estimation engine, the baseline removal section, and the minimization engine may be implemented separately in hardware, software, or a combination of hardware and software. For example, at least one of the signal processor, the baseline estimator engine, the baseline removal section, and the minimization engine may be at least partially implemented by a subroutine, a part of a general-purpose processor, such as a CPU, and / or a dedicated processor, such as a CPU, GPU, FPGA, vector processor, and / or ASIC.

[0146] Another way to implement any of the above embodiments of the device and method is to train an artificial neural network, for example, a convolutional neural network, using input signal data and output signal data pairs, where the output signal data is generated using an embodiment of the above method. The neural network device trained in this way can be regarded as an implementation of the method for generating the training pairs of input and output signal data.

[0147] According to another aspect, the neural network device may be adapted to generate an output signal O(x i ), where spatial features with a characteristic length fl less than at least one characteristic length of the system response and spatial features with a characteristic length fl greater than at least one characteristic length of the system response and included in the digital input signal are removed.

[0148] It should be noted that if the input signal I(xi ) If it has not been convolved or deconvolved before, the calculation and / or removal of the baseline provides the best results. If the input signal I(x i ) is preprocessed by the baseline removal of the present invention, deconvolution provides the best results.

[0149] According to one embodiment, the signal processing device is configured to use different feature lengths fl 1...K and corresponding penalty terms to calculate more than one, i.e., multiple output signals O i ) from a single digital input signal I(x 1…K (x i ), and perform multi-image deconvolution on the multiple output signals O 1…K (x i ) to obtain the deconvolved output signal J(x i ). For each different feature length, a different regularization length scale is obtained, and thus a different penalty term. Therefore, for each different feature length fl 1…K , the baseline estimate will be different. Each different baseline estimate f 1…K (x i ) is removed from a separate copy of the input signal to obtain multiple output signals: O k = I(x i ) - f k (x i ). Here, K represents the total number of output signals and thus also the total number of different feature lengths.

[0150] Therefore, the first output signal O1(x i ) can be calculated using the first feature length fl1 that results in the first baseline estimate, and the second output signal O2(x i ) can be calculated using a second feature length fl2 different from the first feature length fl1, and the second feature length fl2 produces the second baseline estimate. Then, multi-image deconvolution can be performed based on O1(x i ) and O2(x i ).

[0151] Multi-image deconvolution is an image fusion method that reduces shadows and blurring caused by scattering of the light sheet and the sample to be analyzed (see, for example, the publication "Efficient Bayesian-based multiview deconvolution", Stephan Preibisch et al., Nat Methods., June 2014, 11(6):645–648). Castello, Diaspro, Vicidomini, (2014) further describes multi-image deconvolution: "Multi-images deconvolution improves signal-to-noise ratio on gated stimulated depletion microscopy", Applied Physics Letters 105.23:234106; in Faramarzi, Rajan, Christensen (2013): "Unified blind method for multi-image super resolution and single / multi-image blur deconvolution", IEEE Transaction on Signal processing, (22)6; Ingaromo, Yoork, Hoogendorn et al. (2014): "Richardson-Lucy deconvolution as a general tool for combining images with complementary strengths", Wiley Online Library; and Harmeling, Sra, Hirsch, "Multiframe deconvolution, super-resolution, and saturation correction via incremental EM", Proceedings of the 17th International Conference on Signal Processing, IEEE, 2010.

[0152] If the multi-image deconvolution is based on at least one output signal O(xi ), an unexpected improvement in the resolution of the input signal can be obtained, where the output signal has been calculated using a characteristic length fl that is less than the characteristic length cl, in particular less than 0.4...0.5cl, and at least one other output signal O(x i ), which has been calculated using a characteristic length fl that is equal to or greater than the characteristic length cl.

[0153] If the individual system responses H k (x i ) are estimated separately for each of the multiple output signals O 1...K (x i ) and used in multi-image deconvolution, a further improvement in resolution can be obtained. Alternatively, if a low computational load is to be maintained, the system response of the input signal I(x i ) can be used. However, re-estimating the system response for each output signal O k (x i ) yields better results because the effective system response is corrected after removing the baseline.

[0154] Preferably, the maximum a posteriori probability (MAP) distribution is used to estimate the system response from the output signal O k (x i ).

[0155] More specifically, the following MAP can be maximized:

[0156] P(T k (x i ),H k (x i )|O k (x i ))

[0157] This is the probability of the unknown attribute T k (x i ), i.e., the true image, and H k (x i ) for a given output signal O k (x i ), and is used as the input for multi-image deconvolution. Assuming Poisson probability, the solution to this maximization is an iterative algorithm called blind Richardson-Lucy deconvolution. This algorithm is described in Fish, Brinicombe, Pike (1995): "Blind deconvolution by the Richardson-Lucy algorithm", J. Opt. Soc. Am. A. (12) 1.

[0158] The following further illustrates the embodiments only by way of exemplary embodiments, which are also shown in the accompanying drawings. In the drawings, the same reference numerals are used for features that correspond to each other in at least one aspect of function and design.

[0159] The combinations of features shown in the appended embodiments are for illustrative purposes only and may be modified. For example, features of an embodiment that do not have a technical effect required for a particular application may be omitted. Similarly, if a particular application requires the technical effect associated with this feature, features not shown as part of the embodiment may be added.

[0160] As can be clearly seen from the above, any type of image data can be used. BRIEF DESCRIPTION OF THE DRAWINGS

[0161] Figure 1 A schematic diagram showing an embodiment of a device for reducing noise in an input signal.

[0162] Figure 2 A flowchart showing baseline removal in an input signal.

[0163] Figure 3 Shows Figure 2 Detail III of

[0164] Figure 4 A schematic diagram showing a method for increasing the resolution of an input signal.

[0165] Figure 5 Shows a sample input signal.

[0166] Figure 6 Shows an output signal based on Figure 5 the input signal.

[0167] Figure 7 Shows Figure 5 Detail VII of

[0168] Figure 8 Shows Figure 6 Detail VIII of

[0169] Figure 9 Shows the intensity distribution along Figure 7 and Figure 8 the cross-section of

[0170] Figure 10 Shows a deconvolved output signal of multi-image deconvolution based on Figure 5 the input signal.

[0171] Figure 11 Shows Figure 10 Detail XI of ; and

[0172] Figure 12 A schematic diagram showing input signal data, content components in the input signal data, noise components in the input signal data, baseline estimation data, and output signal data. Detailed implementation

[0173] First, refer to Figure 1 to explain the structure of device 1. Device 1 can be a medical observation device 2, such as an endoscope or a microscope 2a, or any other device configured to capture high-quality images, such as a device for aerial or astronomical reconnaissance. For the purpose of explanation only, microscope 2a is shown as an example of device 1. For the purpose of the following description of the embodiment, there is no difference between an endoscope and a microscope.

[0174] Device 1 can include a recording system 4, which is adapted to capture input signal data 6. If the recording system 4 is an imaging system, it can be provided, for example, with a camera 8 configured to record digital input images, preferably in digital format. The camera can include an image sensor 9. The camera 8 can be a CCD, multi-spectral, or hyperspectral camera, which records input signal data 6 in a plurality of channels 10, where each channel 10 preferably represents a different spectral range from infrared to ultraviolet. The input signal data 6 is also designated as input signal I(x i ).

[0175] Of course, other types of input signal data 6 can be recorded by devices or sensors other than cameras, such as point detectors - for example, in the case of a confocal microscope - one or more microphones, vibration sensors, accelerometers, speed sensors, antennas, pressure sensors, temperature sensors, capacitance sensors, magnetic sensors, by radiography, tomography, by ultrasound, and any combination thereof.

[0176] For example, in the case of a CCD camera recording in the RGB color space, three channels 10, for example, can provide an R channel, a G channel, and a B channel to represent the visible light input signal of the object 12. In the case of a multi-spectral or hyperspectral camera, there can be a total of more than three channels 10 for at least one of the visible light range, IR light range, NIR light range, and ultraviolet light range.

[0177] The recording system 4 has a known system response H(x i ), for example, from measured values, or it can be estimated using a MAP algorithm, such as the Richardson-Lucy algorithm. The system response H(x i ) has a characteristic length cl, which can also be known and / or can be direction-independent. Alternatively, there can be different characteristic lengths cl i in each different dimension, such as cl1 in the x1 direction and cl2 in the x2 direction, as Figure 1as shown. The characteristic length can be the HWHM or FWHM length. The variable x i can specify a spatial or temporal dimension.

[0178] In the case of an imaging system as a recording system 4, the object 12 is located in the detector volume 13. The detector volume can be configured to receive the object 12 to be examined by the device 1. In the case of an imaging system as a recording system, the detector volume is preferably located in the field of view 14 of the imaging system. The object 12 can include living and / or inanimate matter. The object 12 can further include one or more fluorescent materials, such as at least one fluorophore 14.

[0179] A multispectral or hyperspectral camera may have a channel 10 for each different fluorescence spectrum of the fluorescent material in the object 12. For example, each fluorophore 14 can be represented by at least one channel 10 that matches the fluorescence spectrum triggered by the illumination system 16. Alternatively or additionally, a separate channel 10 can be provided for the autofluorescence spectrum or the secondary fluorescence spectrum, which is triggered by the fluorescence excited by the illumination system 16, or for lifetime fluorescence data. Of course, the illumination system 16 can also or separately emit white light or any other combination of light without triggering fluorescence in the object 12.

[0180] The microscope 2 can be adapted to excite fluorescence, for example, the fluorescence of the fluorophore 15 in the object 12 with light of a suitable fluorescence excitation wavelength of the illumination system 16. The illumination system 16 can be arranged relative to the detector volume 13 opposite to and / or on the same side as the recording system 4.

[0181] If the illumination system 16 is arranged on the same side as the recording system 4, its light can be guided through the lens 17, through which the input signal I(x i ) is also acquired. The illumination system 16 can include or consist of one or more flexible lights to direct light onto the object 12 from one or more different directions. A suitable blocking filter (not shown) can be arranged in the optical path in front of the camera 8, for example, to suppress glare. In the case of fluorescence, the blocking filter preferably only blocks the illumination wavelength and allows the fluorescence of the fluorophore 15 in the object 12 to pass through to the camera 8.

[0182] If the illumination system is arranged opposite to the detector volume 13, its light can pass through the detector volume 13.

[0183] Obviously - not limited to the general case - the input signal data 6 can be captured by any type of camera or photodetector.

[0184] If a single channel 10 is included in a two-dimensional image, the input signal data 6 is two-dimensional. If more than two channels 10 are included and / or if the input signal data 6 represents a three-dimensional array, such as a three-dimensional image, the input signal can have a dimension higher than two. If the signal represents, for example, a time trace or a one-dimensional spatial measurement, the signal may be one-dimensional.

[0185] The three-dimensional input signal data 6 can be recorded by the device 1, for example, by using light field techniques, z-stacking in a microscope, three-dimensional reconstruction of images obtained by SCAPE microscopy and / or images obtained by SPIM microscopy. Other sources of three-dimensional input signal data can be tomographic images. In the case of a three-dimensional image, each plane of the three-dimensional input signal data 6 can be considered as a two-dimensional input signal 6. Similarly, each plane can include several channels 10.

[0186] Each channel 10 can be regarded as a separate two-dimensional image or signal. Alternatively, multiple channels can be interpreted together as a multi-dimensional array.

[0187] The input signal data 6 is a digital representation of discrete real or integer values I(x i ), such as intensity or phase, where x i represents the position in the input signal data 6, and I is the quantity at that position, which constitutes the input signal. The term x i is an abbreviated notation for the tuple {x1; …; x N}, which contains N dimensions, N≥1, and represents the discrete position x i in the discrete input signal data. The position x i can be a pixel or preferably a set of coherent pixels in the input signal data. The discrete position x i represents, for example, a pair of discrete position variables {x1; x2} in the case of two-dimensional input signal data and a triple of discrete position variables {x1; x2; x3} in the case of three-dimensional input signal data. In the i-th dimension, the array may contain M i positions, that is in the case of N dimensions, I(x i ) may contain a total of (M1×…×M N ) elements.

[0188] The device 1 can further include a storage section 20, which is adapted to contain the input signal data 6 at least temporarily. The storage section 20 can include volatile or non-volatile memory, such as the cache memory of the CPU 22 of a computing device 24, such as a PC, and / or the GPU 26. The storage section 20 can further include RAM, a hard disk drive, or a removable storage section, such as a USB stick or an SD card. The storage section 20 can include any combination of these types of memory.

[0189] For obtaining input signal data 6, for example, from a camera 8, a signal input section 28 may be provided. The signal input section 28 may include standardization connection means 30, such as a standard data exchange protocol, a hardware connector, and / or a wireless connection, or any combination thereof. Examples of standard connectors that may be connected to the camera 8 are HDMI, USB, and RJ45 connectors.

[0190] The apparatus 1 may further include a signal output section 32, which may include standardization connection means 34, such as a standard data exchange protocol, a hardware connector, and / or a wireless connection, each configured to output output signal data 36 to one or more displays 37. The output signal data 36 preferably has the same dimension as the input signal data 6 and is represented by a discrete array of discrete values, forming an output signal O(x i ).

[0191] To calculate the output signal O(x i ) from the input signal I(x i ), a signal processor 38 may be provided. The signal processor 38 may be at least partially hardware, at least partially software, and / or a combination of both hardware and software. For example, the signal processor 38 may include at least one of the CPU 22 and the GPU 26 of the computing device 24, and has been encoded in software and temporarily exists as a structural entity in the CPU 22 and / or the GPU 26 as part of the operating state. The signal processor 38 may also include additional hardware, such as one or more ASICs, which are specifically designed to perform the operations required by the apparatus and method.

[0192] Before continuing Figure 1 with the further description, the general principle of enhancing the input signal I(x Figure 12 ) by estimating and removing the baseline if necessary is explained with reference to i . To remove the baseline, the signal processor 38 may include a baseline removal section 40.

[0193] Assume that the input signal I(x i ) is composed of the sum of a content component I2(x i ) and a noise component I1(x i ), the content component I2(x i ) contains the content of interest, especially may correspond to an unknown "true" input signal, and the noise component I1(x i ) contains artifacts and noise that do not belong to the true input signal. Hereinafter, it is assumed that the noise component I2(x i ) is composed of components with low spatial frequencies or mainly includes components with low spatial frequencies. Therefore, the noise component represents a smooth baseline, and the content component I1(xi ) fluctuate at a higher spatial frequency around this baseline. Assume that the noise component I2(x i ) is smooth and has a large length scale; in contrast, the content component I1(x i ) is assumed to be non-smooth and contain at least one of peaks and valleys, and consists of structures or features with a length scale or characteristic length fl smaller than that of the noise component. Subtracting the noise component, i.e., the baseline, enhances the image contrast and reduces the noise, as Figure 12 shown. However, in some cases, this may be the opposite, with the content residing in large-scale structures and the noise in small-scale structures. I1(x i ) and I2(x i ) are not known, so they must be estimated.

[0194] To enhance the input signal, an estimate of the noise component I2(x i ) is calculated. This estimate is represented by the baseline estimate f(x i ), i.e., the data representing the baseline estimate. The baseline estimate f(x i ) is a discrete preferably real-valued array, which preferably has the same dimension as the input signal data 6 and / or the output signal data 36. In Figure 1 the baseline estimate f(x i ) is represented by the baseline estimate data 44. The baseline estimate f(x i ) can also exist at least temporarily in the storage section 20. Once the baseline estimate is calculated, the output signal, here represented as O(x i ), is obtained by subtracting the baseline estimate f(x i ) from the input signal I(x i ) at each position x i ).

[0195] According to Figure 1 , the signal processor 38 can include a baseline estimator engine 42, which is configured to calculate the baseline estimate f(x i ) by fitting to at least one subset of the input signal data 6. Preferably, the fit to at least the subset of the input signal data is a spline fit.

[0196] For a computationally efficient spline fit, the baseline estimator engine 42 can include a semi-quadratic or quadratic minimization engine 46, which can be, for example, a combination of a subroutine or a hard-wired algorithm and software. The minimization engine 46 can be configured to perform a quadratic or semi-quadratic minimization scheme, and for this purpose, can include two iterative levels 48, 50.

[0197] Preferably, the minimization engine 46 uses convolution in the second iteration stage 50 to calculate the baseline estimation data 44. Since convolution can be more efficiently calculated on an array processor using the FFT, preferably the signal processor 38 includes an array processor such as the GPU 26. In operation, the signal processor includes the minimization engine 46.

[0198] Reference Figure 2 , the step of calculating the output signal O(x i ) from the input signal I(x i ) is described as being performed by the apparatus 1. It should be noted that in the case where the input image is the input signal, preferably each channel 10 is processed separately.

[0199] In a first step 60, various parameters of the preset baseline estimator engine 42 can be defined by the user, for example, using a graphical user interface 62( Figure 1 ). The parameters can include the type of fit to the input signal data 6 that will be performed by the baseline estimator engine 42. For example, the user can choose between a polynomial fit and a spline fit of the input signal data 6 to the baseline estimation data 44. The user can also set a characteristic length fl, which can be used to separate the content component from the noise component.

[0200] In addition, the user can choose between various penalty terms P(f(x i )) for the minimization scheme. The penalty terms determine the shape of the baseline estimate by penalizing the representation of the content component I1(x i ) in the baseline estimate.

[0201] For example, the user can be presented with a selection of various penalty terms that penalize the non-smooth characteristics of the baseline estimation data 44. For example, the penalty term can be a high-pass spatial frequency filter for the baseline estimation data 44, which becomes larger if the baseline estimation data 44 contains components with high spatial frequencies. Other penalty terms can include the gradient of the baseline estimation data 44. Another example of a penalty term can be the curvature of the baseline estimation data 44. In addition, for feature extraction filters such as Sobel, Laplace, and / or FIR band-pass, the user can choose a high-pass or low-pass filter as a penalty term. In addition, a linear combination of any of the above can be selected. Different penalty terms can be selected for different dimensions or different channels of the input signal data 6.

[0202] The general representation of the penalty term is as follows

[0203]

[0204] where ζ (j) is the general operator of the penalty term, which defines the nature of the penalty term, γ jis the regularization length scale. Adjust the dimension of the regularization length scale such that the penalty term is scalar. The regularization length scale is a function of the characteristic length, fl,γ j = g(fl).

[0205] In the following, it is assumed that the user selects a gradient-based roughness penalty term based on the gradient of the baseline-estimated data f(x i,m ) or 44, which has the following form

[0206]

[0207] This penalty term penalizes large gradients in the baseline-estimated data. The operator represents the first derivative or gradient in dimension j.

[0208] Using the above gradient-based penalty term, the parameters specified by the user can further include an array of regularization length scales γ j . The regularization length scale represents the length scale, i.e., the characteristic length below which the structure in the input signal is considered content. Structures in the input signal with lengths greater than are considered noise. It can be clearly seen from the index j of γ j that the regularization length scale and thus the characteristic length may be different in each direction. Of course, it is also possible to use only one direction-independent characteristic length.

[0209] Surprisingly, if the characteristic length fl used as the regularization length scale in the penalty function or in the regularization length scale is less than the characteristic length cl, fl < cl, fl i < cl i , the resolution of the input signal is greatly improved.

[0210] If the characteristic length fl is set to be equal to or less than half of the characteristic length cl of the system response, or less than or equal to 40% of the characteristic length cl of the system response, fl ≤ 0.4…0.5cl, fl i ≤ 0.4…0.5cl i , these results can be further improved.

[0211] When selecting parameters for the baseline estimator engine, the user can choose between symmetric and asymmetric quadratic terms , which also determines the shape of the baseline estimate by specifying the influence of large peaks on the baseline-estimated data.

[0212] For example, the user can choose the following asymmetric truncated quadratic:[[]]

[0213]

[0214] Where s represents the threshold value input by the user. The threshold value defines the maximum deviation between the input signal data and the baseline estimation data. Peaks above the baseline estimation will not attract the baseline estimation more than peaks deviating from the threshold value.

[0215] Finally, the user can select the convergence criterion and / or the threshold value t that the convergence criterion must reach.

[0216] After the initial parameters of the baseline estimator engine 42 have been set, data is initialized for the iterative minimization scheme 66 in step 64.

[0217] From then on, the iterative minimization scheme 66 is executed by the minimization engine 46 until the convergence criterion 68 is satisfied. In this embodiment, the following convergence criterion is adopted:

[0218]

[0219] Where l represents the current iteration, and t is a constant scalar threshold value that can be specified by the user.

[0220] If the convergence criterion 68 is satisfied, it is assumed that the baseline estimation data 44 has been successfully calculated. Therefore, in step 70, the baseline estimation data f(x i ) is subtracted from the input signal data I(x i ) to obtain the output signal data O(x i ).

[0221] After calculating the output signal O(x i ), post-processing operations 72, such as deconvolution, can be performed on the output signal data 36.

[0222] More specifically, based on the same input signal I(x i ) but using a different feature length fl each time and thus a different regularization length scale y j , the above calculation of the baseline estimation is performed at least twice. This will result in K different baseline estimations for K different regularization length scales. Each of these K different baseline estimations f k (x i ) is separately removed from the input signal, specifically subtracted, to obtain multiple output signals O k (x i ), where k = 1...K, O k (x i ) = I(x i ) - f k (x i ).

[0223] Among the K output signals O k (x iIn each of (), represents a different regularization length scale y j,k or an equivalent characteristic length fl j,k . At least one output signal is preferably calculated using a regularization length scale that includes a characteristic length fl that is 40% to 50% less than the characteristic length cl, and preferably at least one output signal is calculated using a regularization length scale having a characteristic length equal to or greater than the characteristic length, fl>cl, fl i >cl i . A ±20% deviation between the characteristic length and the characteristic length can still be considered equal.

[0224] For the output signal O k (x i ), for each of them, using the MAP algorithm, such as the Richardson-Lucy algorithm, to calculate an estimate of the corresponding corrected system response. Then, for a set of output signals O k (x i ), k = 1…K, multi-signal or multi-image deconvolution is performed to obtain the deconvolved output signal data 36, representing the deconvolved output signal J(x i ). The deconvolved output signal J(x i ) is an array of discrete digital real or integer values, whose dimension is preferably the same as that of the input signal I(x i ).

[0225] The output signal O k (x i ) and the deconvolved output signal J(x i ) can be displayed on the display 37 with or without post-processing.

[0226] In Figure 3 , Figure 2 Details III of are shown to more specifically explain the minimization scheme 66. The minimization scheme 66 includes a first iterative stage 48 and a second iterative stage 50.

[0227] In principle, the minimization scheme 66 executed by the minimization engine 46 can be the LEGEND algorithm. However, it is preferably to modify the second step of the LEGEND algorithm, which can be modified to significantly reduce the computational burden.

[0228] In the illustrated embodiment, after initializing the data in step 64, it enters the second iterative stage 50. At this time, the first estimate f1(x i ) of the baseline estimate data is calculated by using the convolution of the input signal data with the Green's function G(x i ).

[0229] f1(x i ) = G(x i)*I(x i )

[0230] For the gradient-based penalty term used in this embodiment, the Green's function is defined as follows:

[0231]

[0232] where F[…] is the discrete N-dimensional Fourier transform, F -1 […] the inverse discrete N-dimensional Fourier transform, γ j is the regularization length scale of the roughness penalty term and

[0233]

[0234] Then, in the first iteration stage 48, the auxiliary data d l (x i ) can be calculated using the following current baseline estimate data 44 for an updated version:

[0235]

[0236] The parameter α is a constant and may have been specified by the user.

[0237] Next, in the second iteration stage 50, the updated auxiliary data d l (x i ) of the current iteration l is used to calculate the updated baseline estimate data 44 as follows

[0238] f l (x i ) = G(x i )*(I(x i ) + d l (x i ))

[0239] In the next step, it is checked whether the convergence criterion 68 is satisfied. If this is not the case, the minimization scheme 66 proceeds to the iteration step 48 using the updated baseline estimate data f l (x i ).

[0240] Figure 4 An example process flow for obtaining the deconvolved output signal I(x i ) from the input signal I(x i ) is shown. In step 400, the output signal O1(x 1,i ) is calculated using the first characteristic length fl i ), preferably fl 1,i <0.4…0.5cl 1,i holds. In step 402, the second characteristic length fl2,i Calculate another output signal O2(x i ), where this second characteristic length is different from the first characteristic length and preferably fl 2,i ≥cl 2,i holds. In an optional step 404, another output signal O3(x i ) is calculated using a characteristic length different from the characteristic lengths in other output signals that use further different characteristic lengths. Step 404 can be repeated to obtain more output signals s. The steps performed in steps 400 to 404 correspond to what was explained above with reference to Figures 1 to 3 .

[0241] In step 406, the system response is estimated for each output signal O k (x i ).

[0242] In step 408, multi-image deconvolution is performed using the multiple output signals calculated in steps 400 to 404 and the estimated system response of step 406 to obtain a deconvolved output signal J(x i ).

[0243] In Figures 5 to 9 , resolution enhancement by baseline removal is demonstrated for a real image of Paramecium.

[0244] Figure 5 The input signal I(x i ) is shown, which is the input image and was recorded using a DMi8 wide-field microscope with an HCX FLUOTAR 100x / 1.30 OIL lens, a numerical aperture of 1.3, and a peak emission wavelength of 520 nm.

[0245] Figure 6 The output signal O(x i ) is shown, which is the output image where a characteristic length fl of 80 nm, i.e., fl = 0.4cl, has been used for the regularization length scale in the penalty term, which, as described above, includes the square of the first derivative of the baseline estimate. The improvement in resolution enhancement is clearly visible.

[0246] Figure 7 And Figure 8 respectively show Figure 5 and Figure 6 Details VII and VIII. It can be seen that an important part of the noise component I2(x i ), regarded here as blur, has been removed in the output signal.

[0247] This is confirmed by Figure 9 , which respectively show Figure 9 ​Figure 7 and Figure 8 the intensity distribution of 700 along the line in. It can be seen that by removing the baseline estimate f(x i ), the intensity peaks representing the content are preserved and are more clearly visible in the output signal O(x i ).

[0248] Figure 10 shows the result of multi-image deconvolution, where the output signal o(x Figure 6 shown in is used as the first input. As the second input for multi-image deconvolution, a characteristic length corresponding to the characteristic length of the system response, fl = cl, is used to calculate the second output signal. Then, these images are subjected to multi-image deconvolution, as described above, to obtain the deconvolved output signal J(x i ). The obtained result is improved because the signal-to-noise ratio does not decrease significantly as it does when using an output signal with a small characteristic length. This can be seen in i which shows the details XI of Figure 11 . Figure 10

[0249] As a general note for baseline removal and multi-image deconvolution applications, the dimensionality of the data may be changed by rearranging the array. For example, two-dimensional data can be presented as one or more sets of one-dimensional data. This can be achieved by stringing subsequent rows or columns together. Additionally, three-dimensional data can be reduced to two-dimensional data by stringing subsequent planes together one by one. By recursively using this principle, any N-dimensional data can be reduced to one-dimensional or two-dimensional data to which the above-described scheme can be applied.

[0250] Conversely, any one-dimensional array can be arranged as a two-dimensional or higher-dimensional array by simply decomposing it into smaller one-dimensional arrays and indexing those smaller arrays, preferably of the same length, in a two-dimensional or higher-dimensional scheme. Additionally, any type of data can be regarded as and displayed as image data or an image, for example, as described above, by assigning a gray intensity to each value of the input signal data and displaying it in a two-dimensional or three-dimensional arrangement.

[0251] As used herein, the term "and / or" includes any and all combinations of one or more of the related listed items and may be abbreviated as " / ".

[0252] ​Although some aspects have been described in the context of apparatuses, it is clear that these aspects also represent a description of corresponding methods, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of corresponding blocks or items or features of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some one or more of the most important method steps may be performed by such an apparatus.

[0253] According to certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. A non-transitory storage medium such as a digital storage medium like a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory may be used to perform the implementation, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer-readable.

[0254] Some embodiments according to the present invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0255] Generally, embodiments of the present invention may be implemented as a computer program product having program code that is operable to perform one of the methods when the computer program product is run on a computer. For example, the program code may be stored on a machine-readable carrier.

[0256] Other embodiments include a computer program stored on a machine-readable carrier for performing one of the methods described herein.

[0257] In other words, embodiments of the present invention are thus a computer program having program code that is used to perform one of the methods described herein when the computer program is run on a computer.

[0258] Thus, a further embodiment of the present invention is a storage medium (or data carrier, or computer-readable medium) having stored thereon a computer program for performing one of the methods described herein when executed by a processor. The data carrier, digital storage medium, or recording medium is generally tangible and / or non-transitory. A further embodiment of the present invention is an apparatus as described herein, including a processor and a storage medium.

[0259] Thus, a further embodiment of the present invention is a data stream or a signal sequence that represents a computer program for performing one of the methods described herein. The data stream or signal sequence may be configured, for example, to be transmitted via a data communication connection, such as via the Internet.

[0260] Further embodiments include a processing device, e.g., a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0261] Further embodiments include a computer having installed thereon a computer program for performing one of the methods described herein.

[0262] A further embodiment according to the invention includes a device or system configured to transmit (e.g., electronically or optically) to a receiver a computer program for performing one of the methods described herein. For example, the receiver can be a computer, a mobile device, a storage device, etc. For example, the device or system can include a file server for transmitting the computer program to the receiver.

[0263] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0264] Reference numeral

[0265] 1 Device

[0266] 2 Observation device

[0267] 2a Microscope

[0268] 4 Recording system

[0269] 6 Input signal data I(x i )

[0270] 8 Camera

[0271] 9 Image sensor

[0272] 10 Channel

[0273] 12 Object

[0274] 13 Detector volume

[0275] 14 Field of view

[0276] 15 Fluorophore

[0277] 16 Illumination system

[0278] 17 Objective lens

[0279] 20 Storage section

[0280] 22 CPU

[0281] 24 Computing device

[0282] 26 GPU

[0283] 28 Signal input section

[0284] 30 Connection device of the signal input section

[0285] 32 Signal output section

[0286] 34 Connection device of the signal output section

[0287] 36 Output signal data O(x i )

[0288] 37 Display

[0289] 38 Signal processor

[0290] 40 Baseline removal section

[0291] 42 Baseline estimator engine

[0292] 44 Baseline estimation f(x i )

[0293] 46 Quadratic or semi - quadratic minimization engine

[0294] 48 First iteration stage

[0295] 50 Second iteration stage

[0296] 60 Setting of baseline estimation parameters

[0297] 62 Graphical user interface

[0298] 64 Initialize the minimization engine and / or scheme

[0299] 66 Quadratic or semi - quadratic minimization scheme

[0300] 68 Convergence criterion

[0301] 70 Calculation of output signal data

[0302] 72 Post - processing operation

[0303] 400 - 404 Deblurring

[0304] 406 Estimate system response

[0305] 408 Multi - image deconvolution

[0306] 700 Line

Claims

1. A signal processing apparatus (1) for enhancing a digital input signal (I(x i )) recorded by a recording system (16) having a system response (H(x i ))), wherein the apparatus is configured to - Retrieve the digital input signal; - Calculate a baseline estimate (f(x i )) of the digital input signal, the baseline estimate representing a baseline of the digital input signal and including spatial features of the digital input signal that are greater than a characteristic length (fl); - Remove the baseline estimate from the digital input signal to obtain an output signal (O(x i )) that includes spatial features smaller than the feature length; wherein the device is configured to - Retrieve the characteristic length (cl) of the system response (H(x i )); and - Using a characteristic length less than the system response (H(x i )) for at least the characteristic length of the input signal A subset is fitted to compute the baseline estimate (f(x i )) 2. The signal processing device (1) according to claim 1, wherein the characteristic length (fl) is less than half of the characteristic length (cl) of the system response (H(x i )) 3. The signal processing device (1) according to claim 1, wherein the device (1) is configured to - Calculate a plurality of output signals (O i )) from the digital input signal (I(x 1…K (x i )); - Calculate each of the plurality of output signals based on a baseline estimate (f(x i )) having different characteristic lengths (fl); - Estimate the system response (H(x 1…K (x i )) for each of the plurality of output signals (O i )); - Performing multi-image deconvolution of the plurality of output signals (O 1…K (x i )) to obtain a deconvolved output signal (J(x i )) 4. The signal processing device (1) according to claim 3, wherein a subset of the plurality of output signals (O 1…K (x i )) is determined based on a characteristic length (fl) greater than the characteristic length of the system response (H(x i )) 5. The signal processing device (1) according to claim 3 or 4, wherein a subset of the plurality of output signals (O 1…K (x i )) is determined based on a characteristic length (fl) equal to the characteristic length (cl) of the system response (H(x i )) 6. The signal processing device (1) according to any one of claims 1 to 4, wherein the device is configured to - Applying different feature lengths (fl) in at least two of the dimensions of the digital input signal (I(x i ))).

7. The signal processing device (1) according to any one of claims 1 to 4, wherein, The characteristic length (cl) of the system response (H(x i )) is different in at least two dimensions (i) of the digital input signal (I(x i )) 8. The signal processing device (1) according to any one of claims 1 to 4, wherein the characteristic length (fl) is included in the least squares minimization criterion (M(f(x i ))) that includes the baseline estimation.

9. The signal processing device (1) according to claim 8, wherein the least squares minimization criterion (M(f(x i ))) includes a penalty function (P(f(x i ))), and the penalty function includes the characteristic length (fl).

10. The signal processing device (1) according to claim 8, wherein the least squares minimization criterion (M(f(x i ))) comprises a combination of the characteristic length (fl) and at least one derivative of the baseline estimate (f(x i )) .

11. The signal processing device (1) according to any one of claims 1 to 4, wherein the characteristic length (fl) is included in the regularization length scale (γ j ).

12. The signal processing device (1) according to any one of claims 1 to 4, wherein the device (1) is configured to perform a quadratic minimization scheme (66) having two iterative levels (50, 60).

13. An observation device, comprising the signal processing device (1) according to any one of claims 1 to 12.

14. A signal processing method for enhancing a digital input signal (I(x i )) recorded by a recording system (4) having a system response (H(x i ))), the signal processing method being configured to operate a signal processing device (1) according to one of claims 1 to 12 or to operate an observation device according to claim 13, the system response (H(x i )) having at least one characteristic length (cl) in at least one dimension (i), wherein spatial features having a feature length (fl) greater than the at least one characteristic length of the system response (H(x i )) are removed from the digital input signal to obtain an output signal (O(x i ))).

15. The signal processing method according to claim 14, wherein at least one additional output signal (O2(x i )) is calculated, and wherein calculating the at least one additional output signal includes removing from the digital input signal a spatial feature having a feature length (fl) equal to at least one characteristic length (cl) of the system response (H(x i )) or greater than at least one characteristic length (cl) of the system response (H(x i )) of one kind, and wherein multi-image deconvolution is calculated from the output signal (O(x i )) and the at least one additional output signal O2(x i )) to obtain a deconvolved output signal (J(x i ).

16. The signal processing method according to claim 14 or 15, wherein a regularization length scale (γ j ) depending on the characteristic length (fl) is used to calculate a baseline estimate (f(x i )) 17. A computer program product having program code for performing the method according to any one of claims 14 to 16 when the computer program is run on a processor.

18. A non-transitory computer-readable medium storing a computer program that causes a computer to execute the signal processing method according to any one of claims 14 to 16.

19. A method for enhancing a system having a response (H(x i )) of the recording system (16) to record the digital input signal (I(x i )) of a neural network device, wherein the system response (H(x i )) having at least one characteristic length (cl) in at least one dimension (i); Among them, The neural network device includes a memory and a processor, where the processor is configured to generate an output signal (O(x i ))), where the output signal has a spatial feature of a characteristic length (fl) that is less than at least one characteristic length of the system response (H(x i ))), and a spatial feature of a characteristic length (fl) that is greater than at least one characteristic length of the system response (H(x i )) and is included in the digital input signal is removed.

Images