Data processing apparatus and method for correcting image shading

Abstract

The invention relates to a data processing device for imaging equipment. The data processing device is configured to obtain a length threshold value; accessing input image data representing the at least one digital input image, where the input image data includes a shadow signal representing a decrease in brightness towards an edge of the at least one digital input image and a content signal representing an image feature of the at least one digital input image having a length less than a length threshold; calculating a baseline image based on the input image data and the length threshold, wherein the baseline image represents an estimate of the shadow signal; at least one digital output image representing an estimate of the content signal is generated by at least one of a) subtracting the baseline image from the input image data, and b) dividing the input image data by the baseline image. The data processing device allows shading correction without acquiring any individual background image or reducing the image size. The invention also relates to a computer-implemented method and a method for operating the imaging device.

Description

Technical Field

[0001] The invention relates to a data processing device for an imaging device, such as a microscope, an endoscope, a scanner, a camera, a photon counter or any other type of scientific and medical observation device. The invention also relates to a computer-implemented method and a method for operating such an imaging device. Furthermore, the invention relates to a computer program and a computer-readable medium. Finally, the invention relates to a neural network device trained by the data processing device or the computer-implemented method. Background Art

[0002] When capturing images of objects (eg samples) in medicine and other scientific disciplines, lens vignetting and uneven illumination often cause so-called shading effects, which reduce the image quality.

[0003] A known method to overcome the shading effect is to acquire a background image using an empty sample holder. The background image can then be used to correct the actual image of the sample. This shading correction is very tedious, as the user must repeat it for each optical objective and contrast method. Furthermore, if the illumination changes slightly, the entire process must be repeated.

[0004] Another approach is to reduce the image size so that only the well-illuminated central area of ​​the field of view is acquired. However, due to the smaller image size, more images need to be taken to cover the same area of ​​interest, which is very time-consuming. In addition, continuously illuminating the sample longer than necessary can cause bleaching or even damage living samples.

[0005] Therefore, there is a need to conveniently apply shading correction to images in the field of medicine and other scientific disciplines. Summary of the invention

[0006] It is therefore an object of the present invention to provide means for improving the image quality in general and for overcoming shadowing effects in particular with less effort and time.

[0007] The object is achieved by a data processing device for an imaging device, the device being configured to: obtain a length threshold; access input image data representing at least one digital input image, wherein the input image data includes a shadow signal and a content signal, wherein the shadow signal represents a decrease in brightness toward an edge of the at least one digital input image, and the content signal represents an image feature of the at least one digital input image, the image feature having a length less than the length threshold; calculate a baseline image based on the input image data and the length threshold, wherein the baseline image represents an estimate of the shadow signal; generate at least one digital output image representing an estimate of the content signal by at least one of the following methods: a) subtracting the baseline image from the input image data, and b) dividing the input image data by the baseline image.

[0008] Here, the length of each image feature is a length greater than zero in the input image domain, for example given as the number of length units or the number of pixels in a direction in which the corresponding image feature extends. The length threshold may be obtained as a direct user input or may be obtained by calculation, as will be described in further detail below.

[0009] The data processing device of the present invention is advantageous and achieves the above-mentioned objects due to the following reasons:

[0010] Shadow effects caused by lens vignetting and uneven illumination result in a non-uniform distribution of brightness in the image, where brightness is the relative image intensity. The non-uniform brightness distribution tends to follow a definite pattern, with brighter centers and darker edges. This pattern usually extends over a greater distance than the length of the image features. Furthermore, this pattern is reflected in the shadow signal mentioned above.

[0011] As will be described in further detail below, the baseline image calculated based on the input image data and the length threshold can be considered to represent mainly an approximation of the shadow signal excluding the content signal. This is because the baseline image is calculated based on the length threshold, which is greater than the length of the image features contained in the content signal. Therefore, by calculating the baseline image based on the length threshold, the baseline image can be processed to represent only certain patterns of changes in image brightness of at least one digital input image, which changes occur gradually over a distance greater than the length threshold.

[0012] In other words, the baseline image can be viewed as a rough “brightness map.” This “brightness map” can be used to identify which parts of at least one input image are too dark due to shadow effects.

[0013] Therefore, the estimated pattern contained in the baseline image can be used for shading correction in a subtractive or divisive manner to make dark areas of at least one digital input image brighter in at least one digital output image and / or to make bright areas of at least one digital input image darker in at least one digital output image. As a result, in at least one digital output image, the image brightness is effectively homogenized and the shading effect is minimized or at least mitigated.

[0014] Advantageously, the data processing means operates without acquiring any separate background image or reducing the image size.

[0015] The initial purpose is also achieved by a computer-implemented method for processing input image data of an imaging device, the method comprising the following steps: obtaining a length threshold; accessing input image data representing at least one digital input image, wherein the input image data includes a shadow signal and a content signal, the shadow signal representing a decrease in brightness toward an edge of the at least one digital input image, and the content signal representing an image feature of the at least one digital input image, the image feature having a length less than the length threshold; calculating a baseline image based on the input image data and the length threshold, wherein the baseline image represents an estimate of the shadow signal; generating at least one digital output image representing an estimate of the content signal by at least one of: a) subtracting the baseline image from the input image data, and b) dividing the input image data by the baseline image.

[0016] In other words, the method may include the following steps: retrieving at least one digital input image containing image features, the size of each image feature being characterized by a length; obtaining a length threshold, the length threshold representing a threshold greater than the length of all image features; estimating a baseline image based on the at least one input image and the length threshold, the baseline image representing a pattern of changes in image brightness, wherein the changes occur over a distance greater than the length threshold; calculating at least one digital output image by at least one of: a) subtracting the baseline image from the input image data, and b) dividing the input image data by the baseline image.

[0017] The method has the same advantages as explained for the data processing device of the present invention. In particular, the method can be performed simultaneously with the imaging process or after the imaging process.

[0018] The at least one digital input image may be, for example, a one-dimensional, two-dimensional or three-dimensional image or an image of higher dimensions. In particular, the digital input image may originate from a line scan camera, an area scan camera, a rangefinder camera, a stereo camera, a microscope camera, an endoscopic camera, etc. The digital input image may be represented by input image data at a plurality of input image locations (e.g. pixels or voxels). The input image data may be, for example, discrete intensity data and / or discrete color data.

[0019] Conversely, the input image data is best expressed as an n-dimensional array I(X i ), where N is an integer greater than or equal to 1. i ) can be position X i Any value or combination of values ​​at , such as a value representing the intensity of a color or "channel" in a color space, such as the intensity of the color R in an RGB color space, or a value representing the combined intensity of multiple colors, such as

[0020] As will be described in further detail below, a baseline image may be calculated using a fit to the input image data. Computationally, the result of this fit (i.e., the baseline image) is represented by discrete baseline image data at a plurality of baseline image locations (e.g., pixels or voxels). The baseline image data may be an n-dimensional array F having the same dimensions as the input image data. b (X i ). Therefore, for one-dimensional input image data, the baseline image can be presented as a discrete curve graph, and for two-dimensional input image data, the baseline image can be presented as a discrete surface graph.

[0021] At least one digital output image may include an n-dimensional array O(X) at a plurality of output image locations (eg, pixels or voxels). i ) represents the output image data.

[0022] Item X i is a tuple containing m position values ​​{X1; ...; X M}, which represents the discrete position X in the array representing the input image data. i Or the positioning vector to that position. Position X i It can be represented by pixels or preferably a coherent set of pixels in the input image data. For example, a discrete position X i represents a single position variable {X1} in the case of one-dimensional input image data, a pair of discrete position variables {X1; X2} in the case of two-dimensional input image data, and a triplet of discrete position variables {X1; X2; X3} in the case of three-dimensional input image data. In the i-th dimension, the array can contain M i positions, i.e. I(X i ) can contain a total of (m1×...×m n ) elements. In the following, no specific position or specific dimension will be mentioned, but only X i to represent a position in a digital input image, a digital output image, or a baseline image.

[0023] As described above, the input image data may consist of a shading signal and a content signal, wherein the baseline image mainly represents the shading signal. That is, the above-described data processing apparatus functions particularly well for applications in which the content signal may be assumed to have a high spatial frequency, e.g., it is responsible for intensity and / or color variations occurring over short distances in the digital input image, while the shading signal may be assumed to have a low spatial frequency, i.e., it results in mainly gradual intensity and / or color variations extending over a relatively large area of ​​the digital input image.

[0024] Due to its low spatial frequency, the shadow signal can be considered to follow a more or less smooth baseline. The baseline image is an estimate of the baseline in the intensity data and / or color data. Therefore, the baseline image represents an approximation of the shadow signal.

[0025] Under this assumption, the length threshold can be expressed as the cutoff frequency ω cutoff , where the spatial frequencies represented in the input image data are higher than the cutoff frequency ω cutoff The structure of is considered to constitute the content signal, and the spatial frequencies represented in the input image data are lower than the cutoff frequency ω cutoff In other words, the length threshold can be used to distinguish whether a structure is considered to belong to the content signal or the shadow signal.

[0026] Alternatively, the length threshold may be given as a predetermined threshold length λ, which represents the spatial extent of the largest or average image feature of interest along a single dimension. All structures smaller than the length threshold are considered to constitute the content signal, while structures larger than the length threshold are considered to be part of the shadow signal.

[0027] In addition, alternatively, the length threshold may also be expressed as the square root of the number of pixels (preferably the square number) occupied by the largest or average image feature of interest in at least one digital input image. For example, if the largest or average image feature of interest occupies 100 pixels, the length threshold may be 10.

[0028] The data processing device and method can be further improved by adding one or more features described below. Each of these features can be added to the method and / or data processing device independently of other features. In particular, a person skilled in the art (having knowledge of the data processing device of the present invention) can configure the method of the present invention so that the method of the present invention can operate the data processing device of the present invention. In addition, each feature has its own advantageous technical effects, as described below.

[0029] The accessed input image data may be recorded in real time by the imaging device. Alternatively, the accessed input image data may be retrieved, received or read from a storage medium. This allows shading correction to be performed during and after the imaging process.

[0030] At least one digital input image may be a transmitted light microscopy image. In a transmitted light microscopy image, lens vignetting may cause a shadow effect resulting in a decrease in brightness towards edges and corners. In such an application, the data processing apparatus and method may effectively mitigate the shadow effect.

[0031] Alternatively or additionally, at least one digital input image may be a fluorescence image. In particular, at least one digital input image may be a photon counting image. Such a photon counting image represents photon count values ​​obtained from a photon detector. In the input image data, these photon count values ​​are assigned to corresponding image locations, such as pixels. Since the image intensity and illumination of the photon counting image are multiplicatively related, it is preferred to generate at least one digital output image by dividing the input image data by a baseline image during a shading correction process.

[0032] Dividing the input image data by the baseline image can be done by dividing the input image data by the inverse of the baseline image F b -1 (X i ), where the inverse of the baseline image is, for example, the inverse color image of the baseline image:

[0033] O(X i )=I(X i )·F b -1 (X i )

[0034] Alternatively, a pixel-by-pixel division can be performed on all pixels:

[0035]

[0036] Optionally, the baseline image may be normalized to 1 before the pixel-by-pixel division. That is, in the normalized baseline image, the brightest pixel of the baseline image may be assigned a value of 1. All values ​​of the remaining pixels in the normalized baseline image may be obtained by performing linear interpolation, bilinear interpolation, nearest neighbor interpolation, Lanczos interpolation, or an AI-based interpolation algorithm on the pixel values ​​in the baseline image. Therefore, the dark pixels of the normalized baseline image contain values ​​less than 1, which after the pixel-by-pixel division will result in the corresponding pixels in at least one digital output image being brighter than in at least one digital input image.

[0037] Subtracting the baseline image from the input image data (which is an alternative to the division method described above) can be done by adding the inverse of the baseline image to the input image data:

[0038] O(X i )=I(X i )+F b -1 (X i )

[0039] Alternatively, a pixel-by-pixel subtraction approach can be used:

[0040]

[0041] As described above, this will result in brightness homogenization as bright areas of the at least one digital input image become darker in the at least one digital output image. Optionally, the maximum intensity value of the baseline image may be added to the entire digital output image after the pixel-by-pixel subtraction to increase the overall brightness of the at least one digital output image.

[0042] If the length threshold depends on the size of the at least one digital input image, the shading correction will produce good results. For example, the apparatus may be configured to calculate the length threshold based on the image size. In particular, the length threshold may be proportional to the image size. Preferably, the length threshold is 5%, more preferably 10%, most preferably 20% of the image size, wherein the image size is the width of the at least one digital input image in one of the image directions measured in pixels or in length.

[0043] As briefly mentioned above, the baseline image may be calculated using a fit to the input image data, such as a polynomial fit or a spline fit. To this end, the apparatus may be configured to calculate the baseline image using a fit to the input image data, wherein the fit is calculated by an iterative minimization scheme. Advantageously, using an iterative minimization scheme facilitates implementation of the scheme as compared to analytical methods.

[0044] Optionally, the iterative minimization scheme may include a least squares minimization criterion, which is to minimize the fit. Therefore, the iterative minimization scheme has a specifically defined objective function. The least squares minimization criterion may include a penalty term and a cost function. In particular, the least squares minimization criterion M(F b (X i )) can have the following forms:

[0045] M(F b (X i ))=P(F b (X i ))+C(F b (X i ))

[0046] Where P(F b (X i )) is the penalty term, C(F b (X i )) is the cost function, as will be explained later.

[0047] The least squares minimization criterion can include a penalty term P(F b (X i )) to ensure that the baseline image data F b (X i) accurately represents only the shadow signal and avoids fitting the baseline image data to the content signal. In particular, the penalty term may take any form as long as it introduces a penalty when the baseline image data is fitted to data that is considered to belong to the content signal. If the content signal is represented in the baseline image data, such a penalty may be created by increasing the value of the penalty term.

[0048] Assuming that the shadow signal has a lower spatial frequency, the penalty term may be a term that becomes larger as the spatial frequency of the baseline image becomes larger. In one embodiment, such a term may be a roughness penalty term that penalizes non-smoothed baseline image data that deviates from the smooth baseline. Such a roughness penalty term effectively penalizes fitting of data with high spatial frequencies.

[0049] For example, deviations from a smooth baseline may result in at least one of a first-order derivative (i.e., steepness or gradient) and a second-order derivative (i.e., curvature) of the baseline image data having a larger value. Therefore, the roughness penalty term may include at least one of a first-order spatial derivative of the baseline image (especially the square and / or absolute value of the first-order spatial derivative) and a second-order derivative of the baseline image (especially the square and / or absolute value of the second-order spatial derivative). In general, the penalty term may include any order of spatial derivatives of the baseline image represented by the baseline image data. More generally, the penalty term may include any linear combination of spatial derivatives of the baseline image data representing the baseline image.

[0050] Without loss of generality, for the one-dimensional case, the penalty term P(F b (X i ))For example, it can have the following form:

[0051]

[0052] where r is the regularization parameter, is a discrete operator used to compute the first-order derivative. The utility of the regularization parameter r will be explained further below.

[0053] More generally, the penalty term P(F b (X i )) can have the following forms:

[0054]

[0055] in is a discrete operator for calculating the j-th order derivative. For multidimensional digital input images, different penalty terms P(F b (X i )).

[0056] The regularization parameter r is used as a multiplicative weighting factor in the penalty term. Therefore, the unit of the regularization parameter r can be selected so that the penalty term produces a dimensionless scalar. In addition, the regularization parameter r can be selected based on a length threshold. For example, the regularization parameter r can be equal to the above threshold length λ:

[0057] r=λ

[0058] For larger length thresholds, the penalty term will be amplified by the weighting factor; conversely, for smaller length thresholds, the effect of the penalty term will be attenuated by the weighting factor. Due to the amplification and attenuation of the penalty term, the baseline image will be forced to be smoother in the former case, while it will have a certain degree of roughness in the latter.

[0059] Additionally or alternatively, a length threshold may be included in the penalty term as an additive weighting constant.

[0060] Instead of a multiplicative weighting factor or an additive weighting constant, a length threshold can also be used to select the best polynomial order K for the above polynomial fitting. That is, the baseline image data can be represented by a K-order polynomial in any dimension i among the n dimensions:

[0061]

[0062] where a i,k are the coefficients of the polynomial in the i-th dimension. For each dimension i=1,…,n, a separate polynomial can be calculated. The optimal value of the maximum polynomial order K depends on the required smoothness of the baseline image data. For a smooth baseline (i.e., a large length threshold), the polynomial order K must be set as low as possible, while fitting a highly irregular background (i.e., a small length threshold) may require a higher order K.

[0063] In the case of polynomial fitting, the baseline image data can be obtained by simply using the polynomial coefficients a i,k Or the intensity data of a graph representing a K-th order polynomial.

[0064] The least squares minimization criterion can also include a cost function C(F b (X i )), which represents the input image data I(X i ) and the baseline image data F b (X i ) is the difference between the baseline image represented by . An example of a cost function in the one-dimensional case is:

[0065] C(F b (X i )=‖I(X i )-F b (X i )‖

[0066] Where ‖…‖ represents the L1 norm, that is, the sum of absolute values. For multi-dimensional cases, the sum of the root mean square values ​​of the sum of the squared differences between the input image data and the baseline image data in the i-th dimension in all dimensions can be used instead of the L1 norm.

[0067] Shadow signals usually do not contain peaks, while content signals are likely to contain peaks. Therefore, the cost function C(F b (X i )) may be a truncated difference term to avoid fitting the baseline image data to the content signal. To achieve this, the data processing device may be configured to obtain a predetermined peak threshold s from an external user input, which is greater than zero and defines at what peak height the difference term is truncated.

[0068] The truncated difference term may be symmetrical or asymmetrical. Preferably, the truncated difference term may be a term representing the input image data I(X i ) and baseline image data F b (X i ), where if the difference between the input image data and the baseline image data is greater than the peak threshold s, the output value of the truncated quadratic term is limited to a constant value. Otherwise, the value of the truncated quadratic term is equal to the square of the difference between the input image data and the baseline image data. Thus, the baseline image data will follow the high intensity peaks of the content signal only to a limited amount, and will follow the shallow intensity peaks of the shadow signal more. The truncated quadratic term It can be in the following forms:

[0069]

[0070] Using truncated quadratic terms Cost function C(F b (X i )) can be expressed as:

[0071]

[0072] Optionally, the above iterative minimization scheme may include a first iterative stage and a second iterative stage, wherein the first iterative stage and the second iterative stage together represent an iterative step or an iterative cycle.

[0073] For example, the data processing apparatus may be configured to calculate the baseline image using an iterative semi-quadratic minimization scheme aiming to minimize a least squares minimization criterion. The semi-quadratic minimization scheme may, for example, comprise at least a portion of the computationally efficient LEGEND algorithm. The LEGEND algorithm is described in the following articles: "Convex Half-Quadratic Criteria and Interacting Variables for Image Restoration" by Idier J. (2001) in IEEE Transactions on Image Processing, 10(7), pp. 1001-1009; and "Background Removal from Spectra by Designing and Minimizing a Non-Quadratic Cost Function" by Mazet V., Carteret C., Bire D, Idier J., and Humbert B. (2005) in Chemometrics and Intelligent Laboratory Systems, vol. 76, pp. 121-133. Both articles are incorporated herein by reference in their entirety.

[0074] The LEGEND algorithm introduces a method that preferably has the same i ) Discrete auxiliary data D of the same dimension l (x i ). These auxiliary data are updated in each iteration cycle based on the input image data and the latest baseline image data representing the baseline image, preferably in the first iteration phase based on the input image data and the latest baseline image data representing the baseline image. In particular, the auxiliary data are updated using a variant of the truncated quadratic term as shown below:

[0075]

[0076] Where l=1...L is the index of the current iteration cycle, and α is a constant value, for example, 0.493.

[0077] In the second iteration phase, the baseline image data may be updated according to the previously calculated updated auxiliary data, the baseline image data from the previous iteration cycle, and the penalty term using the following formula:

[0078]

[0079] Alternatively, in this second iteration stage, the discrete Green's function G(X i ) and the input image data I(X i ) and updated auxiliary data D l (X i ) is convolved with the sum of the two to calculate the baseline image data representing the baseline image. In other words, the second iteration stage of the LEGEND algorithm can be replaced by the following iteration steps, where the Green function G(X i ) Calculate the updated baseline image data F in the lth iteration cycle b,l (X i ):

[0080] F b,l (X i )=G(X i )*(I(X i )+D l (X i ))

[0081] Without loss of generality, for the one-dimensional case, the Green function G(X i ) can have the following forms:

[0082]

[0083] in, is the discrete Fourier transform, is the inverse discrete Fourier transform, r is the regularization parameter, is the function derivative.

[0084] Compared to the conventional LEGEND algorithm, this step significantly reduces the computational burden. The reduced computational burden is due to the calculation of the convolution. This calculation can be performed efficiently using the FFT algorithm. In addition, due to the use of the FFT algorithm, the second iteration step can fully utilize an array processor, such as a graphics processing unit (GPU) or FPGA.

[0085] As a starting step in the iterative minimization scheme, an initial set of baseline image data (e.g., F b,1 (X i )=I(X i )) and an initial set of auxiliary data (e.g. D1(X i )=0). Then, the first and second iteration stages are preferably repeated until a convergence criterion is satisfied. A suitable convergence criterion may be, for example, the convergence of the current baseline image data with all positions X. i The sum of the differences between the previous baseline image data on the image is less than a predetermined convergence threshold.

[0086] The calculation of the baseline image itself is not limited to the above-described procedure and can be accomplished by nearly any known method for estimating a baseline that correctly represents the shading signal of a given digital image.

[0087] According to a feasible embodiment of the data processing device, the input image data can represent multiple digital input images. Here, each individual digital input image is represented by a different subset of the input image data. In particular, the multiple digital input images can be partial images in a tile scan and / or a Z-axis image stack (Z-stack). In a tile scan, multiple images showing adjacent areas are stitched together. The shadow effect in each image produces a checkerboard pattern that can be seen in overlapping areas, making the image look unprofessional or unusable for publication. In a Z-axis image stack, multiple images showing different layers of the sample are stacked. Since each image is taken with a different objective lens and / or illumination setting, each layer displays a slightly different shadow effect.

[0088] Advantageously, when the data processing device is configured to calculate a baseline image for each of the plurality of digital input images and to calculate an average baseline image based on the baseline images of the plurality of digital input images, the data processing device allows shading correction to be performed for both cases. The average baseline image may be calculated as an arithmetic, geometric or quadratic mean by interpolating the baseline images of the plurality of digital input images as a median. Averaging makes the data processing device less prone to outliers.

[0089] Furthermore, the data processing apparatus may be configured to generate a plurality of digital output images by at least one of: a) subtracting an average baseline image from the input image data, and b) dividing the input image data by the average baseline image. In particular, one output image is generated for each of the plurality of digital input images. The generation of the output images may be implemented as a plurality of image-by-image subtractions / divisions or as a single subtraction / division operation applied to the entire input image data. In the former case, each subset of the input image data is subtracted / divided by the same baseline image. In the latter case, the average baseline image is replicated for each subset. If a tile scan is involved, the replicates are stitched together; if a Z-axis image stack is involved, the replicates are stacked.

[0090] According to another embodiment, the apparatus may be configured to first calculate average input image data from a plurality of digital input images, and then calculate an average baseline image based on the average input image data and a length threshold. The average input image data may be image data representing a median, interpolation, arithmetic mean, geometric mean, or quadratic mean of the plurality of digital input images. The plurality of digital output images may then be generated by at least one of: a) subtracting the average baseline image from the input image data, and b) dividing the input image data by the average baseline image. This embodiment requires a smaller number of total computational steps because the baseline image is only computed once in the form of an average baseline image.

[0091] According to another aspect, the data processing device may include a microprocessor, or may be configured as a microprocessor or as part of such a microprocessor. The data processing device and / or the microprocessor may include a memory, such as a combination of a read-only memory (ROM) and any number of random access memories (RAM).

[0092] Alternatively, the data processing device may be part of a computer separate from the imaging apparatus.In particular, the data processing device and the computer-implemented method may be implemented in a personal computer, a single board computer and / or a cloud computing network.

[0093] The data processing device may also be an embedded processor. An embedded processor is a small computer or computer chip that can be embedded in various machines and devices to control the electrical and mechanical functions therein.

[0094] Alternatively, the data processing means may be part of such an embedded processor. This also allows the data processing means to be integrated into an imaging device where a baseline image needs to be calculated.

[0095] In particular, the embedded processor may be an embedded processor of / for a microscope, in particular a confocal microscope, a light sheet microscope or a fluorescence microscope. Advantageously, the inventive concept may therefore be applied in the field of microscopy.

[0096] Therefore, the initial object is also achieved by a microscope and any other type of imaging device comprising a data processing apparatus according to one of the above embodiments, wherein the imaging device is configured to record input image data representing at least one digital input image. In addition to a microscope, the imaging device can also be an endoscope, a scanner, a camera, a photon counter and / or a photon detector.

[0097] The imaging device benefits from the advantages of the above-mentioned data processing apparatus. In particular, the calculation of the baseline image can be performed in real time or retrospectively according to the needs of the user.

[0098] Likewise, the initial object is achieved by a method of operating an imaging device, the method comprising the steps of recording input image data representing at least one digital input image and carrying out a computer-implemented method. As a result, at least one shading-corrected output image can be obtained with little effort.

[0099] According to another aspect, the method of the present invention can be adapted to operate a data processing device according to any of the above embodiments. Advantageously, this allows the method of the present invention to be implemented on hardware dedicated to a specific purpose. In particular, the method of the present invention can be executed on an embedded processor of a microscope.

[0100] Alternatively, the method of the present invention may also be applicable to operating a general purpose computer. Thus, the applicability of the method of the present invention is improved. Therefore, the initial object is achieved by a computer program product comprising a program code, which, when executed by a computer (e.g., a general purpose computer), causes the computer to implement the method of the present invention. Likewise, the initial object is also achieved by a computer readable medium comprising a program code, which, when executed by a computer, causes the computer to implement the method of the present invention. Both the computer program product and the computer readable medium are advantageous and achieve the initial object because they represent means for implementing the method of the present invention.

[0101] The at least one digital output image is not necessarily the final result for the user. For example, the shading correction according to the invention can be used as a pre-processing step. That is, further data processing steps, such as image deblurring, can be performed on the at least one digital output image. In particular, different length thresholds can be used to calculate another baseline image to approximate the out-of-focus component I in the content signal. 失焦 , as described in European Patent EP3571663B1, which is incorporated herein by reference in its entirety.

[0102] The resulting final image can then be calculated as follows:

[0103]

[0104] The initial object of the present invention is also achieved by a neural network device trained by a plurality of digital input images and a plurality of digital output images, wherein the digital output images are calculated from the digital input images by a data processing device or by a computer-implemented method. Thus, the neural network device is capable of generating digital output images based on the digital input images. Advantageously, since the neural network device is trained by pairs of different digital input images and digital output images, the neural network device is capable of emulating the calculations of the data processing device or the computer-implemented method.

[0105] The initial object is also achieved by a method for training a neural network device by pairs of different digital input images and digital output images, wherein the digital output images are calculated from the digital input images by means of a data processing device or by a computer-implemented method. The training method is advantageous because it allows the creation of a neural network device having the above-mentioned characteristics.

[0106] The data processing device and each of its functions may be implemented in hardware, software, or a combination of hardware and software. For example, at least one function of the data processing device may be implemented at least in part by a subroutine, a portion of a general-purpose processor (e.g., a CPU), and / or a special-purpose processor (e.g., a GPU, FPGA, a vector processor, and / or an ASIC). BRIEF DESCRIPTION OF THE DRAWINGS

[0107] The invention will now be described using example embodiments, which are also shown in the drawings. In the drawings, the same reference numerals are used to indicate features that correspond to each other at least in function and / or design.

[0108] The feature combinations shown in the attached embodiments are for illustrative purposes only and may be modified. For example, features of the embodiments that have technical effects that are not required for a particular application may be omitted. Similarly, if a particular application requires a technical effect associated with a feature that is not shown as part of the embodiments, the feature may be added.

[0109] Figure 1 : shows a schematic representation of an imaging device with a data processing device according to an exemplary embodiment of the present invention;

[0110] Figure 2 : shows a schematic representation of an exemplary input image, an exemplary baseline image, and an exemplary output image;

[0111] Figure 3 : shows a schematic graphical representation of the calculation steps;

[0112] Figure 4 : shows a schematic representation of an imaging device with a data processing apparatus according to another exemplary embodiment;

[0113] Figure 5 : A schematic representation of a flow chart showing a computer-implemented method according to an exemplary embodiment of the present invention;

[0114] Figure 6 : shows a schematic representation of an imaging device with a data processing apparatus according to another exemplary embodiment; and

[0115] Figure 7 : Schematic representations of exemplary input images and exemplary output images are shown. DETAILED DESCRIPTION

[0116] First, refer to Figures 1 to 4 , Figure 6 and Figure 7 The structure and function of the data processing device 101 are explained. Figure 5 A computer-implemented method 501 is described.

[0117] The data processing device 101 can be part of an imaging device 102, such as a microscope 136, an endoscope (not shown), a scanner 401, a camera (not shown), a photon counter 606, or any other type of scientific or medical observation device. The following description will focus on the microscope 136, but it is also applicable to other types of imaging devices 102.

[0118] The data processing device 101 may be integrated in the microscope 136 as an embedded processor 138 or as a part of such an embedded processor 138. Furthermore, the microscope 136 may comprise an image forming section 140 adapted to capture a digital input image 108 in the form of input image data 106 with a microscope camera 142. In particular, the digital input image 108 may comprise input image data 106 at a plurality of input image positions m (e.g. pixels or voxels). The input image data 106 may, for example, be discrete intensity data and / or discrete color data. The input image data 106 is preferably represented by an n-dimensional array I(X i ), where n is an integer equal to or greater than 1.

[0119] Optionally, the microscope camera 142 may capture a plurality of digital input images 301 in the form of input image data 106 (see Figure 3 ). It should be understood that the input image data 106 may be captured by any type of microscope, in particular using a fluorescence microscope operable in a wide-field mode and / or using a confocal laser scanning microscope.

[0120] The microscope camera 142 may record the input image data 106 in monochrome. Alternatively, the microscope camera 142 may be a CCD, multispectral or hyperspectral camera that records the input image data 106 in a plurality of channels, wherein each channel preferably represents a different spectral range from infrared to ultraviolet. In the case of a CCD camera, three channels may be provided, such as an R channel, a G channel and a B channel, to represent the digital input image 108 as a visible light image of an object 144 (e.g., a sample 146). In the case of a multispectral or hyperspectral camera, a total of more than three channels may be used in at least one of the visible light range, the infrared (IR) range, the near infrared (NIR) range and the ultraviolet range.

[0121] The object 144 may include living and / or inanimate matter. The object 144 may also include one or more fluorescent materials, such as at least one fluorophore. The multispectral or hyperspectral camera may have a channel for each different fluorescence spectrum of the fluorescent materials in the object 144. For example, each fluorophore may be represented by at least one channel that matches the fluorescence spectrum triggered by the illumination 148. The microscope 136 may be adapted to excite fluorescence, such as fluorophores, within the object 144 using light having a suitable fluorescence excitation wavelength emitted by the illumination 148. Alternatively or additionally, channels may be provided for autofluorescence spectra, or for secondary fluorescence spectra triggered by fluorescence excited by the illumination 148, or for lifetime fluorescence data. Of course, the illumination 148 may also emit or only emit white light or any other light component without triggering fluorescence in the object 144.

[0122] The microscope 136 may include an optical objective 150 having at least one lens 152. Light directed through the at least one lens 152 is directed to an image sensor 402, through which input image data 106 (see Figure 4 ). The illumination 148 may include or consist of one or more flexible light guides to direct light from one or more different directions onto the object 144. A suitable blocking filter (not shown) may be arranged in the light path in front of the microscope camera 142, for example to suppress glare. For fluorescence imaging, the blocking filter preferably blocks only the illumination wavelength and allows the fluorescence of the fluorophores in the object 144 to pass to the microscope camera 142.

[0123] If recorded using a line scan camera, the input image data 106 may be one-dimensional. Alternatively, if a single channel is included in a two-dimensional image, the input image data 106 is two-dimensional. If multiple channels are included and / or if the input image data 106 represents a three-dimensional image, the dimensionality of the digital input image may be higher than two dimensions.

[0124] The three-dimensional input image data 106 can be recorded by the microscope 136, for example, using light field technology, Z-axis image stacks in a microscope, images obtained by SCAPE microscopy, and / or three-dimensional reconstruction of images obtained by SPIM microscopy. In the case of a three-dimensional image, each plane of the three-dimensional input image data 106 can be regarded as a two-dimensional digital input image 108. Similarly, each plane can include multiple channels. Each channel can be considered as a separate two-dimensional image. Alternatively, multiple channels can be interpreted together as a multi-dimensional array.

[0125] Figure 4The microscope 136 shown includes the elements / components shown therein. The microscope 136 can be considered as a device for imaging the object 144 located on the slide 404. The slide 404 can be moved relative to the microscope 136. Therefore, the object 144 can be scanned by capturing multiple digital input images 301 showing mutually adjacent areas of the object 144. The multiple digital input images 301 can be stitched together to form a tile scan 701 (see Figure 7 ).

[0126] The microscope 136, in particular the data processing device 101, may further comprise an image storage 154, which is adapted to contain the input image data 106 at least temporarily. The data processing device 101 may also be part of a computer 608, such as a general-purpose computer, in particular a workstation 610 of the microscope 136 comprising the image storage 154. The image storage 154 may comprise a volatile or non-volatile memory, such as a cache memory of a CPU of the computer 608 and / or a GPU of the computer 608. The image storage 154 may also comprise a RAM, a hard drive or an exchangeable storage system, such as a USB stick or an SD card. The image storage 154 may comprise any combination of these types of memory.

[0127] like Figure 1 , Figure 2 and Figure 7 As shown, the digital input image 108 contains image features 118 representing the actual content that the user is focusing on. Each image feature 118 extends along a specific length 120 that is greater than zero. The length 120 can be a number of length units or a number of pixels in a direction.

[0128] from Figure 1 , Figure 2 and Figure 7 148, the digital input image 108 is also subject to a so-called shading effect. In particular, the at least one lens 152 may cause lens vignetting, such that the brightness gradually decreases towards the image periphery of the digital input image 108, i.e. towards the edges 114 and corners 115 of the digital input image 108. In alternative cases, this brightness reduction 112 may also be caused or further amplified by non-uniformities in the illumination 148. In any case, the brightness reduction 112 tends to follow a specific pattern, such as having a brighter center and darker edges. In the tile scan 701, the shading effect produces a clearly perceptible checkerboard pattern (see Figure 7 ).

[0129] Thus, it may be assumed that the input image data 106 representing the digital input image 108 comprises a shading signal 110 representing a brightness reduction 112 due to shading effects and a content signal 116 representing an image feature 118 .

[0130] The data processing apparatus 101 is primarily used to generate a digital output image 126 in which shading effects are minimized or at least mitigated. To this end, the apparatus 101 is configured to access the input image data 106 and perform shading correction as described below.

[0131] The shading correction is described exemplarily for an application in which the pattern of brightness reduction 112 is distributed over a distance greater than the length 120 of the image feature 118. Under this assumption, it can be assumed that the content signal 116 exhibits high spatial frequencies, e.g., which are responsible for intensity and / or color variations occurring over short distances in the digital input image 108, while the shading signal 110 is assumed to exhibit low spatial frequencies, i.e., which results in predominantly gradual intensity and / or color variations extending over a relatively large area of ​​the digital input image 108.

[0132] Due to its low spatial frequency, the shadow signal 110 (i.e., the brightness reduction 112) can be seen as a more or less smooth baseline, which is superimposed by the content signal 116 having a high spatial frequency. Figure 2 , where the shaded signal 110 is represented by I SS (X i ), the content signal 116 is represented by I CS (X i ).

[0133] The data processing device 101 is configured to calculate a baseline image 122, which is an estimate of the baseline in the intensity data and / or color data. Therefore, the baseline image 122 also represents an estimate of the shadow signal 110. In short, the baseline image 122 is conceived as a rough "brightness map" that is used to identify which parts of the digital input image 108 are too dark due to shadow effects.

[0134] The baseline image 122 is preferably represented by discrete baseline image data at a plurality of baseline image locations (eg, pixels or voxels). b (X i ) represents the shadow signal I SS (X i ). The baseline image data may be an image data having a value similar to the input image data I(X i ) n-dimensional array F of the same dimension b (X i ).

[0135] The baseline image 122 is calculated based on the input image data 106 and a length threshold 104 which is considered to be greater than the length of the largest or average image feature 118. To this end, the data processing device 101 is configured to obtain the length threshold 104. This will be described in further detail below.

[0136] The apparatus 101 is configured to generate a digital output image 126 once the baseline image 122 is calculated by removing the shadow signal 110 from the input image data 106 and thereby effectively extracting the content signal 116. This is done by at least one of: a) subtracting the baseline image 122 from the input image data 106, and b) dividing the input image data 106 by the baseline image 122. The resulting digital output image 126 then represents an estimate of the content signal 116. The digital output image 126 may include an n-dimensional array O(x) at a plurality of output image locations (e.g., pixels or voxels). i ) represents the output image data 156. Output image data O(X i ) indicates content signal I CS (X i ) is an approximate value of .

[0137] The shading correction may also be performed on a computer 608 as a computer-implemented method 501. Thus, a computer program 601 may be provided that includes a program code 612 that, when executed by the computer 608, causes the computer 608 to perform the method 501. Thus, a computer-readable medium 602 including the program code 612 may also be provided. In this case, the data processing device 101 may be used as an image processor that is configured to read out the input image data 106 from the image storage 154 or from an external image forming unit (e.g., the image forming unit 140 of the microscope 136). To this end, the computer 608 may include an image input unit 614.

[0138] Further details of various aspects of the present invention are described below with respect to the computer-implemented method 501. It should be understood that the data processing apparatus 101 may be configured to perform the computer-implemented method 501 accordingly.

[0139] like Figure 5 As shown, the method comprises step 502: obtaining a length threshold 104. The length threshold 104 may be obtained, for example, as a predetermined threshold length λ from a direct user input 512. Alternatively, the length threshold 104 may be obtained by calculation. In particular, the length threshold 104 may depend on the size 132 of the digital input image 108. For example, the apparatus 101 may be configured to calculate the length threshold 104 based on the image size 132. In particular, the length threshold 104 may be proportional to the image size 132. Preferably, the length threshold 104 given as the predetermined threshold length λ is equal to 5%, more preferably 10%, most preferably 20% of the image size 132, wherein the image size 132 is the width of the digital input image 108 measured in pixels or length units in one of the image directions.

[0140] Before, after, or simultaneously with step 502, the input image data 106 is accessed in step 504. The accessed input image data 106 may be recorded in real time by the imaging device 102. Alternatively, the accessed input image data 106 may be retrieved, received, or read from the image storage 154.

[0141] If the input image data 106 comprises a plurality of digital input images 301, an optional step 520 may be performed in which average input image data 516 is calculated from the plurality of digital input images 301. The average input image data 516 may be image data representing a median, interpolation, arithmetic mean, geometric mean, or quadratic mean of the plurality of digital input images 301.

[0142] The baseline image 122 is then calculated in step 506 using the length threshold 104 and the input image data 106. In particular, the baseline image 122 may be calculated using a fit to the input image data 106, such as a polynomial fit or a spline fit. Preferably, the fit is calculated by an iterative minimization scheme 510.

[0143] Optionally, the iterative minimization scheme 510 may include a least squares minimization criterion 512 that minimizes the fit. The least squares minimization criterion 512 may include a penalty term and a cost function. In particular, the least squares minimization criterion 512 may be expressed as M(F b (X i )) and has the following form:

[0144] M(F b (X i ))=P(F b (X i ))+C(F b (X i ))

[0145] Where P(F b (X i )) is the penalty term, C(F b (X i )) is the cost function.

[0146] Without loss of generality, for the one-dimensional case, the penalty term P(F b (X i )) can, for example, have the following general form:

[0147]

[0148] Where r is the regularization parameter, is a discrete operator for calculating the j-th order derivative. For multidimensional digital input images, different penalty terms P(F b (Xi )).

[0149] The regularization parameter r is used as a multiplicative weighting factor in the penalty term. Therefore, the unit of the regularization parameter r can be selected so that the penalty term produces a dimensionless scalar. In addition, the regularization parameter r can be selected based on the length threshold 104. For example, the regularization parameter r can be equal to the above-mentioned threshold length λ:

[0150] r=λ

[0151] The least squares minimization criterion can also include a cost function C(F b (X i )), which represents the input image data I(X i ) and the baseline image data F b (X i ) represents the difference between the baseline images. In particular, the cost function C(F b (X i )) may be a truncated difference term obtained using a predetermined peak threshold s, which is greater than zero and defines at what peak height the difference term is truncated. The peak threshold s may be obtained, for example, from an external user input.

[0152] The truncated difference term may be symmetrical or asymmetrical. Preferably, the truncated difference term may be a term representing the input image data I(X i ) and baseline image data F b (X i ), wherein if the difference between the input image data and the baseline image data is greater than the peak threshold s, the output value of the truncated quadratic term is limited to a constant value. Otherwise, the value of the truncated quadratic term is equal to the square of the difference between the input image data and the baseline image data. The truncated quadratic term It can be in the following forms:

[0153]

[0154] Using truncated quadratic terms Cost function C(F b (X i )) can be expressed as:

[0155]

[0156] Optionally, the iterative minimization scheme 510 can be a semi-quadratic minimization scheme that includes at least a portion of the LEGEND algorithm, which is described in the following articles: "Convex Half-Quadratic Criteria and Interacting Variables for Image Restoration" by Idier J. (2001), IEEE Transactions on Image Processing, 10(7), pp. 1001-1009; and "Background Removal from Spectra by Designing and Minimizing a Non-Quadratic Cost Function" by Mazet V., Carteret C., Bire D, Idier J., and Humbert B. (2005), Chemometrics and Intelligent Laboratory Systems, vol. 76, pp. 121-133. Both articles are incorporated herein by reference in their entirety.

[0157] The semi-quadratic minimization scheme according to the LEGEND algorithm aims to minimize the least squares minimization criterion 512 and utilizes the optimal input image data I(X i ) Discrete auxiliary data D with the same dimension l (x i As a starting step in the semi-quadratic minimization scheme, an initial set of baseline image data (e.g., F b,1 (X i )=I(X i )) and an initial set of auxiliary data (e.g. D1(X i )=0).

[0158] Furthermore, the semi-quadratic minimization scheme may include a first iteration phase and a second iteration phase, the first and second iteration phases together representing one iteration step or iteration cycle. The first and second iteration phases are then preferably repeated until a convergence criterion 518 is satisfied. A suitable convergence criterion 518 may be, for example, the current baseline image data and all positions x i The sum of squares of differences between the previous baseline image data on is less than a predetermined convergence threshold c.

[0159] In the first iteration phase of each iteration cycle, the auxiliary data is updated based on the input image data and the latest baseline image data representing the baseline image. In particular, the auxiliary data is updated using a variant of the truncated quadratic term as shown below:

[0160]

[0161] Wherein l=1...L is the index of the current iteration cycle, and α is a constant value such as 0.493.

[0162] In the second iteration phase, the baseline image data may be updated according to the previously calculated updated auxiliary data, the baseline image data from the previous iteration cycle, and the penalty term using the following formula:

[0163]

[0164] Alternatively, in this second iteration stage, the discrete Green's function G(X i ) and the input image data I(X i ) and updated auxiliary data D l (X i ) is convolved with the sum of the two to calculate the baseline image data representing the baseline image. In other words, the second iteration stage of the LEGEND algorithm can be replaced by the following iteration steps, where the Green function G(X i ) Calculate the updated baseline image data F in the lth iteration cycle b,l (X i ):

[0165] F b,l (X i )=G(X i )*(I(X i )+D l (X i ))

[0166] Without loss of generality, for the one-dimensional case, the Green function G(X i ) can have the following forms:

[0167]

[0168] in, is the discrete Fourier transform, is the inverse discrete Fourier transform, r is the regularization parameter, is the function derivative.

[0169] The calculation of the baseline image itself is not limited to the above-described procedure and can be accomplished by almost any known method for estimating a baseline that correctly represents the shading signal of a given digital image.

[0170] If optional step 520 has been performed beforehand, the average input image data 516 may be used to calculate the baseline image 122 instead of the input image data 106. If the input image data 106 comprises a plurality of digital input images 301, but optional step 520 has not been performed, one baseline image 122 is calculated for each of the plurality of digital input images 301. Thereafter, in a further optional step 306, an average baseline image 302 is calculated from all baseline images 122 of the plurality of digital input images 301 (see also Figure 3 ). The average baseline image 302 may be calculated by interpolating the baseline images 122 of the plurality of digital input images 301 as an arithmetic mean, a geometric mean, or a quadratic mean as a median.

[0171] Next, in step 508, the output image data O(X) may be generated by a) subtraction or b) division as described above. i ) represents the digital output image.

[0172] Subtracting the baseline image from the input image data can be done by taking the inverse of the baseline image F b -1 (X i ) is added to the input image data I(X i ), the inverse of the baseline image is, for example, the inverse of the baseline image:

[0173] O(X i )=I(X i )+F b -1 (X i )

[0174] Alternatively, a pixel-by-pixel subtraction approach can be used:

[0175]

[0176] Optionally, the maximum intensity value F of the baseline image can be taken as b,max added to the entire digital output image to increase the overall brightness of at least one digital output image (see Figure 2 ).

[0177] Similarly, the input image data I(X i ) divided by the baseline image F b (X i ) can be implemented by performing a pixel-by-pixel division on all pixels:

[0178]

[0179] Optionally, before the pixel-by-pixel division, the baseline image Fb (X i ) is normalized to 1. That is, in the normalized baseline image F b,norm (X i ), the brightest pixel of the baseline image can be assigned a value of 1.

[0180] F b,norm (X i,max )=1

[0181] Normalized baseline image F b,norm (X i ) can be obtained by calculating the values ​​of the remaining pixels in the baseline image F b, (X i ) are obtained by linear interpolation, bilinear interpolation, nearest neighbor interpolation, Lanczos interpolation, or AI-based interpolation algorithm of the pixel values ​​in the normalized baseline image. Therefore, the dark pixels of the normalized baseline image contain values ​​less than 1, which after the pixel-by-pixel division will cause the corresponding pixels in at least one digital output image to be brighter than in at least one digital input image (see Figure 2 ).

[0182] Alternatively, the input image data I(X i ) and the inverse of the baseline image F b -1 (X i ) multiplication method:

[0183] O(X i )=I(X i )·F b -1 (X i )

[0184] Figure 4 The microscope 136 includes an integrated display 406 adapted to display at least one of the digital input image 108, the baseline image 122, and the digital output image 126. Alternatively or additionally, an external monitor 158 may be provided for these display purposes.

[0185] If the input image data 106 comprises a plurality of digital input images 301, and if the above-mentioned average baseline image 302 is calculated in optional step 306, the plurality of digital output images 304 may be generated in step 508 by at least one of: a) subtracting the average baseline image 302 from the input image data 106, and b) dividing the input image data 106 by the average baseline image 302 (see Figure 3 ).

[0186] The term "and / or" used herein includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0187] Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0188] Some embodiments involve a microscope 136 that includes a Figures 1 to 5 Alternatively, the microscope may be a system described in connection with one or more of Figures 1 to 5 One or more of the relatedly described systems may be part of or may be connected to the system. Figure 6 A schematic diagram of a system 600 configured to perform the methods described herein is shown. System 600 includes a microscope 136 and a computer system 608. Microscope 136 is configured to capture images and is connected to computer system 608. Computer system 608 is configured to perform at least a portion of the methods described herein. Computer system 608 can be configured to execute a machine learning algorithm. Computer system 608 and microscope 136 can be separate entities, but can also be integrated in a common housing. Computer system 608 can be part of a central processing system of microscope 136, and / or computer system 608 can be part of a subcomponent of microscope 136 (e.g., a sensor, actuator, camera, or lighting unit of microscope 136, etc.).

[0189] The computer system 608 may be a local computer device (e.g., a personal computer, a laptop, a tablet computer, or a mobile phone) having one or more processors and one or more storage devices, or a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in different locations, for example, distributed in a local client and / or one or more remote server farms and / or data centers). The computer system 608 may include any circuit or combination of circuits. In one embodiment, the computer system 608 may include one or more processors, which may be any type of processor. In this document, a processor may represent any type of computing circuit, such as a microscope or a microscope component (e.g., a camera), such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 608 may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits (such as communication circuits) for wireless devices (such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems). The computer system X20 may include one or more storage devices, which may include one or more storage elements suitable for a specific application, such as a main memory in the form of a random access memory (RAM), one or more hard disk drives, and / or one or more drives for manipulating removable media (such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc.). The computer system 608 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 608.

[0190] Some or all of the method steps may be performed by (or using) a hardware device, such as a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0191] According to certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transient storage medium (e.g., a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) in which electronically readable control signals are stored that cooperate with (or can cooperate with) a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0192] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0193] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may be stored, for example, on a machine readable carrier.

[0194] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0195] In other words, therefore, one embodiment of the present invention is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0196] Therefore, another embodiment of the present invention is a storage medium (or a data carrier or a computer-readable medium) comprising a computer program stored thereon for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, comprising a processor and a storage medium.

[0197] Therefore, another embodiment of the present invention is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection (eg via the Internet).

[0198] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0199] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0200] Another embodiment of the invention comprises an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, or a memory device. The apparatus or system may comprise, for example, a file server for transmitting the computer program to the receiver.

[0201] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0202] Embodiments may be based on the use of machine learning models or machine learning algorithms. Machine learning may refer to algorithms and statistical models that a computer system can use to perform specific tasks without the use of explicit instructions, but rather relying on models and inferences. For example, in machine learning, data transformations inferred from analysis of historical and / or training data may be used instead of rule-based data transformations. For example, a machine learning model may be used or a machine learning algorithm may be used to analyze the content of an image. In order for a machine learning model to analyze the content of an image, a machine learning model may be trained using training images as input and training content information as output. By training a machine learning model using a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the image, so the machine learning model may be used to recognize the content of the image that is not included in the training data. The same principle applies to other types of sensor data: by training a machine learning model using training sensor data and desired outputs, the machine learning model "learns" the transformation between sensor data and output, which can be used to provide outputs based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain a feature vector, which is used as an input to the machine learning model.

[0203] The machine learning model can be trained using training input data. The above example uses a training method called "supervised learning". In supervised learning, the machine learning model is trained using multiple training samples, each of which may include multiple input data values ​​and multiple expected output values, i.e., each training sample is associated with an expected output value. By specifying the training samples and the expected output values, the machine learning model "learns" which output value to provide based on input samples similar to the samples provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms (e.g., classification algorithms, regression algorithms, or similarity learning algorithms). When the output is limited to a limited set of values ​​(classification variables), a classification algorithm can be used, i.e., the input is classified into one of the limited set of values. When the output can have any numerical value (within a certain range), a regression algorithm can be used. Similarity learning algorithms can be similar to classification and regression algorithms, but are based on learning from examples using a similarity function that measures the degree of similarity or correlation between two objects. In addition to supervised or semi-supervised learning, unsupervised learning can also be used to train machine learning models. In unsupervised learning, input data may be (only) provided, and an unsupervised learning algorithm may be used to find structure in the input data (e.g., find commonalities in the data by grouping or clustering the input data). Clustering is the assignment of input data comprising multiple input values ​​into subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but are dissimilar to input values ​​included in other clusters.

[0204] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Rewards are calculated based on the actions taken. Reinforcement learning is based on training one or more software agents to select actions that increase the cumulative reward so that the software agent performs better at a given task (evidence is the increase in reward).

[0205] In addition, some techniques can be applied to some machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained at least in part using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms (which can be called representation learning algorithms) can retain information in their inputs, but can also transform that information to make it useful, typically as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0206] In some examples, anomaly detection (i.e., outlier detection) can be used, the goal of which is to identify input values ​​that are sufficiently different from the majority of inputs or training data to raise suspicion. In other words, a machine learning model can be trained at least in part using anomaly detection, and / or a machine learning algorithm can include an anomaly detection component.

[0207] In some examples, a machine learning algorithm may use a decision tree as a prediction model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to the item may be represented by a leaf of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree; if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0208] Association rules are another technique that can be used with machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. Machine learning algorithms can identify and / or utilize one or more relationship rules that represent knowledge derived from the data. For example, rules can be used to store, manipulate, or apply knowledge.

[0209] Machine learning algorithms are often based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a data structure and / or rule set that represents the learned knowledge (e.g., based on the training performed by the machine learning algorithm). In an embodiment, the use of a machine learning algorithm may mean the use of an underlying machine learning model (or multiple underlying machine learning models). The use of a machine learning model may mean that the machine learning model and / or the data structure / rule set that is the machine learning model is trained by a machine learning algorithm.

[0210] For example, a machine learning model can be an artificial neural network (ANN). ANN is a system inspired by biological neural networks (e.g. neural networks in the retina or brain). ANN includes multiple interconnected nodes and multiple connections (so-called edges) between nodes. There are usually three types of nodes: input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node can represent an artificial neuron. Each edge can transfer information from one node to another. The output of a node can be defined as a (non-linear) function of its input (e.g., the sum of its inputs). The input of a node can be used in this function based on the "weight" of the edge or node that provides the input. The weights of nodes and / or edges can be adjusted during the learning process. In other words, the training of an artificial neural network can include adjusting the weights of the nodes and / or edges of the artificial neural network, i.e., achieving the desired output for a given input.

[0211] Alternatively, the machine learning model can be a support vector machine, a random forest model or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm, which can be used to analyze data (e.g., in classification or regression analysis). A support vector machine can be trained by providing an input with multiple training input values ​​belonging to one of two categories. A support vector machine can be trained to assign new input values ​​to one of the two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graph model. A Bayesian network can use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics a natural selection process.

[0212] Reference numerals

[0213] 101 Data processing device

[0214] 102 Imaging equipment

[0215] 104 Length Threshold

[0216] 106 Input image data

[0217] 108 digital input image

[0218] 110 Shadow Signal

[0219] 112 Brightness reduction

[0220] 114 Edge

[0221] 115 corner

[0222] 116 Content Signals

[0223] 118 Image Features

[0224] 120 Length

[0225] 122 Baseline images

[0226] 124 Estimated

[0227] 126 Digital output images

[0228] 128 Estimated

[0229] 130 photon counting images

[0230] 132 Size

[0231] 134 Fitting

[0232] 136 Microscope

[0233] 138 Embedded Processors

[0234] 140 Image forming unit

[0235] 142 Microscope Camera

[0236] 144 objects

[0237] 146 samples

[0238] 148 Lighting

[0239] 150 Optical lens

[0240] 152 Lens

[0241] 154 Image storage unit

[0242] 156 Output image data

[0243] 158 External Monitor

[0244] 301 Multiple digital input images

[0245] 302 Average baseline images

[0246] 304 Multiple digital output images

[0247] 306 Optional Steps

[0248] 401 Scanner

[0249] 402 Image Sensor

[0250] 404 Slide

[0251] 406 Integrated Display

[0252] 501 Method

[0253] 502 Method Steps (Get)

[0254] 504 Method Steps (Access)

[0255] 506 Methods and Steps (Calculation)

[0256] 508 Method Steps (Generate)

[0257] 510 Iterative Minimization Scheme

[0258] 512 Least Squares Minimization Criterion

[0259] 514 User Input

[0260] 516 Average input image data

[0261] 518 Convergence Criteria

[0262] 520 Optional Steps

[0263] 600 System

[0264] 601 Computer Programs

[0265] 602 Computer readable medium

[0266] 604 processor

[0267] 606 Photon Counter

[0268] 608 Computer

[0269] 610 Workstation

[0270] 612 Program Code

[0271] 614 Image input unit

[0272] 701 Tile Scan

[0273] m is the number of input image locations

[0274] M Least squares minimization criterion

[0275] P penalty term

[0276] C Cost Function

[0277] r Regularization parameter

[0278] Λ Threshold length

[0279] s Peak Threshold

[0280] Truncated quadratic term

[0281] X Image Position

[0282] O Output image data

[0283] I Input image data

[0284] c Convergence threshold

[0285] D l Auxiliary data

[0286] G Green's function

[0287] F b Baseline image

Claims

1. A data processing device (101) for an imaging device (102), the device (101) being configured to: - Get length threshold (104), - accessing input image data (106) representing at least one digital input image (108), wherein The input image data (106) includes o a shadow signal (110) representing a decrease in brightness (112) towards an edge (114) of the at least one digital input image (108), and o a content signal (116) representing an image feature (118) of the at least one digital input image (108), the length (120) of the image feature (118) being less than the length threshold (104), - calculating a baseline image (122) based on the input image data (106) and the length threshold (104), wherein the baseline image (122) represents an estimate (124) of the shadow signal (110), - generating at least one digital output image (126) representing an estimate (128) of the content signal (116) by at least one of: a) subtracting the baseline image (122) from the input image data (106), and b) Dividing the input image data (106) by the baseline image (122).

2. The data processing device (101) according to claim 1, wherein: The at least one digital input image (108) is a photon counting image (130), and the at least one digital output image (126) is generated by dividing the input image data (106) by the baseline image (122).

3. The data processing device (101) according to claim 1 or 2, wherein: The length threshold (104) depends on a size (132) of the at least one digital input image (108).

4. The data processing device (101) according to any one of claims 1 to 3, wherein: The apparatus (101) is configured to compute the baseline image (122) using a fit (134) to the input image data (106), and wherein the fit (134) is computed by an iterative minimization scheme (510).

5. The data processing device (101) according to claim 4, wherein: The iterative minimization scheme (510) includes a least squares minimization criterion (512) that minimizes the fit (134), wherein the least squares minimization criterion (512) includes a penalty term and a cost function.

6. The data processing device (101) according to claim 5, wherein: The penalty term includes a derivative of the baseline image (122).

7. The data processing device (101) according to claim 5 or 6, wherein: The cost function represents the difference between the input image data (106) and the baseline image (122).

8. The data processing device (101) according to any one of claims 5 to 7, wherein: The iterative minimization scheme (510) includes a first iterative phase and a second iterative phase, wherein: - in said first iteration phase, updating auxiliary data based on said baseline image (122) of a previous iteration, truncated quadratic terms and said input image data (106), and, - In the second iteration phase, the baseline image (122) is calculated using a convolution of a discrete Green's function with the sum of the input image data (106) and the updated auxiliary data.

9. The data processing device (101) according to any one of claims 1 to 8, wherein: The input image data (106) represents a plurality of digital input images (301).

10. The data processing device (101) according to claim 9, wherein: The device (101) is configured as follows: - calculating said baseline image (122) for each of said plurality of digital input images (301), and - calculating an average baseline image (302) based on the baseline images (122) of the plurality of digital input images (301).

11. The data processing device (101) according to claim 10, wherein: The apparatus (101) is configured to generate a plurality of digital output images (304) by at least one of the following methods: a) subtracting the average baseline image (302) from the input image data (106), and b) Dividing the input image data (106) by the average baseline image (302).

12. The data processing device (101) according to claim 9, wherein: The device (101) is configured as follows: calculating average input image data from the plurality of digital input images (301), calculating an average baseline image (302) based on the average input image data and the length threshold (104), and Generating a plurality of digital output images (304) by at least one of the following: a) subtracting the average baseline image (302) from the input image data (106), and b) Dividing the input image data (106) by the average baseline image (302).

13. An imaging device (102), comprising a data processing apparatus (101) according to any one of claims 1 to 12, wherein: The device (102) is configured to record input image data (106) representing the at least one digital input image (108).

14. A computer-implemented method (501) for processing input image data (106) of an imaging device (102), the method comprising the steps of: Get (502) the length threshold (104), accessing (504) input image data (106) representing at least one digital input image (108), wherein the input image data (106) comprises o a shadow signal (110) representing a decrease in brightness (112) towards an edge (114) of the at least one digital input image (108), and o a content signal (116) representing an image feature (118) of the at least one digital input image (108), the length (120) of the image feature (118) being less than the length threshold (104), calculating (506) a baseline image (122) based on the input image data (106) and the length threshold (104), wherein the baseline image (122) represents an estimate (124) of the shadow signal (110), generating (508) at least one digital output image (126) representing an estimate (128) of the content signal (116) by at least one of: a) subtracting the baseline image (122) from the input image data (106), and b) Dividing the input image data (106) by the baseline image (122).

15. A method for operating an imaging device (102), the method comprising the steps of: recording input image data (106) representing at least one digital input image (108), and A computer-implemented method (501) according to claim 14 is performed.

16. A computer program (601) or a computer readable medium (602) comprising program code for performing the method (501) according to claim 14 when the program code is executed on a processor (604).

17. A neural network device, which is trained by a plurality of digital input images (301) and a plurality of digital output images (304), wherein the digital output images (304) are calculated from the digital input images (301) using a data processing device (101) according to any one of claims 1 to 11 or by a method (501) according to claim 14.

Images