Fourier lamination imaging generation image

Abstract

Fourier lamination imaging is required to be simplified to generate an image. To this end, the object is illuminated using a plurality of illumination elements arranged in a distributed manner in a respective plurality of positions in the space. A plurality of spatial frequency patterns (16) resulting from illumination of the object with a single illumination element or a plurality of illumination elements, respectively, of the plurality of illumination elements (4) are detected. Each spatial frequency pattern center is centered in Fourier space at a position corresponding to a nominal spatial frequency (8) of one or more respective lighting elements. The image is reconstructed by using an entirety of all respective centered spatial frequency patterns.

Description

Technical Field

[0001] The invention relates to a method for generating an image of an object by Fourier phantom imaging by means of a color-corrected optical unit by illuminating the object with a plurality of illumination elements, wherein the illumination elements are arranged in a distributed manner at a corresponding plurality of locations in space. The invention also relates to a corresponding device for generating an image of an object by Fourier phantom imaging, having a color-corrected optical unit and an illumination device. Background Art

[0002] Stack imaging is a microscopic method. The sample to be examined is sampled by an electromagnetic or particle beam, which scatters from the material and forms a diffraction and / or interference pattern. The beam changes its position slightly each time (e.g. different positions of the light source), so that it hits the sample at different angles and accordingly also produces different diffraction and / or interference images. From the diffraction and / or interference images from multiple positions of the sample, an image of the entire sample can be calculated by an algorithm (e.g. inverse Fourier transform). The advantage of this technique is that the diffraction and / or interference pattern also contains information about the phase of the wave and can therefore also image transparent structures, for example.

[0003] In Fourier stack imaging (FP), a wavelength parameter is required for image reconstruction in order to perform a correct reconstruction. Technically, the spatial spectrum of the diffraction pattern must be associated with the lateral spatial frequency pattern of the image, e.g. detected by a detector and generated by a subsequent Fourier transformation of the image. During imaging, a narrower or broader emission spectrum of the light source is usually used instead of a single wavelength. In current implementations of Fourier stack imaging, the spectral distribution is not taken into account, but instead the wavelength parameter is determined as a (effective or average) value in the emission spectrum of the light source, in such a way that the reconstructed image is optimized, e.g. by varying the value of the wavelength parameter.

[0004] For example, in the evaluation codes available in software packages for Fourier ptychographic imaging or in the literature (e.g. G. Zheng: "Fourierptychographic imaging", 2016, published by Morgan & Claypool Publishers, ISBN 978-1-64327-860-5), currently only one wavelength parameter can be optimized in each reconstruction in order to optimize the image quality. However, none of the schemes take into account the spectral distribution of the light source. Everything is indirectly covered by one of the wavelength parameters, where this wavelength parameter roughly corresponds to the weighted average of the effective wavelengths transmitted through the entire optical system. Summary of the invention

[0005] It is therefore an object of the present invention to simplify the Fourier stack imaging reconstruction of images of an object.

[0006] According to the invention, this object is achieved by a method and a device according to the independent patent claims. In addition, a corresponding computer program or a computer-readable storage medium is provided. Advantageous developments of the invention are derived from the dependent claims.

[0007] According to the invention, a method is provided for generating an image of an object by Fourier stack imaging with the aid of color-corrected optical units. The invention makes use of the fact that the images of an imaging system are usually color-corrected to a very high degree. This is a prerequisite for all imaging systems, from microscopes to macroscopic imaging systems, cameras, etc. Even in critical applications such as lithography, the individual optical units are color-corrected for the spectrum used. Even in color confocal imaging systems (e.g. confocal white light sensors or color scanning microscopes, such as the Zeiss CSM with Nipkow disk), lateral chromatic aberrations are compensated and only axial chromatic aberrations with specific properties are incorporated into the system.

[0008] In the method according to the invention, a plurality of lighting elements arranged in a distributed manner at a corresponding plurality of positions in space are used to illuminate an object. The illumination source for the object is a lighting device which comprises a number of individual lighting elements which can be located at a plurality of predetermined positions in space. For example, it involves a matrix emitter. Each individual lighting element is preferably regarded as point-shaped and can therefore be located at a corresponding point accordingly.

[0009] In a further step of the method according to the invention, a plurality of spatial frequency patterns generated by illuminating the object with a single lighting element or a plurality of lighting elements, respectively, of the plurality of lighting elements, are detected. Thus, for example, the object is illuminated separately with a single lighting element of the lighting device. However, it is also possible to illuminate with a plurality of groups of lighting elements of the plurality of lighting elements. Thus, the object is illuminated successively by a single lighting element or a plurality of lighting elements of the plurality of lighting elements. At the same time, in this case, either a single lighting element or a group of lighting elements is switched on for illumination.

[0010] Next, each spatial frequency pattern is centered at a (corresponding) position in Fourier space corresponding to the nominal spatial frequency of a corresponding illumination element or elements. The nominal spatial frequency is the point in Fourier space that represents the center number of the spatial frequency pattern of the illumination element relative to its position. Thus, each spatial frequency pattern or interference pattern is centered at a specific position in Fourier space corresponding to its nominal spatial frequency.

[0011] Finally, the image is reconstructed by using the ensemble of all the individually centered spatial frequency patterns. For example, all spatial frequency patterns are "summed" (i.e. spatially combined) and an image is obtained therefrom by inverse Fourier transformation. Since the color-corrected optical unit has no dispersion, the reconstruction can be performed independently of wavelength.

[0012] In an advantageous embodiment of the method according to the invention, it is provided that the plurality of positions at which the lighting elements are arranged in a distributed manner in space are located on an area designed in the shape of a plane, an ellipse or a spherical shell portion. Thus, the lighting elements can be evenly distributed on a rectangular area, for example. This results in, for example, a matrix emitter. Alternatively, the lighting elements can also be arranged in a distributed manner on the surface of a sphere (or other body). In particular, they can be evenly distributed on a spherical shell portion. In particular, such a distribution can also be provided on a hemisphere. Not only the distribution on a plane area, for example a rectangle, but also the distribution on a spherical portion also enables a simple reconstruction algorithm for reconstructing the image.

[0013] In one embodiment of the method according to the invention, the respective nominal spatial frequency is determined (directly) from the angle of incidence of the respective lighting element on the object or from the position of the respective lighting element. For example, in the case of a rectangular arrangement of the lighting elements, it is advantageous to determine the nominal spatial frequency of the respective lighting element directly from its position, for example the x and y coordinates. In contrast, in the case of a spherical arrangement of the lighting elements, i.e. in the shape of a (partial) spherical shell, it is advantageous to determine the nominal spatial frequency of the respective lighting element directly from its angle of incidence on the object. Thus, the coordinates in K-space (Fourier space) can be determined directly from the polar angle and the azimuth angle of each lighting element.

[0014] As already pointed out above, if the two orthogonal spatial directions x and y have a uniform spacing P from each other x and P y If multiple lighting elements are arranged in a rectangular shape, the nominal spatial frequency can be calculated using the following formula: K x,i = atan(i*P x / D) and K y,j =atan(j*P y / D), where D corresponds to the distance between the object plane where the object is located and the illumination plane where the plurality of illumination elements are arranged. Alternatively, if the plurality of illumination elements are arranged in a spherical shape, wherein the azimuth angle of the illumination elements arranged in the ring i is and the polar angle relative to the optical axis of the optical unit is θ i , then the nominal spatial frequency can be calculated using the following formula:

[0015] and in is the uniform azimuthal spacing of the lighting elements on the corresponding ring i.

[0016] Thus, in the case of a sphere or ball, the nominal spatial frequency can be calculated from spherical coordinates describing the position of the light source, while in the case of a planar array of illumination elements, the nominal spatial frequency can be calculated from the Cartesian coordinates of the corresponding light source position. In the case where the sphere or array is also "centered" on the optical axis of the imaging system and the position of the light source is known for aligning the illumination system, a simple equation results. If there is no centering, then an "offset" must be taken into account to achieve centering in the calculation.

[0017] As described above, reconstructing the image may comprise inverse Fourier transforming the population of all the individually centered spatial frequency patterns. This means that all spatial frequency patterns obtained by individual lighting elements or groups of lighting elements are taken into account in the inverse transform. Thus, the inverse transform may be implemented by a simple algorithm.

[0018] In a particularly preferred embodiment, it is provided that the spatial frequency domain or Fourier space for reconstructing the image is limited based on the maximum spatial frequency defined by the numerical aperture of the color correction optical unit. The spatial frequency of the resolution limit thus results directly from the numerical aperture. However, there is a wavelength dependency in this case, as follows: the numerical aperture is divided by the wavelength to calculate the spatial frequency.

[0019] Furthermore, each spatial frequency pattern can be limited by a gradient or apodization filter configured according to the optical and / or wavelength-dependent transmission characteristics of the optical system (e.g. based on a modulation transfer function). By limiting the spatial frequency spectrum of each individual image, the computational effort in the reconstruction can be reduced.

[0020] In a further advantageous embodiment of the method according to the invention, it is provided that each spatial frequency pattern is corrected by means of an inverse modulation transfer function of the color-corrected optical unit. This correction has the effect that the influence of the color-corrected optical unit on the image content is reduced and preferably completely compensated. In this way, a spatial frequency pattern which is virtually undistorted can be determined.

[0021] In one embodiment, it can be provided that during reconstruction, each individual image obtained using a single illumination element or a plurality of illumination elements from the plurality of illumination elements is freed from the modulation transfer function of the color correction optical unit by deconvolution, each individual image corrected in this way is subsequently transformed into a corresponding corrected spatial frequency pattern, and only these corrected spatial frequency patterns are combined. In this case, the individual images are thus corrected in the spatial domain before the transformation into the spatial frequency domain.

[0022] Alternatively, during reconstruction, each individual image obtained using a single illumination element or a plurality of illumination elements from a plurality of illumination elements may be transformed into a corresponding spatial frequency pattern by Fourier transformation, whereupon each spatial frequency pattern is then freed from the influence of the respective transfer function of the optical unit and / or illumination element on the Fourier components of the respective spatial frequency pattern, and only the spatial frequency patterns corrected in this way are combined. Thus, here, the correction takes place in the spatial frequency domain instead of in the spatial domain. This may also bring advantages in terms of computational speed.

[0023] In another advantageous embodiment of the method according to the invention, it is provided that during image reconstruction all spatial frequency patterns are combined by means of an iterative optimization algorithm. In this way it can be achieved that the spatial frequency patterns are combined in such a way that a high-resolution image can ultimately be obtained.

[0024] According to the present invention, the above object is also achieved by a device for generating an image of an object by Fourier stack imaging, the device comprising:

[0025] – a colour correction optical unit; and

[0026] - a lighting device for illuminating the object using a plurality of lighting elements, wherein the lighting elements are arranged in a distributed manner at a corresponding plurality of locations in space, the device also having

[0027] - detection means for detecting a plurality of spatial frequency patterns, wherein the plurality of spatial frequency patterns are generated by illuminating the object with a single lighting element or a plurality of lighting elements, respectively, from a plurality of lighting elements, and

[0028] - Computational means for centering each spatial frequency pattern at a position in Fourier space corresponding to a nominal spatial frequency of a respective illumination element or elements and for reconstructing the image by using the ensemble of all the respective centered spatial frequency patterns. The computational means may be integrated into the optical sensor or form a separate unit.

[0029] The advantages and developments described above in conjunction with the method according to the invention also apply in the same sense to the device according to the invention. Therefore, the described method features in the case of the device should be regarded as functional features of the corresponding device.

[0030] According to the present invention, a computer program is also provided, which includes instructions, and during the execution of the program by the above-mentioned device, the instructions cause the device to perform the method described above. In addition, a computer-readable storage medium can also be provided, which includes instructions, and during the execution by the above-mentioned device, the instructions cause the device to perform the method described. The storage medium can be, for example, at least partially designed as a non-volatile data memory (e.g., flash memory and / or SSD-solid state drive) and / or at least partially designed as a volatile data memory (e.g., RAM-random access memory). In addition, the storage medium can also be implemented in the data memory of the processor circuit. However, the storage medium can also be operated, for example, as a so-called application store server in the Internet. A processor circuit with at least one microprocessor can be provided by a computer or a computer network. Instructions can be provided as binary code or assembler and / or source code of a programming language (e.g., C).

[0031] For application situations or application scenarios that may occur in the method and are not explicitly described here, provision may be made for outputting error messages and / or requests for inputting user feedback and / or for setting standard settings and / or predetermined initial states according to the method.

[0032] The invention also includes combinations of features of the described embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The following describes an embodiment of the present invention.

[0034] Figure 1 The construction of the FP microscope is shown;

[0035] Figure 2 An enlarged view of the illumination device of the FP microscope is shown;

[0036] Figure 3 A magnified view of the object plane of the FP microscope is shown;

[0037] Figure 4 A schematic diagram of a rectangular lighting arrangement is shown;

[0038] Figure 5 A schematic plan view of a spherical lighting arrangement is shown;

[0039] Figure 6 Shows Figure 4 The Fourier space of the arrangement in ;

[0040] Figure 7 Shows Figure 5 The Fourier space of the arrangement in ;

[0041] Figure 8 The spectrum of the lighting element is shown;

[0042] Fig. 9 The resolution limit in Fourier space is shown;

[0043] Fig.10 shows the amplitude distribution in Fourier space;

[0044] Fig.11 A schematic diagram of the reconstruction process is shown;

[0045] Fig.12 The transfer function of the optical system is shown;

[0046] Fig.13 The normalized transfer function is shown;

[0047] Fig.14 shows the spectral distribution when the optical system is considered;

[0048] Fig.15 The modulation transfer function of a real optical system is shown;

[0049] Fig.16 shows a flowchart of the reconstruction algorithm;

[0050] Fig.17 A flow chart showing an alternative reconstruction algorithm;

[0051] Fig.18 shows the Fourier space of a rectangular illumination arrangement with the K-space of the image sensor, and

[0052] Fig.19 The Fourier space of a spherical illumination arrangement with the K-space of the image sensor is shown. DETAILED DESCRIPTION

[0053] The embodiments described in more detail below constitute preferred embodiments of the invention.

[0054] Figure 1 The conventional design of a Fourier ptychography microscope 1 is shown. It has an object plane in which an object 2 to be imaged is arranged. The object 2 is illuminated from below by means of an illumination device 3 .

[0055] Figure 2 An enlarged section of a lighting device 3 is shown. The lighting device 3 has a plurality of regularly arranged lighting elements 4. In the present example, the lighting elements are arranged in a matrix in orthogonal directions x and y. Each lighting element can be implemented, for example, as an LED, which emits a lighting beam 5 toward the object 2. Figure 2, arrows 6 indicate the direction in which the individual lighting elements 4 can be switched on and off again in succession. Thus, for example, the switching-on sequence can be: first the lighting elements 4 in the first row are switched on successively, then those in the second row, and so on. In principle, any sequence can be chosen. Furthermore, it is also possible for a plurality of lighting elements 4 in a group to be switched on and off again together.

[0056] The illumination beam of each illumination element 4 is diffracted by the object 2 and collected by the optical unit 6. On the other side of the optical unit 6, a Fourier space 7 for imaging the object 2 is generated.

[0057] The Fourier space 7 is Figure 3 It is shown in an enlarged plan view. It has coordinates K x and K y For each individual lighting element 4, a nominal spatial frequency 8 is generated in the Fourier space 7. This nominal spatial frequency 8 represents, for example, the center point of the circle shown and is similar to Figure 2 Move according to arrow 9 in the figure.

[0058] The Fourier space 7 is captured or sampled by the optical sensor 10 of the FP microscope 1 .

[0059] With this FP microscope (here a transmitted light microscope) it is possible to capture image sequences of an object in brightfield and / or darkfield mode using different collimated or very low-aperture spatial illuminations. Instead of a planar arrangement of the illumination element 4, a (hemispherical) arrangement can also be used. It can be assumed that the microscope is optically corrected in such a way that images of the sample are captured in good quality without noticeable distortions or chromatic aberrations. Optionally, the illumination wavelength can be selected by the LED used or in combination with a bandpass filter with a half-width at half maximum of, for example, 10, 50 or 80 nm in front of a black and white camera. The tube lens of the FP microscope may lead to imaging aberrations. These effects may occur in particular in the paraxial region of the object space, for example <+ / - 500 μm relative to the optical axis.

[0060] Reference now Figures 4 to 7 Rectangular and spherical arrangements of lighting elements in the spatial domain and their impact in the spatial frequency domain are explained in more detail.

[0061] Figure 4 A rectangular arrangement of lighting elements 4 is shown. Here, the lighting device has 81 lighting elements 4 (eg LEDs). They have a spacing P in the horizontal direction x. x , with a spacing P in the vertical direction y y Each lighting element has corresponding coordinates (i, j). The position of each lighting element 4 in the lighting angle coordinate system can be specified as:

[0062] Kx,i = atan(i*P x / D) and

[0063] K y,j =atan(j*P y / D),

[0064] wherein i is, for example, an integer in the range from −m to m, and j is an integer in the range from −n to n. D corresponds to the distance between the object plane of the object 2 and the illumination plane of the illumination device 3 . Figure 6 Shows Figure 4 The corresponding illumination coordinate system in Fourier space for a rectangular illumination arrangement in . Each point represents a nominal spatial frequency 8 for this rectangular arrangement. Therefore, the sampling is not equidistant and the symmetry axis may show aliasing effects. The spot size of the illumination element is determined by the numerical aperture of the illumination of each LED from the point of view of the object. In the case of a planar arrangement, the spot size decreases with increasing distance from the optical axis.

[0065] Figure 5 In FIG. 1 , a spherical arrangement of lighting elements 4 is shown in plan view. The lighting elements or LEDs are at a constant radial distance R from the intersection of the optical axis (+) and the object plane. The rings have a fixed angular increment Δθ. The LEDs in these rings have a constant angular spacing within the respective ring. For each ring the individual azimuth displacement Used to avoid crosstalk effects.

[0066] Figure 5 The arrangement in FIG. 4 has, by way of example only, 76 lighting elements 4 in four rings around the center. represents the azimuth angle in ring i. i Corresponds to the polar angle relative to the optical axis.

[0067] The center position of the lighting element in the lighting angle coordinate system can be expressed as follows:

[0068] and

[0069]

[0070] or explicitly

[0071] and

[0072]

[0073] Where i is 0 to N R -1, and j is 1 to N depending on the number of rings. R1 or 1 to N R2 or 1 to N R Here, NR is the maximum number of lighting elements on the corresponding ring. Since the spherical structure has a constant angular increment, the angular coordinate spacing between the individual lighting elements is uniform. Figure 7 Shown with Figure 5 The illumination coordinate system in Fourier space corresponds to the spherical arrangement of the illumination elements 4 in the spherical arrangement. Here, equidistant sampling can be performed in the radial and circumferential directions. Therefore there is no axis of symmetry.

[0074] The combination of the illumination coordinate system and the sampling coordinate system of an optical sensor in Fourier space is explained below. The illumination coordinate system extends over a larger range in the Fourier space coordinate system than the image coordinate system (also called the sampling coordinate system). In order to combine the two coordinate systems, their measurements or physical units must match. The measurement in the illumination coordinate system is the unitless numerical aperture, i.e. the sine of the angle. The measurement in the sampling coordinate system is, for example, 1 / mm or 1 / μm, i.e. the inverse length.

[0075] The link between these two coordinate systems is the wavelength λ used, since the diffraction formula uses sinα n =n*λ / Λ and the structure width Λ, diffraction angle α n and the corresponding wavelength λ are coupled to each other, where n indicates the diffraction order (usually n=1).

[0076] This implies some general aspects. On the one hand, for a given structure width Λ, the diffraction pattern is highly dispersive (sin α ~ λ). On the other hand, the wavelength and bandwidth of the illumination are of great significance for the reconstruction in the case of standard methods. In particular, the effective wavelength plays an important role in the transformation of the illumination coordinate space into the sampled Fourier coordinate space. Regarding the bandwidth, the dispersion is very important in two aspects. The higher the numerical aperture of the optical unit, the greater the influence may be. This is only not a problem if the optical unit is well corrected. Regarding diffraction, the bandwidth of the illumination light is directly related to the "ambiguity" in the diffraction angle in the aperture plane. The effective wavelength is obtained as the convolution of the LED emission plus the filter and sensor sensitivity and the modulation transfer function in the case of the corresponding wavelength.

[0077] This has the following consequences and advantages for current reconstruction algorithms:

[0078] – For a given wavelength / effective wavelength and a given numerical aperture, if the illumination system is calibrated at an angle relative to the sensor coordinate axes, then there exists a deterministic connection transformation for the illumination and sampling coordinate systems.

[0079] – If high numerical aperture and high-resolution, high-detail images are desired, then the bandwidth of the imaging system only needs to be controlled or limited to avoid image degradation due to chromatic aberration during imaging. As long as the camera optics and image quality are high, wavelength bandwidth is not an issue.

[0080] – Different imaging wavelengths can be used to generate Fourier patterns even when the sampling overlap at each wavelength is small.

[0081] Compared to conventional reconstruction algorithms, the new approach according to the invention is based on the observation that the image captured by the optical unit and therefore by the image sensor in a plane conjugate to the object plane has a high quality. Therefore, in a fully color-corrected optical unit there is no dispersion in the image and therefore no wavelength effects. Therefore, a dispersion-free (wavelength-independent) stylographic image reconstruction method is provided according to the invention. For this purpose, the numerical aperture of the illumination has to be related to the image frequency in space without introducing the concept of diffraction angle.

[0082] To clarify this difference, first refer to Fig.11 Let us briefly explain how the FP reconstruction process according to the prior art is carried out by means of a wavelength-dependent angular transformation. The starting point is, for example, an object pattern 11 in the spatial domain. After a Fourier transformation (FFT), a corresponding harmonic spectrum 11' with line spacing and complex conjugate components is generated in the spatial frequency domain or Fourier space. For the reconstruction, it should be taken into account that the blue wavelength (λ1) is shorter than the green wavelength (λ2), which in turn is shorter than the red wavelength (λ3). The angle of incidence set for each illumination is independent of the wavelength and is α i and β j , where α i is the polar angle, β j is the azimuth angle. Therefore, each light source or each lighting element can be assigned an angular coordinate (α i , β j ). With the wavelength as parameter, the corresponding angle can be converted to the grating constant, e.g. for light incident perpendicular to the grating plane simplified:

[0083] G1=λ1 / sin(α1), G2=λ2 / sin(α2) and G3=λ3 / sin(α3).

[0084] Each Fourier transformed pattern is then located in Fourier space with its center point at position (g i ,β j ) where g i =1 / G i is the corresponding grating frequency in a common coordinate system for the Fourier transform and the grating spatial frequency. g i The maximum filter that can be implemented is gi,max =2π*NA / λ i .

[0085] Therefore, filtering the Fourier space with a maximum diffraction angle defined by the numerical aperture of the optical unit and the wavelength yields: the intensity in Fourier space is the polar angle α i , azimuth angle β j , wavelength λ i and the function of numerical aperture, namely:

[0086] I1(g i ,β j )=f(α i ,β j ,λ1,NA),

[0087] I2(g i ,β j )=f(α i ,β j ,λ2,NA) and

[0088] I3(g i ,β j )=f(α i ,β j ,λ3,NA).

[0089] A high-resolution image with the combined amplitude and phase of the partial images (self-consistency in the Fourier space overlapping regions) is generated from a set of images with boundary conditions as input by an iterative algorithm 12. This generates corresponding monochrome output images 14 of the corresponding colors and one phase image 15 for each color.

[0090] Another possibility to couple the illumination opening angle and the spatial frequency of the image plane is to use a minimum structure d min The concept of resolution limit:

[0091] d min =k1*λ / NA

[0092] Here, k1 is a parameter that depends on the resolution standard used. For example, using the Rayleigh standard k1=0.61 or the Sparrow standard k1=0.5=k 1,min For modern camera sensors with linear intensity scaling: k1 = 0.5 applies.

[0093] The resolution limit d for a given imaging configuration with fixed k1 and NA is min (λ,k1,NA) can be directly converted to spatial frequency like camera pixels:

[0094] k(λ)=1 / (2*d min (λ) = NA / (2*k1*λ)

[0095] Alternatively, for a given camera sensor and k1, there is a limiting wavelength λ that can transmit the full information content of the image:

[0096] d min =2*d pixel =k1*λ / NA

[0097] This results in:

[0098] λ min =2*NA*d pixel / k1

[0099] All parameters on the right side of the last equation are fixed and determined by the parameters of the imaging configuration. min is the shortest wavelength that can be imaged using the full numerical aperture NA of the objective lens, so that a specific pixel pitch d pixel The detector images the structure, which can limit the resolution.

[0100] The corresponding spatial frequency is calculated as follows:

[0101] k NA =2*π / λ min =2*π / (2*NA*d pixel / k1)=π*k1 / (NA*d pixel )

[0102] In a two-dimensional plane, the spatial frequency can be determined according to the azimuth angle β of the illumination. j Rotation, and can be decomposed into Cartesian components, where:

[0103] k x,i,j =k NA,ij cos(β j ) k y,i,j =k NA,ij sin(β j )

[0104] For each light source (ij), the Fourier transform of the corresponding image is set to the position (K) in the resulting Fourier space for image reconstruction. x,i,j ; K y,i,j ).

[0105] For a given optical system with maximum numerical aperture and different wavelengths in the illumination spectrum, the spatial frequency limit that needs to be calculated at all wavelengths and with a fixed k1 is wavelength dependent and is calculated precisely according to the following formula:

[0106] k(λ)=2*π*NA / (2*k1*λ)=π*NA / (k1*λ)

[0107] The factor 2*π in the above formula may depend on the way and method of normalization performed in the Fourier transform. (If necessary, the normalization / calibration of the Fourier transform is also performed in other ways.) This allows the opportunity to incorporate wavelength and bandwidth into the image reconstruction process in an inherent way.

[0108] The longest wavelength used has the lowest resolution limit that can be achieved by the spatial frequency associated with it. The shortest wavelength has the largest spatial frequency content in the Fourier plane. min To max All wavelengths in the range have the same intensity (see Figure 8 ), then for this Fourier space, we can get Fig. 9 Therefore, the short-wavelength light λ in Fourier space min The resolution limit is significantly higher than that of long-wavelength light max The resolution limit of is further out. In addition, Fig.10 A radial cross section through Fourier space is shown.

[0109] Although lower than k(λ max ) is constant (all frequencies contribute), but the contribution of each frequency varies across k(λ max ) decreases outside the spatial frequency k(λ max ) above, the long-wavelength Anteile can no longer contribute. Therefore, the contribution of these parts continues to decrease until finally even with the shortest wavelength λ min Can't contribute either.

[0110] Using this insight, a new approach was derived for the reconstruction (see Fig.11 ), according to which the angle of incidence for each illumination source or each illumination element can be determined or calibrated. Here, for example, a list of light source positions is obtained:

[0111] LS k =(α i, β j ) or (x i, y j )

[0112] In this case, α i is the polar angle, and β j is the azimuth.

[0113] For all used (α i ,β j ) or (x i,y j ) can be calculated as follows:

[0114] K x,i,j = cos(β j )sin(α i );

[0115] K y,i,j = sin(β j )sin(α i )

[0116] Or, alternatively

[0117] K x,i,j =K x,i = atan(x i / D);

[0118] K y,i,j =K y,j =atan(y j / D)

[0119] Thus, each Fourier transform pattern is located in Fourier space with its center point at position (K x,i,j ,K y,i,j ) or (K x,i ,K y,j ) at the site. Further reconstruction is carried out according to Fig.11 This is done in the iterative algorithm 12.

[0120] In an extended version of the reconstruction pipeline, the Fourier space can be filtered according to the maximum spatial frequency defined by the numerical aperture of the optical unit. The spectral bandwidth information can be taken into account in the algorithm.

[0121] Furthermore, there is a further potential for improvement in new approaches to the reconstruction process. On the one hand, spectral information based on color or wavelength band spectra is possible and meaningful through new approaches based on space vectors or illumination, which was not possible with the prior art. Furthermore, the Fourier transform of the original image can be filtered by an apodization filter according to the spectral distribution / wavelength distribution of the light source used to capture the corresponding image.

[0122] The wavelength spectrum is usually an idealized mask-shaped or rectangular distribution, which reaches λ min To max For square pixel sizes, the Fourier plane has a square shape. The step size of the sampling in the Fourier plane in the direction of the sensor coordinate axis depends on the number of pixels of the sensor in the corresponding axis. In the case of a square array, the sampling size in both axes is equal. Circles (see Fig. 9) shows the area of ​​the Fourier space of the image covered by the corresponding wavelength. The area of ​​the Fourier plane not covered by the circle is related to the following spatial frequencies in the image that are not accessible for the contribution of a specific wavelength to the entire image. This involves sampling frequencies that exceed the Nyquist theorem.

[0123] The above mentioned Fig.10 The Fourier plane profile in is of particular interest. It shows how the wavelength spectrum for a correct reconstruction appears. min and λ max The triangular shape between is a direct consequence of the assumption that the light intensity is the same for each wavelength. In real situations, light sources usually do not have such a uniform emission distribution, and optical units and samples do not have such an ideal transmission power.

[0124] In real-world situations there are effective transmission profiles for optical systems, which have e.g. Fig.12 The spectral power distribution curve (considered as a density function) passes through the specific wavelength to λ min Integrating and resulting in Fig.13 The weighted distribution function is shown in max and λ min The area under the curve between the maximum and the minimum is normalized to 1.

[0125] Taking this fact into account, the sketch of the radial section in the Fourier plane (see Fig.10 )according to Fig.14 changes. In this case, the spectral distribution is contained in the transfer function of the Fourier components in order to filter out the higher spatial frequency contributions. This transfer function can be interpreted as the following wavelength-dependent pupil function, which corresponds to the pupil stop in the pupil of the optical imaging system. From a functional point of view, in the case of spectrally corrected pupil imaging, this representation corresponds exactly to the optical function of the physical stop in the aperture plane, where the diameter of the physical stop corresponds to the maximum aperture NA of the objective. max Match.

[0126] Real optical systems do not have a constant transfer function over the entire frequency range from 0 to the maximum spatial frequency transmitted. In the real world, the frequency response is described by a modulated transmission frequency. For a frequency of 0, the modulated transmission frequency has a value of 1 or its maximum value, and for the maximum spatial frequency transmitted by the system, the modulated transmission frequency drops to 0 (see Fig.15 ). This effect must be compensated when reconstructing an image from a Fourier stack, if different spatial frequency patterns of the Fourier transforms from different illumination directions are combined.

[0127] The modulation transfer function (MTF) can be determined by any known method, for example from imaging a test structure and evaluating the contrast, or by determining characteristic variables of the imaging performance of the optical system (e.g. Seidel coefficients, Zernike coefficients) and calculating the MTF therefrom using a suitable form. The image or the Fourier spectrum of the image can then be corrected with the MTF to obtain an image of higher quality, preferably in the manner in which an ideal optical system would image the object. The corrected Fourier spectra can then be merged into a common spectrum with a greater spatial frequency content to generate a high-quality reconstructed image. For the merging process, a known (iterative) optimization algorithm 12 can be used for image search, such as the Gerchberg-Saxton or Levenberg-Marquardt algorithms, or similar methods for iteratively solving multidimensional and / or nonlinear problems.

[0128] In general, the entire optimization can also be performed in one step, as is the case in the prior art for Fourier stack imaging reconstruction. Fig.16 and 17 The functional flow chart in shows only a general strategy to implement the whole reconstruction process, which can be refactored without deviating from the reconstruction target framework.

[0129] exist Fig.16 In the example of , in step S1, an image is provided to an optimization algorithm 12 (e.g., Gerchberg-Saxton or Levenberg-Marquardt algorithm) for image generation. Within the scope of the algorithm 12, deconvolution of the image and the MTF is performed in step S2. In a subsequent step S3, a Fourier transform (FFT) to Fourier space is performed. In a subsequent step S4, these Fourier images are merged. In a subsequent step S5, an inverse Fourier transform for image generation is performed. Steps S3 to S5 can be iteratively executed, wherein in the iterative loop there can also be a step of comparing the current image with at least one of these input images from S1 or the optimized input image after step S2. Finally, a high-quality output image can be provided in step S6.

[0130] Alternatively, the reconstruction process can also be based on Fig.17 . In this case, first, step S1 involves providing an image to the optimization algorithm 12. However, therein, a Fourier transformation is now directly performed according to step S3. This is followed by step S7, in which the contributions of the imaging system and the object are separated in Fourier space. A merging of the real / Fourier images is then performed in step S8, and then an inverse Fourier transformation is performed again according to step S5, so that a high-quality output image can be provided according to step S6.

[0131] refer to Fig.18 and 19 The concept of combining illumination and sampling coordinate systems in Fourier space is vividly illustrated, with appropriate scaling chosen for the purpose. Fig.18 and 19 The basis is Figure 6 and 7 In the Fourier space for a planar rectangular illumination device or a spherical illumination device. In the center of the two Fourier spaces, there is the spatial frequency pattern of the optical sensor for illumination on the optical axis. A further spatial frequency pattern 16 of the optical sensor is plotted symbolically in the Fourier space (with Fig.11 The illumination coordinate system has a larger extension in the Fourier space coordinate system than the imaging coordinate system. In addition to the central image with coaxial illumination on the optical axis of the imaging coordinate system, multiple images can therefore be captured at other illumination angles. After merging these images in Fourier space, an inverse transformation can be performed to obtain a high-resolution image.

Claims

1. A method for generating an image of an object by Fourier stack imaging with the aid of a color correction optical unit (6) in the following manner: - illuminating the object using a plurality of lighting elements (4) arranged in a distributed manner at a corresponding plurality of locations in space, It is characterized in that - detecting a plurality of spatial frequency patterns (16) generated by illuminating the object with a single lighting element or a plurality of lighting elements from the plurality of lighting elements (4), respectively, - centering each spatial frequency pattern (16) at a position in Fourier space (4) corresponding to a nominal spatial frequency (8) of a corresponding lighting element (4) or a plurality of corresponding lighting elements, and - Reconstructing the image by using the ensemble of all the respectively centred spatial frequency patterns (16).

2. The method according to claim 1, wherein: A plurality of locations where the lighting elements (4) are arranged in a distributed manner in space are located on an area designed in the shape of a plane, an ellipse or a spherical shell portion.

3. The method according to claim 1 or 2, wherein: The respective nominal spatial frequency (8) is determined as a function of the angle of incidence of the respective lighting element (4) on the object or the position of the respective lighting element (4).

4. The method according to claim 3, wherein the corresponding nominal spatial frequencies (8) are calculated, a) If the two orthogonal spatial directions x and y have a uniform spacing P from each other x and P y If the plurality of lighting elements (4) are arranged in a rectangular shape, the corresponding nominal spatial frequency is calculated using the following formula: K x,i = atan(i*P x / D) and K y,j =atan(j*P y / D), where wherein D corresponds to the distance between the object plane (2) where the object is located and the illumination plane where the plurality of illumination elements (4) are arranged, or b) If a plurality of lighting elements are arranged in a spherical or elliptical shape, wherein the lighting elements are arranged at an azimuth angle of and the polar angle relative to the optical axis of the optical unit (6) is θ i , then the corresponding nominal spatial frequency is calculated using the following formula: and in is the uniform azimuthal spacing of the lighting elements (4) on the corresponding ring i.

5. A method according to any one of the preceding claims, wherein: Reconstructing the image comprises inverse Fourier transforming the population of all the respective centred spatial frequency patterns (16).

6. The method according to any of the preceding claims, wherein a spatial frequency domain (7) of the image used for reconstructing the image is limited based on a maximum spatial frequency defined by a numerical aperture of the color correction optical unit (6).

7. A method according to any of the preceding claims, wherein each spatial frequency pattern (16) is limited by means of a gradient or apodization filter, wherein the gradient or apodization filter is configured according to the wavelength distribution of the lighting elements (4).

8. Method according to any of the preceding claims, wherein each spatial frequency pattern (16) is corrected by an inverse modulation transfer function of the color correction optical unit (6).

9. The method according to claim 8, wherein: During reconstruction, each individual image obtained using a single illumination element or a plurality of illumination elements from the plurality of illumination elements (4) is freed from the modulation transfer function of the color correction optical unit (6) by deconvolution, each individual image corrected in this way is subsequently transformed into a corresponding corrected spatial frequency pattern, and only the corrected spatial frequency patterns are combined.

10. The method according to claim 8, wherein: During reconstruction, each individual image obtained using a single illumination element or a plurality of illumination elements from the plurality of illumination elements (4) is transformed by Fourier transformation into a respective spatial frequency pattern, each spatial frequency pattern subsequently being freed from the influence of the respective transfer function of the optical unit (6) and / or the illumination element (4) on the Fourier components of the respective spatial frequency pattern, and only the spatial frequency patterns corrected in this way are combined.

11. The method according to any of the preceding claims, wherein all spatial frequency patterns are merged during reconstruction of the image by means of an iterative optimization algorithm.

12. An apparatus for generating an image of an object by Fourier stack imaging, comprising: - a color correction optical unit (6) and - a lighting device (3) for illuminating the object with a plurality of lighting elements (4), wherein the lighting elements are arranged in a distributed manner at a respective plurality of locations in space, It is characterized in that - a detection device (10) for detecting a plurality of spatial frequency patterns, wherein the plurality of spatial frequency patterns are generated by illuminating the object with a single lighting element or a plurality of lighting elements of the plurality of lighting elements (4), respectively, and - computing means for centering each spatial frequency pattern (16) at a position in Fourier space corresponding to a nominal spatial frequency (8) of a respective lighting element or elements (4), and for reconstructing the image by using the ensemble of all the respective centered spatial frequency patterns (16).

13. A computer program comprising instructions which, during execution of the program by a device according to claim 12, cause the device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium comprising instructions which, when executed by the apparatus according to claim 12, cause the apparatus to perform the method according to any one of claims 1 to 11.

Images