Method and apparatus for generating a luminance corrected image

Abstract

The invention relates to a method for generating a brightness-corrected image (KB), wherein at least two corrected image sections (TB1,..., TBn) of an object (2) are captured in each case as a plurality of image pixels and combined by calculation. According to the invention, the method is characterized in that the image data of each corrected image section are captured during a capture duration, and the object (2) is displaced during the capture duration of a first corrected image section (TB1) in a first direction (R1) transverse to an optical axis of a detection beam path by at least one image pixel and during the capture duration of the other corrected image sections in a second direction (R2) transverse to the optical axis of the detection beam path by at least one image pixel, the smaller of the angles between the first and second directions (R1, R2) being greater than 0° and not more than 90°. The invention also relates to a device for implementing the method according to the invention.

Description

Technical Field

[0001] This invention relates to a method for generating a brightness-corrected image according to the preamble of the independent claim. Furthermore, this invention relates to an apparatus for carrying out the method. Background Technology

[0002] Reducing system-related imaging aberrations is crucial in many imaging methods, especially in microscopy. For example, such aberrations can arise from the influence of the optical elements used in the corresponding beam path, as well as from the technical limitations of the detector used. Furthermore, sample holders, such as coverslips, slides, microtiter plates, or the bottom of petri dishes, can cause imaging aberrations that are not directly caused by the object being imaged (the sample).

[0003] Correction of brightness differences (shadows; shading correction) in captured images is particularly important. This is especially significant if multiple image patches (slices) are captured and stitched together to form a complete image. In this process, individual image patches overlap by a specific area (overlap area; overlap), allowing the image patches to be stitched together with positional accuracy based on the information from the multiple captured images contained within them. If the image patches have different brightness levels, the observer will perceive it as an error.

[0004] Several methods for brightness correction have been disclosed in the prior art. In principle, these methods can be subdivided into two categories: i) reference-based methods and ii) purely computational methods.

[0005] Reference-based methods rely on a reference image (also known as a brightness-corrected image, hereinafter referred to as the corrected image), such as a reference image of a uniform sample, which should result in a uniform signal. Imaging aberrations that will occur anyway are contained in the reference image, and therefore the captured image of the sample can be corrected by subtracting the reference image from it. Many procedures for generating reference images are known (e.g., DE 10 2014 112 002 A1).

[0006] Instead of purely computational methods using reference images, captured images of the sample, which include imaging aberrations, are employed. The BaSiC tool software (published by Peng et al. in 2017) provides an example of such a method.

[0007] The following aims to emphasize reference-based correction methods.

[0008] For example, so-called camera-based brightness correction can be achieved as a global correction. In this process, a sample holder, such as a slide, is positioned in the beam path such that a region without a sample is captured. The optical conditions correspond to those during sample capture. For instance, the selected region contains the same coverslip and embedding medium as the sample. The captured image of this region can then be used as a reference image. This process is specific to currently used objective lenses and is particularly useful for routine image capture under transmitted light.

[0009] For example, the correction just described can be performed on each individual channel in a multicolor experiment and finds particular applications in the correction of reflected light fluorescence recording.

[0010] Modifications to the described correction method can be applied in a variety of ways. For example, multiple image slices of the sample-free region of a slide can be captured and averaged, for example, to eliminate sample contamination present in the image or to compensate for variance between image slices. This process can be implemented in a channel-specific manner.

[0011] It is also possible to capture multiple image slices of a sample and determine a reference image from them. In this case, it is assumed that, for example, the image structure is removed within the range that forms the mean, and only information about the shadows is retained. This process is particularly applicable to complex or intricate fluorescence images and requires many image slices, typically more than 200. This requires a slide with the desired fluorophore and the sample, the size of which allows for the capture of a sufficient number of image slices. In this case, channel-specific corrections can also be performed.

[0012] In another variation, the beam is focused on the sample, and then the slide is removed from the beam path. At least one image of the "free space" is captured and used as a reference image.

[0013] A particular challenge arises when calibrating fluorescence images in which the sample to be imaged is labeled with molecules (fluorophores) that can excite fluorescence. In this case, the inherent structure of the sample in the reference image should be prevented from being expressed in the reference image. Furthermore, bleaching should be avoided during the generation of the reference image, if possible, as this can lead to discrepancies in optical conditions between the creation of the reference image and sample capture. Summary of the Invention

[0014] The present invention aims to propose alternative options for generating reference images, thereby reducing the disadvantages of the prior art.

[0015] This objective is achieved by a method for generating a brightness-corrected image. Furthermore, the invention includes an apparatus for implementing this method. Advantageous development examples are also described below.

[0016] To generate a brightness-corrected image, at least two corrected image patches of the object are captured as multiple image pixels in each case. Therefore, each image patch consists of multiple image pixels (picture elements). The brightness-corrected image is formed by combining the corrected image patches through computation.

[0017] The method according to the invention is characterized in that image data of each correction image patch is captured during the capture duration, and the object is displaced by at least one image pixel in a first direction transverse to the optical axis of the detection beam path during the capture duration of the first correction image patch, and at least one image pixel is displaced in a second direction transverse to the optical axis of the detection beam path during the capture duration of the second correction image patch, the smaller of the angles between the first and second directions being between 0° and no more than 90°, advantageously between 45° and up to 90°.

[0018] The capture duration specifies a duration during which an image, particularly a corrected image patch, is captured by the detector. In this case, the capture duration can be the duration of a frame and / or the open duration of the detector shutter. Therefore, depending on the detection beam path and the detector, the capture duration can be determined by the time control of the readout of the detector elements of the detector, or particularly mechanically and electronically by the opening and closing of the detection beam path, and can be set in this respect. In other embodiments of the invention, a light source for illuminating the object without mechanical shielding (without a "shutter") can also be connected. Illumination can be achieved, for example, by reflected light for exciting fluorescence radiation, or in a bright field, particularly by transmitted light.

[0019] Relative motion between the object and the detection beam path can be achieved through an adjustable sample stage and / or by pivoting or shifting the detection beam path. Advantageously, the relative motion is generated by a motor and caused by the movement of the sample stage.

[0020] This invention utilizes the effect of existing blurred or trailing image structures. To this end, a deliberate blurring of the corrected image patch is produced by means of the object's significant movement during the capture duration, which prevents sharp imaging of the object's structure from occurring.

[0021] Since a single movement in the first or second direction results in the formation of stripes in the captured corrected image patch, the two directions are advantageously at an angle of at least 45° relative to each other, so as to achieve high reliability and similar blurring by calculating the stripe formation contained in the result of combining at least two corrected image patches.

[0022] A brightness-corrected image obtained by calculating a combination of at least two corrected image patches can then be used to correct the captured image of the sample.

[0023] To obtain a sufficiently high-quality luminance-corrected image for practical use, currently available luminance-corrected images can be analyzed and evaluated according to predefined quality criteria. If the quality criteria are not met, the existing luminance-corrected image can be combined with other images of the object through calculation. Optionally, this step can be repeated until the quality criteria are met. The variance of the luminance-corrected image is a possible quality criterion. The goal is to produce a low-variance luminance-corrected image, where a uniform distribution of luminance intensity values ​​exists.

[0024] Other corrected image slices can also be captured and new luminance-corrected images generated from them, optionally using previous luminance-corrected images as well. In this case, similarly, the process can be terminated, for example, once the variance of the luminance-corrected image is sufficiently small.

[0025] The capture duration (i.e., the time period for capturing calibration image patches in various cases) and / or the angle between the first and second directions can be selected, optionally taking into account corresponding predefined allowable ranges. In this way, the parameters during the capture of calibration image patches can be adapted to the results of inspections for current requirements and / or quality criteria. If, for example, the expected variance of the brightness-corrected image is obtained, the capture of other calibration image patches can be completed. For this purpose, a control loop can be implemented between the detector, the evaluation unit, and the control unit. Depending on the image data captured by the detector and the results of the evaluation unit analyzing the image data, information can be transmitted to the control unit, which then generates control commands based on this information. For example, the control commands can be used to set the illumination, the movement of the sample stage, and / or the settings of the detector in a controlled manner, particularly the capture duration related to the direction and / or speed of movement of the sample stage.

[0026] Corrected image patches in the first and second directions can image the same area of ​​the object. In other configurations of the method, it is also possible to image partially different or completely different areas of the object.

[0027] In principle, this invention can be used in any microscopy method that utilizes contrast. Therefore, a sample holder can be selected as the object, for example, to reduce system-related imaging aberrations. This method can be particularly advantageously used in fluorescence microscopy. In this case, a sample containing molecules (e.g., fluorophores) emitting radiation to be captured and detected is selected as the object.

[0028] The corrected image sheet according to the method of the invention can be generated by manually or by driving the drive of the sample stage using generated control commands. Image capture can be achieved by controlling the detector to capture frames, that is, by assigning the captured image data to a single image. A shutter placed in front of the detector can also be opened (e.g., mechanically or electronically) during the corresponding capture duration. The object can be illuminated continuously or only during capture. The latter is used to protect the object, for example, to reduce unwanted bleaching.

[0029] An apparatus for implementing the method according to the invention is used to capture images of objects present on an adjustable sample stage in a sample chamber, and includes a detection beam path, a detector unit, and a control unit for generating control commands. The control unit is configured to generate control commands.

[0030] The operating mode for generating brightness-corrected images can be implemented via control commands. In this mode, the sample stage is displaced by at least one image pixel in a first direction, transverse to the optical axis of the objective lens or detection beam path, while simultaneously recording a first corrected image of the object. Conversely, the sample stage is displaced by at least one image pixel in a second direction, transverse to the optical axis of the objective lens or detection beam path, while simultaneously recording a second corrected image of the object. In this configuration, the smaller of the angles between the first and second directions is greater than zero degrees and not greater than 90 degrees. Advantageously, the angle values ​​are between 45° and 90°. In other configurations, switching can occur between the first direction, the second direction, and optionally at least one other direction during the recording of the corrected image. The switching between the directions can be implemented according to a fixed scheme. In other configurations, the switching of directions can occur randomly.

[0031] Therefore, this method can also be used to generate corrected images. In this case, multiple image patches of an object can be captured, each image patch being captured as multiple image pixels, and each image patch overlapping in area with at least one other captured image patch in an image patch overlap region having a defined minimum size. By stitching the image patches together to produce a composite image, image data from the multiple captured image pixels in the overlap region is used to arrange the patch images in the correct positions and with the correct orientation relative to each other. Each captured image patch and / or composite image is corrected by a brightness correction image, where each image pixel of the image patch and / or composite image is combined by calculating image data from the corresponding pixel in the brightness correction image.

[0032] The brightness-corrected images generated according to the present invention can potentially be used in all contrast-based microscopy methods, and are particularly suitable for fluorescence microscopy and bright-field microscopy, especially in transmitted light bright fields. They can also be used in methods that capture and combine non-overlapping image slices through computation. Attached Figure Description

[0033] The invention will now be explained in more detail with reference to the accompanying drawings, in which:

[0034] Figure 1 A schematic example of generating a brightness-corrected image according to the prior art is shown;

[0035] Figure 2 A schematic representation of the configuration of the method according to the invention is shown; and

[0036] Figure 3 A schematic representation of an exemplary embodiment of the device according to the present invention is shown. Detailed Implementation

[0037] Existing methods for generating luminance-corrected images KB (also simply called corrected images KB) are known in the art. Figure 1 The diagram illustrates this very schematically. Multiple (corrected) image patches TB1, TB2 to TBn are captured, and the individual pixels of the image patches are combined by applying alternatively selected mathematical methods (represented by the feature sequence: ∑ / n), such as averaging, and this is used to create the corrected image KB. It is evident that, compared to the first image patch TB1, the corrected image KB exhibits a smaller variance and more uniform luminance intensity values.

[0038] Figure 2 The configuration of the method according to the invention is depicted. The previous line shows the first to nth corrected image patches TB1 to TBn, which are placed in the first direction R1 (see [reference]) during the capture duration of detector 5. Figure 3 The images were captured during the upward displacement. As an example, the next line shows the first to nth corrected image patches TB1 to TBn captured in the second direction R2. In this case, the corresponding variance of each corrected image patch TB1 to TBn has, for example, a value ranging from 18 to 80, while the variance of the resulting luminance-corrected image KB is, for example, only 3.

[0039] In this exemplary embodiment, the first direction R1 and the second direction R2 lie in a common plane, and the angle between the first direction R1 and the second direction R2 is 90°. For example, the common plane is an xy plane parallel to the sample stage 1. Figure 3 ).

[0040] The image patches TB1 to TBn captured in the corresponding directions R1 and R2 are combined by calculation, for example, averaging, and combined by calculation to form a brightness-corrected image KB.

[0041] In this case, the image patches TB1 to TBn captured in one of directions R1 or R2 can be calculated and combined, and then the calculated composite image (not shown) can be combined with the resulting image of the image patches TB1 to TBn captured in the other direction R1 or R2 to form a brightness-corrected image KB.

[0042] In various cases, image patches TB1 to TBn can be calculated by combining the first and second directions R1 and R2. For example, the first image patch TB1 is calculated by combining the first and second directions R1 and R2. The resulting image can be combined with a resulting image, such as a second image patch TB2, etc., in the first and second directions R1 and R2.

[0043] Furthermore, in various cases, image pairs of the first and second directions R1, R2 can be generated to produce result images for TB1, TB2 to TBn. The corrected image KB is then determined based on the result images. Depending on whether all result images are used and / or whether different weights are applied to individual result images, this variation results in a corrected image KB that differs from the process described above.

[0044] The goal is to obtain a brightness-corrected image KB with small variance, meaning the intensity values ​​are very uniformly distributed and the structure of the sample itself is no longer present. Errors that may exist in the object or optical path should be largely compensated for and no longer cause structuring in the brightness-corrected image KB.

[0045] Figure 3 An exemplary embodiment of an apparatus for implementing the method according to the invention is schematically depicted. A sample 2 may be arranged on a controllable and adjustable sample stage 1. It is illuminated by a light source 3. Detection radiation emitted from the sample 2 (e.g., fluorescence radiation and / or reflected components of illumination radiation) is recorded by an objective lens 4 and imaged along the optical axis (indicated by dashed lines) in the z-axis direction onto a spatially resolved detector 5, such as a CCD, CMOS, or sCMOS chip, a SPAD (single-photon avalanche diode) array, or an array of multiple PMTs (photomultiplier tubes). The image data captured by the detector 5 is transmitted to an evaluation unit 8 and there, through calculation, combined to form a corrected image KB.

[0046] In an alternative embodiment of the device, illumination can be achieved using transmitted light, that is, for example in a bright field, by a properly positioned light source 3 (shown by dashed lines in an exemplary and optional manner).

[0047] In other exemplary embodiments, the evaluation unit 8 may also perform correction of the captured image of sample 2, wherein the captured image data within the range is combined with the generated corrected image KB by calculation.

[0048] Furthermore, there is a control unit 6, which may be, for example, a computer or an FPGA, and is configured to generate control commands. Generating control commands in this way allows for the execution of commands related to… Figure 2 All modes described or selected modes. The control unit 6 is connected in each case to the drive unit 7, detector 5, and light source 3 of the sample stage 1 in a manner suitable for data transmission.

[0049] The evaluation unit 8 may optionally be connected to the control unit 6, for example, to adjust the control commands generated thereon based on the captured image data or the degree to which selected quality criteria are met.

[0050] List of reference numerals

[0051] 1 Sample Stage

[0052] 2. Samples, objects

[0053] 3 light sources

[0054] 4 objective lenses

[0055] 5 detectors

[0056] 6 control units

[0057] 7 drive units

[0058] 8 assessment units

[0059] TB1, ..., TBn are the first image slice, ..., the nth image slice.

[0060] KB brightness-corrected image

[0061] R1 First Direction

[0062] R2 Second Direction

Claims

1. A method for generating a luminance-corrected image (KB), wherein, - Each of at least two corrected image patches (TB1, ..., TBn) of the object (2) is captured as a plurality of image pixels, and the corrected image patches (TB1, ..., TBn) are combined by calculation. Its features are, - Image data of each correction image patch (TB1, ..., TBn) is captured during the capture duration, and the object (2) is displaced by at least one image pixel in a first direction (R1) transverse to the optical axis of the detection beam path during the capture duration of the first correction image patch (TB1), and displaced by at least one image pixel in a second direction (R2) transverse to the optical axis of the detection beam path during the capture duration of the other correction image patches (TB1, ..., TBn), wherein the smaller of the angles between the first direction and the second direction (R1, R2) is greater than 0° and does not exceed 90°.

2. The method according to claim 1, characterized in that, The sample holder was selected as the object (2).

3. The method according to any one of the preceding claims, characterized in that, A sample containing molecules that emit radiation to be captured and detected is selected as the object (2).

4. The method according to claim 1 or 2, characterized in that, The object (2) is illuminated by transmitted light.

5. An apparatus for capturing an image of an object (2) present on an adjustable sample stage (1) in a sample chamber, comprising a detection beam path, an objective lens (4), a detector (5), and a control unit (6). in, The detector is configured to capture at least two corrected image patches (TB1, ..., TBn) of the object (2) as a plurality of image pixels, and the corrected image patches (TB1, ..., TBn) are combined by calculation. The control unit (6) is configured to generate control commands, and an operation mode for generating a brightness correction image (KB) is achievable through the control commands. Within the range of this operation mode, the sample stage (1) is displaced by at least one image pixel in a first direction (R1) transverse to the optical axis of the detection beam path, while recording a first correction image patch (TB1, ..., TBn) of the object (2). The sample stage (1) is displaced by at least one image pixel in a second direction (R2) transverse to the optical axis of the detection beam path, while recording other correction image patches (TB1, ..., TBn) of the object (2). The smaller of the angles between the first direction and the second direction (R1, R2) is greater than 0° and not greater than 90°.

6. The device according to claim 5, characterized in that, The object (2) includes a sample holder.

7. The device according to claim 5 or 6, characterized in that, The object (2) includes a sample containing molecules that emit radiation to be captured and detected.

8. The device according to claim 5 or 6 further includes a light source configured to illuminate the object (2) with transmitted light.

Images