Method and apparatus for producing an overview contrast image of a sample carrier in a microscope

Abstract

The invention relates to a method and a device for generating an overview contrast image of a sample carrier in a microscope. The method comprises illuminating the sample carrier with transillumination light having a two-dimensional illumination array with individually switchable single light sources; taking a plurality of different raw overview images of the sample carrier, wherein for each raw overview image one of the individual single light sources is switched on; generating intermediate images from the raw overview images, wherein a brightness ranking of the raw overview image pixels is defined, each intermediate image is assigned exactly one brightness ranking, in each intermediate image for each pixel the raw overview image is sought in which the respective pixel has the brightness ranking to which the respective intermediate image is assigned, and the brightness value of the respective pixel of the found raw overview image is used; and combining the intermediate images to an overview contrast image, wherein the intermediate image assigned to the first brightness ranking is not considered and not included into the overview contrast image.

Description

Technical Field

[0001] This invention relates to a method for generating overview contrast images of a sample carrier in a microscope that images a sample arranged on a sample carrier. The sample carrier, containing the sample, is illuminated with transmitted light from a two-dimensional illumination array having individually switchable single light sources, thereby enabling the capture of multiple different original overview images of the sample carrier. Furthermore, the invention also relates to an apparatus for generating overview contrast images of a sample carrier to be arranged in a sample volume in a microscope. This apparatus includes a camera, a two-dimensional illumination array, and a processing unit. The camera has an image field that captures the sample volume on which the sample carrier to be arranged is located. The two-dimensional illumination array has individually switchable single light sources and illuminates the sample volume with transmitted light. The processing unit drives and reads the camera to capture multiple different original overview images of the sample carrier. Background Technology

[0002] A method for generating and analyzing overview contrast images is known from DE 2017 111 718 A1. At least two raw overview images are captured by different illuminations of the sample, and these are combined to form an overview contrast image. Different illuminations are achieved by moving the sample laterally along the illumination pattern, illuminating it at different times, or illuminating it with different illumination patterns.

[0003] US 8,400,710 B2 discloses an apparatus for observing samples. Bright and dark areas are defined by an illumination pattern. This enables the forwarding of images to a processing device via dark-field observation. Summary of the Invention

[0004] Methods and apparatus should be provided for generating an overview contrast image of a sample carrier in a microscope used for imaging the sample.

[0005] The present invention is defined in the methods and apparatus described below. Advantageous improvements will also be given below. Preferred embodiments are applied in the same manner to the methods and apparatus.

[0006] This invention provides a method and apparatus for generating an overview contrast image of a sample carrier in a microscope having a sample volume in which a sample carrier containing the sample can be arranged, and then the sample is imaged. In step a), the sample volume, i.e., the sample carrier on which the sample is typically arranged, is illuminated with transmitted light from a two-dimensional illumination array having individually switchable single light sources. For example, an LED array having N single LEDs is used as the two-dimensional illumination array.

[0007] In step b), multiple raw overview images of the sample carrier are captured. This is performed using a camera within the device that captures the sample volume and is connected to the processing unit via electrical wiring or radio. All raw overview images are captured using the same camera, and preferably using the same image capture parameters. The camera's position remains unchanged throughout the capture of all raw overview images.

[0008] Each original overview image is generated by precisely activating one of the single light sources, wherein a different single light source is activated for each original overview image, thereby capturing N original overview images. In this device, the processing unit is configured to drive and read from the camera to capture multiple different original overview images. Here, the processing unit drives a two-dimensional illumination array to individually activate single light sources. The processing unit is configured to generate each original overview image in such a way that it activates precisely one of the single light sources, wherein the processing unit activates one individual single light source for each original overview image, thereby enabling the processing unit to capture N original overview images.

[0009] Therefore, N original overview images are obtained by sequentially activating individual light sources. However, not all individual light sources need to be used. Therefore, N can be less than the number of available individual light sources. Preferably, N corresponds to the total number of individual light sources. The individual light sources used can be activated sequentially (i.e., from the first to the Nth individual light source) or in any other arbitrary order.

[0010] When the sample carrier is arranged within the sample volume, the N original overview images contain not only the sample structure but also structures that are not classified as part of the sample, making it difficult to locate the sample in the original overview images. These non-sample structures are, for example, components of the image acquisition process, such as a single light source, or parts of the sample carrier, such as the wall of a vessel. They can cause interference in the overview contrast images.

[0011] Therefore, in step c), intermediate images are generated and then combined into an overview contrast image. Intermediate images suppress interference. The process of generating intermediate images is as follows: First, a specific number of brightness rankings are defined for the pixels of the original overview image. Then, each intermediate image to be generated is matched with exactly one brightness ranking. Each intermediate image is generated by finding, for each pixel of the intermediate image, the original overview image that has a brightness ranking that matches the corresponding intermediate image at that pixel. Then, the brightness value of the corresponding pixel of the found original overview image is used for the intermediate image pixel. When combining the images into the overview contrast image, the intermediate image with the highest brightness ranking is not considered and is not included in the overview contrast image. For example, the intermediate images are combined into the overview contrast image by averaging, but other image processing methods are also possible. In this device, the processing unit is configured to perform this step.

[0012] In this implementation, the first intermediate image (ranked highest in brightness) contains images of all single light sources that are considered overstrained regions in the image and thus mask image information. For this reason, the first intermediate image is not considered when combining intermediate images into an overview contrast image. Its pixels would simply be an overstrained region that might mask image information. Therefore, by ignoring the intermediate image corresponding to the first brightness ranking, overstrained pixels are not considered in the overview contrast image.

[0013] The term "ranking" refers to the order of brightness, hence a hierarchy. Absolute brightness is not important here. The intermediate image associated with the first brightness ranking has the brightness value of the brightest pixel found in the corresponding location across all the original overview images. This also applies to other brightness rankings that contain progressively darker pixels.

[0014] Preferably, the number of intermediate images corresponds to the number of brightness rankings. For example, if 12 brightness rankings are defined, 12 intermediate images are generated. Eleven of these are combined into an overview contrast image because the intermediate image with the highest brightness ranking is not included. Generating intermediate images from N original overview images represents the maximum computational cost. The computational cost increases or decreases proportionally with the number of intermediate images M. For example, if the number is chosen to be M = 6, the computational cost and associated time required to compute the intermediate images are less compared to when M = 12.

[0015] The first intermediate image contains all image points ranked highest in brightness, so it typically corresponds at each pixel to the maximum projection over all N original overview images. The intermediate image associated with the second brightness ranking has a brightness value at each pixel equal to the brightness value of the second brightest pixel in that location within the image from all the original overview images. This can also be achieved through a correspondingly set maximum projection. The same applies to the other intermediate images / brightness rankings.

[0016] These measures improve sample visibility and identifiability because the structure of the sample is emphasized in the overview contrast image. The method and apparatus for generating overview contrast images of samples in the microscope clearly image the sample carrier and the sample itself, which is crucial for further examination. This makes it easier for the user to interact with and navigate the sample. Furthermore, automated sample evaluation is achieved by automatically identifying the type of sample and sample carrier using the processing device, allowing for the autonomous setting of specific imaging parameters on the microscope. This significantly improves user-friendliness and demonstrates greater automation of the microscope. Additionally, automatic position determination of the sample in the overview contrast image is possible, enabling the microscope objective to be positioned without user intervention to place the sample within the microscope's imaging area. This measure effectively reduces the time required for the user until the first image is captured.

[0017] Preferably, intermediate images are generated once at least two original overview images have been captured. The intermediate images are continuously updated as additional original overview images are added. This further reduces the time required for observation using a microscope. This process can also be performed continuously, meaning that the intermediate images and overview contrast images are constantly updated. This is advantageous for samples that are varying in size.

[0018] In this implementation, the intermediate images are brightness normalized before being combined into an overview contrast image by transforming the brightness domain of each image point, which is determined by its respective brightness ranking in each intermediate image, to a larger brightness domain, preferably the same for all intermediate images. This is preferably accomplished using min-max normalization, percentile normalization, or histogram equalization. Brightness normalization results in all intermediate images having substantially the same brightness or the same brightness domain, even though they were initially in different brightness rankings, i.e., having different brightness levels before brightness normalization. The type and parameters of brightness normalization determine how bright the structures can be identified in their respective intermediate images. For example, if 10% / 90% percentile normalization is performed, the structures are brighter compared to those in the case of 2% / 98% percentile normalization. This brightness normalization is performed by a processing unit in the device. Since most of the computation occurs when generating the intermediate images, brightness normalization can also be performed multiple times using multiple settings.

[0019] In the modified scheme, two overview contrast images are generated. A first overview contrast image is generated from the intermediate image without brightness normalization. This first overview contrast image is particularly suitable for displaying very bright structures in the intermediate image, such as glass edges. A second overview contrast image is generated from the intermediate image using the aforementioned brightness normalization. It is more suitable for darker structures in the intermediate image. Preferably, the first and second overview contrast images are combined to obtain an improved overview contrast image, which displays both very bright and very dark areas equally well.

[0020] Preferably, the overview contrast image is further processed. A typical example is white balance. The results and intermediate results are preferably displayed to the user on the imaging device. Attached Figure Description

[0021] The invention will now be explained in more detail by way of example with reference to the accompanying drawings. Wherein:

[0022] Figure 1 A flowchart illustrating a method for generating an overview contrast image of a sample carrier in a microscope;

[0023] Figure 2 A schematic diagram illustrating the generation of the intermediate image is shown; and

[0024] Figure 3 This illustrates a device for producing an overview contrast image of a sample carrier in a microscope. Detailed Implementation

[0025] Figure 1 The text shows the use of... Figure 3The flowchart illustrates a method performed by the microscope 1 shown for generating an overview contrast image. Here, the sample volume 3, in which the sample carrier 2 containing the sample 4 is arranged, is imaged. In the first step S1, the sample carrier 2 is illuminated by transmitted light from a two-dimensional illumination array 6, which has individually switchable single light sources 7. For example, an LED array with a single LED is used as the two-dimensional illumination array 6. Here, the single light sources 7 are activated one by one, thereby obtaining multiple different illumination states.

[0026] In the second step S2, a raw overview image R of the sample volume 3 is captured for each illumination condition, i.e., the image is captured according to the illumination conditions. Figure 3 The embodiment includes a raw overview image of the sample carrier 2 containing sample 4. Raw overview images R1 to Rn are thus generated by turning on exactly one of the single light sources 7 for image capture. Therefore, individual single light sources 7 are turned on for each raw overview image R1 to Rn, thereby capturing N raw overview images R1 to Rn. Each raw overview image R1 to Rn is illuminated with a different single light source. All raw overview images R1 to Rn are captured using the same camera. The camera's positioning remains unchanged for capturing all raw overview images R1 to Rn. Preferably, N corresponds to the number of single light sources 7. However, it is not necessary to use all available single light sources. The single light sources 7 can be activated sequentially (i.e., from the first to the Nth single light source 7) or in any other order.

[0027] In the original overview images R1 to Rn, in addition to the structure of sample 4, there are structures that are not classified as sample 4, making it difficult to locate sample 4. These structures that are not classified as sample 4 are, for example, elements of the image capturing process, such as a single light source 7.

[0028] To suppress these interferences, intermediate images Z are generated in the third step S3, and then combined into an overview contrast image in the fourth step S4.

[0029] Intermediate image Z suppresses interference. To generate intermediate image Z, a specific number of brightness rankings are first defined in step S3. These brightness rankings relate to the brightness of pixels P(i,j) in the original overview images R1 to Rn. This yields the brightest pixel P(i,j), the second brightest pixel P(i,j), and so on. The term "brightness ranking" therefore indicates an ordering based on brightness, but not an absolute brightness value. Each intermediate image Z to be generated corresponds to exactly one brightness ranking.

[0030] Then, each intermediate image Z is generated as follows: for each pixel P(i,j) of the intermediate image, a pixel P(i,j) with a brightness ranking corresponding to its respective intermediate image Z is found in the original overview images R1 to Rn. The brightness value of the found pixel is used and applied to pixel P(i,j) of the intermediate image Z. This process is performed in... Figure 2 It is displayed in the middle.

[0031] Figure 2 A schematic diagram illustrates the generation of an intermediate image Z from original overview images R1 to Rn. Each of the original overview images R1 to Rn is represented by a pixel P(i,j). A pixel P(i,j) represents a position in the image within a raster graphic and is assigned a pixel value, representing the brightness value of the image at that position. For each intermediate image Z1 associated with a first brightness ranking, for each pixel P(i,j), the original overview images R1 to Rn that contain the pixel P(i,j) with the first brightness ranking (i.e., the pixel P(i,j) with the largest pixel value) are searched, and the brightness value of that pixel is assigned to pixel P(i,j) in the intermediate image Z1. Figure 2 In this model, brightness is encoded by shading / blackening pixel P(i,j). For example, if pixel P(1,1), i.e., the pixel in row 1, column 1, is taken, then the brightness value of the brightest pixel P(1,1) found at P(1,1) in all the original overview images R1 to Rn is used for the intermediate image Z1. In the example shown, pixel P(1,1) of the original overview image R1 is found. For pixel P(1,3), pixel P(1,3) from R3 is found, for example.

[0032] This method also applies to the intermediate image Z2 associated with the second brightness ranking. To this end, for P(1,1), we search among all the original overview images R1 to Rn for the original overview image that has the second brightest pixel among all pixels P(1,1), i.e., the pixel P(1,1) with the second brightness ranking. Figure 2 In the above, the pixel P(1,1) found at P(1,1) is from Rn, and the corresponding pixel P(1,3) found at P(1,3) is from R2.

[0033] Therefore, this method is then performed on all other intermediate images Z. The result is that each intermediate image Z has only pixels P(i,j) corresponding to its brightness ranking.

[0034] In the fourth step S4, the intermediate images Z are then combined into a general contrast image, wherein the intermediate image Z1 with the first brightness ranking is not considered and is not included in the general contrast image. This suppresses the overshoot effect. This is because the brightest pixel P(i,j), i.e., the pixel with the highest brightness ranking, will be considered an overshoot area and will not be used, thus not obscuring any image content in the overview contrast image. In the implementation, the pixels P(i,j) not considered in the overview contrast image correspond to the images of the single light source 7 that are respectively turned on to capture the original overview images R1 to Rn.

[0035] Furthermore, the visibility and identifiability of sample 4 were improved because the structure to be attributed to sample 4 was highlighted by emphasizing the contrast between sample 4 and its surrounding environment in the overview contrast image.

[0036] In the device, the processing unit 14 is configured to perform this step.

[0037] For example, the intermediate images Z can be combined into an overview contrast image by averaging, but other image processing measures are also possible, such as min-projection, maximal projection, or median-projection.

[0038] Preferably, once at least two original overview images R have been captured, the intermediate image Z is generated. Then, as additional original overview images R are added, the intermediate image is continuously updated. This reduces the required time.

[0039] Optionally, after generating the intermediate image, the intermediate image Z can be luminance normalized by transforming the luminance domain of its pixels P(i,j) to a larger luminance domain, preferably the same for all intermediate images Z. Here, for example, the luminance domain containing luminance values ​​ranging from 55% to 60% can be "expanded" to a luminance domain ranging from 0% to 100%. This is preferably accomplished using min-max normalization, percentile normalization, or histogram equalization. This achieves the luminance equalization described above for the intermediate image Z.

[0040] Preferably, the overview contrast image is further processed. For example, typically white balance.

[0041] exist Figure 3 The apparatus for generating an overview contrast image of sample carrier 2 in microscope 1 is shown. Along the optical axis OA, microscope 1 captures sample volume 3 with an objective lens (not shown separately), in which sample 4 is arranged on sample carrier 2. Sample volume 3, and therefore sample 4, is illuminated by transmitted light from a two-dimensional illumination array 6 having a single light source 7. The image field 8 of camera 10 is pointed towards sample volume 3, and thus towards sample carrier 2. Illumination array 6 and camera 10 are either as... Figure 3The device shown is connected to the processing unit 14 via electrical line 12 or via radio or the like. The processing unit 14 is in turn connected to the display device 16 via electrical line 12 or radio or the like.

[0042] Camera 10 captures the sample volume 3 and the sample 4 on the sample carrier 2 within the sample volume in its image field 8. Processing device 14 drives and reads the camera 10 to capture different raw overview images R1 to Rn of the sample volume 3. Here, processing device 14 also controls the two-dimensional illumination array 6 to individually activate single light sources 7, which have individually switchable single light sources 7 illuminating the sample volume 3 with transmitted light. For example, an LED array with individual LEDs can be used as the two-dimensional illumination array 6, with each individual LED activated sequentially or in any order by processing device 14. The positioning and orientation of camera 10 remain unchanged throughout the capture of the raw overview images R1 to Rn.

[0043] The processing device includes, for example, a processor and RAM and is configured such that it generates each of the original overview images R1 to Rn by turning on exactly one of the single light sources 7, wherein the processing device turns on an individual single light source for each original overview image R1 to Rn, thereby digitally capturing N original overview images. The processing device, based on... Figure 1 The flowchart illustrates the method for generating and processing the original overview images R1 to Rn. Preferably, the processing device displays the results and / or intermediate results to the user on a display device 16, such as a monitor or display.

[0044] List of reference numerals

[0045] 1. Microscope

[0046] 2 Sample carrier

[0047] 3. Sample volume

[0048] 4 samples

[0049] 6 Illumination Array

[0050] 7 Single light source

[0051] 8 Image Field

[0052] 10 cameras

[0053] 12 power lines

[0054] 14 Processing device

[0055] 16 Display devices

[0056] OA optical axis

[0057] Steps S1~S4

[0058] Z-intermediate image

[0059] P(i,j) represents the pixel (row i, column j).

[0060] P(1,1) pixels (row 1, column 1)

[0061] R1~Rn Original overview images

Claims

1. Method for generating an overview contrast image of a sample carrier in a microscope, which microscope images a sample arranged on the sample carrier, wherein a) the sample carrier is illuminated with transmitted light from a two-dimensional illumination array, which has individually switchable single light sources, b) a plurality of different raw overview images of the sample carrier are taken, characterized in that each raw overview image is generated by switching exactly one of the single light sources, wherein for each raw overview image exactly one of the individual single light sources is switched on, so that N raw overview images are taken, c) the raw overview images are used to generate intermediate images by c1) defining a brightness ranking of the pixels of the raw overview images, c2) assigning each intermediate image to be generated to exactly one brightness ranking, and c3) in each intermediate image, for each pixel of the intermediate image, the raw overview image is sought which has the brightness ranking assigned to the respective intermediate image at the respective pixel, and the brightness value of the respective pixel of the sought raw overview image is used, and d) the generated intermediate images are combined into an overview contrast image, wherein the intermediate image assigned to the first brightness ranking is not considered and is not included in the overview contrast image.

2. The method of claim 1, wherein, The seeking carried out in sub-step c3) includes a minimum / maximum projection.

3. The method of claim 1, wherein, The generation of the intermediate images is started as soon as at least two raw overview images are taken, and the intermediate images are constantly updated when further raw overview images are added.

4. The method according to any one of claims 1 to 3, characterized in that, After the generation of the intermediate images, in step d) the intermediate images are brightness-normalized after their generation by converting the domain for brightness which the pixels have due to the respective brightness ranking of each intermediate image onto a larger brightness domain.

5. The method according to any one of claims 1 to 3, characterized in that, After the generation of the intermediate images, in step d) the intermediate images are brightness-normalized after their generation by converting the domain for brightness which the pixels have due to the respective brightness ranking of each intermediate image onto a larger, identical brightness domain for all intermediate images.

6. The method of claim 5, wherein, The brightness-normalization of the intermediate images is carried out by means of minimum-maximum normalization, percentile normalization or histogram equalization.

7. The method according to any one of claims 1 to 3, characterized in that, The overview contrast image is reworked by image processing measures.

8. The method according to any one of claims 1 to 3, characterized in that, In step d) the intermediate images are combined into an overview contrast image using averaging, minimum projection, maximum projection and / or median projection.

9. Device for generating an overview contrast image of a sample carrier to be arranged in a sample volume in a microscope, which device has - a camera, which has an image field which captures the sample volume, - a two-dimensional illumination array, which has individually switchable single light sources and illuminates the sample volume with transmitted light, and - a processing means, which drives and reads out the camera in order to take a plurality of different raw overview images of the sample volume, characterized in that - the processing means drives the two-dimensional illumination array in order to individually switch on the single light sources, - the processing device is configured such that it generates the intermediate images from the raw overview images by the processing device defining a brightness ranking for the pixels of the raw overview images, by each intermediate image to be generated being assigned exactly one brightness ranking, and in each intermediate image, for each pixel of the intermediate image, finding in the raw overview images the raw overview image which has at the respective pixel the brightness ranking assigned to the respective intermediate image, and using the brightness value of the respective pixel of the found raw overview image, and - the processing device is configured such that it combines the generated intermediate images into an overview contrast image, wherein the processing device does not take into account and does not include into the overview contrast image the intermediate image assigned to the first brightness ranking. The processing device is configured such that it uses min / max projection when finding pixels in the raw overview images.

10. The apparatus of claim 9, wherein, The processing device is configured such that it starts generating the intermediate images as soon as at least two raw overview images have been taken, and that it constantly updates the intermediate images when further raw overview images are added.

11. The apparatus of claim 9, wherein, The processing device is configured such that it brightness normalizes the intermediate images after generating the intermediate images by the processing device converting the domain for brightness which the pixels have due to the respective brightness ranking of each intermediate image onto a larger brightness domain.

12. The apparatus of any one of claims 9 to 11, wherein, The processing device is configured such that it brightness normalizes the intermediate images after generating the intermediate images by the processing device converting the domain for brightness which the pixels have due to the respective brightness ranking of each intermediate image onto a larger, identical brightness domain for all intermediate images.

13. The apparatus of any one of claims 9-11, wherein, The processing device is configured such that it brightness normalizes the intermediate images by means of min-max normalization, percentile normalization or histogram equalization.

14. The apparatus of claim 12, wherein, The overview contrast image is reprocessed with image processing measures.

15. The apparatus of any one of claims 9-11, wherein, The processing device is configured such that it combines the intermediate images into an overview contrast image using averaging, min projection, max projection and / or median projection.

16. The apparatus of any one of claims 9-11, wherein, The processing device is configured such that it combines the intermediate images into an overview contrast image using averaging, min projection, max projection and / or median projection.

Images