Fluorescence microscope system and method

Abstract

To provide a fluorescence microscope system and a method for generating a processed image of a sample with the fluorescence microscope system that can extract relevant information about the sample from a raw image of the sample in a reliable and robust manner.SOLUTION: A fluorescence microscope system (100) includes an optical detection system (106) configured to capture a raw image of a sample (102), and the raw image has a plurality of pixels each having a brightness value. The fluorescence microscope system (100) further includes a processor (116). The processor (116) is configured to identify an invalid pixel in the raw image, allocate a predetermined value to each invalid pixel, identify a brightness value range having the brightness values of a majority of the plurality of pixels excluding the invalid pixel, and generate a processed image of the sample (102) on the basis of the identified brightness value range.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a fluorescence microscope system. The present invention further relates to a method for producing a processed image of a sample by a fluorescence microscope system. [Background technology]

[0002] A fluorescence microscopy image has a number of pixels, each corresponding to a data point of a fluorescence microscopy measurement. In most cases, each pixel has an intensity value corresponding to the detected fluorescence intensity. In other cases, a fluorescence microscopy image is the result of a mathematical reconstruction, for example, by a spectral unmixing algorithm. In these cases, each pixel can also be assigned a confidence value that represents the accuracy of the intensity value of each pixel determined by the algorithm.

[0003] There are various reasons why a pixel in a fluorescence microscopy image can be considered invalid. For example, the pixel may be saturated due to, for example, detected fluorescence intensity exceeding the dynamic range of the detector. Errors may occur during detector readout, resulting in one or more pixels having brightness values ​​that do not correspond to the fluorescence intensity. Furthermore, pixels may be considered invalid if errors occur during the mathematical reconstruction of the fluorescence intensity, resulting in a low confidence value associated with each pixel. Invalid pixels are artifacts in fluorescence microscopy images that must be taken into account when displaying or further processing the image.

[0004] In fluorescence microscopy, it is common to work with weak signals that utilize only a small portion of the detector's dynamic range. To better display these weak signals, a range of brightness values ​​that includes the signal, i.e., the signal that carries information about the sample, is selected and expanded to cover the entire dynamic range of the display unit. However, invalid pixels do not carry any information about the sample, and therefore invalid pixels prevent the identification of the brightness value range. Summary of the Invention [Problem to be solved by the invention]

[0005] It is therefore an object of the present invention to provide a fluorescence microscope system and a method for generating a processed image of a sample by the fluorescence microscope system that is capable of extracting relevant information about the sample from a raw image of the sample in a reliable and robust manner. [Means for solving the problem]

[0006] The above-mentioned object is achieved by what is set forth in the independent claims. Advantageous embodiments are defined by the dependent claims and the following description.

[0007] The proposed fluorescence microscope system includes an optical detection system configured to capture a raw image of a sample. The raw image includes a plurality of pixels, each having an intensity value. The fluorescence microscope system further includes a processor configured to identify invalid pixels in the raw image, assign a predetermined value to each invalid pixel, identify an intensity value range that includes the intensity values ​​of a majority of the plurality of pixels excluding the invalid pixels, and generate a processed image of the sample based on the identified intensity value range.

[0008] The predetermined value may be a numeric luminance value, NaN, or another numeric or non-numeric data type. By assigning a predetermined value to an invalid pixel, the invalid pixel is marked. Because the predetermined value replaces the luminance value of the invalid pixel, no additional memory is required to store information about whether a particular pixel is valid or not. Furthermore, the predetermined value can be easily filtered by common image analysis and manipulation programs. Therefore, marking invalid pixels with a predetermined value also increases interoperability.

[0009] All non-invalid pixels in the raw image are hereinafter referred to as valid pixels. The brightness values ​​of these valid pixels correspond to the fluorescence intensity measured and / or determined by the fluorescence microscope system. Therefore, the valid pixels carry relevant information about the sample observed by the fluorescence microscope system. When the brightness value range is selected, predetermined values ​​are excluded. In other words, by excluding predetermined values, only valid pixels, i.e., pixels that are likely to contain relevant information about the sample, are considered when determining the brightness range. This allows the fluorescence microscope system to extract relevant information about the sample from the raw image in a reliable and robust manner. A processed image is then generated based on the brightness value range, for example, by expanding or stretching the brightness value range across the entire dynamic range.

[0010] In a preferred embodiment, the processor is configured to assign a different color value to each luminance value in the luminance value range and generate the processed image as a false color image based on this assignment. Preferably, all color values ​​have the same hue. In this embodiment, the fluorescence microscope system assists the user in identifying relevant information in the processed image by presenting this via a colored image of the sample. This allows the fluorescence microscope system to enable the user to work more efficiently.

[0011] In another preferred embodiment, the processor is configured to assign a predetermined color value to the predetermined value and generate the processed image as a false color image based on this assignment. Preferably, the predetermined color value is a complementary color to the hue of the range of color values. In this embodiment, invalid pixels are highlighted in the processed image, allowing a user to quickly identify them and work with them more efficiently.

[0012] In another preferred embodiment, the fluorescence microscope system includes a memory element. The memory element includes at least one look-up table. The look-up table correlates brightness values ​​to color values. The processor is configured to generate the processed image as a false color image based on the look-up table. In this embodiment, the relationship between the brightness values ​​of the raw image and the color values ​​of the processed image is stored in the form of a look-up table. Alternatively, the relationship between the brightness values ​​of the raw image and the color values ​​of the processed image can be stored in the form of a functional relationship.

[0013] In another preferred embodiment, the processor is configured to determine whether pixels in the raw image are saturated and identify each saturated pixel as one of the invalid pixels. A pixel in the raw image is saturated when its brightness value is at its maximum value. This means, for example, that the fluorescence intensity corresponding to each pixel received at the detector location exceeds the dynamic range of the detector. High fluorescence intensity may be the result of, for example, the accumulation of a high concentration of protein in the sample. Therefore, the actual value of the fluorescence intensity may be much higher than the maximum value detected. Therefore, saturated pixels do not represent actual data and must be discarded, i.e., identified as invalid.

[0014] In another preferred embodiment, the processor is configured to determine whether the brightness values ​​of pixels in the raw image are the result of calculation errors and / or detection errors, and identify each such pixel as an invalid pixel.Calculation errors may occur, for example, during spectral unmixing.An example of a detection error is a light sheet artifact.Neither calculation errors nor detection errors represent actual data, and therefore must be identified as invalid.

[0015] In another preferred embodiment, the processor is configured to determine a confidence value for each pixel of the raw image and identify each pixel having a confidence value below a predetermined threshold as one of the invalid pixels. In this embodiment, the raw image can be obtained, for example, using a machine learning algorithm that assigns a confidence value to each pixel of the raw image. The confidence value is a measure of the accuracy with which the processor has determined the brightness value of each pixel. A low confidence value indicates low reliability of the data. Therefore, by ignoring pixels with low confidence values, the processed image will be more representative of the actual sample.

[0016] In another preferred embodiment, after the processor assigns a predetermined value to each invalid pixel, the processor is configured to determine a luminance histogram of the raw image, the luminance histogram having a number of pixels for each luminance value. The processor is configured to determine a luminance value range based on the luminance histogram. In particular, the processor is configured to exclude predetermined luminance values ​​from the luminance histogram. The luminance histogram counts the number of pixels in the raw image for each luminance value. In particular, the luminance value range can be determined by the processor by analyzing the luminance histogram for peaks and valleys in a plot of the luminance histogram.

[0017] In another preferred embodiment, the predetermined value is a minimum or maximum luminance value. Neither the minimum nor the maximum luminance value is likely to carry actual information about the sample. Therefore, the minimum and maximum luminance values ​​can be used to mark invalid pixels without losing actual information. This saves memory because the luminance values ​​of invalid pixels are replaced.

[0018] In another preferred embodiment, the fluorescence microscope system includes an illumination system configured to emit excitation light that excites at least one fluorophore located in the sample. The light detection system is configured to generate a raw image based on the fluorescence emitted by the excited fluorophore. In this embodiment, the brightness values ​​of the raw image correspond to the intensity of the fluorescence emitted by the excited fluorophore. The illumination system may include one or more light sources, particularly coherent light sources, configured to emit the excitation light. When the illumination system includes two or more different light sources, each of these light sources may be configured to generate a specific excitation light. Alternatively, the illumination system may include a white light source and an interchangeable filter unit having two or more filters that block all wavelengths of white light except for a single wavelength or wavelength range to generate the different excitation lights.

[0019] In another preferred embodiment, the fluorescence microscope system is configured for fluorescence wide-field microscopy. Additionally or alternatively, the fluorescence microscope system is configured for confocal laser scanning microscopy.

[0020] In another preferred embodiment, the fluorescence microscope system comprises an output unit configured to display the raw images and / or the processed images.

[0021] The present invention further relates to a method for generating a processed image of a sample by a fluorescence microscope system, the method comprising the steps of: capturing a raw image of the sample by an optical detection system of the microscope system, the raw image having a plurality of pixels each having an intensity value, the method further comprising the steps of identifying invalid pixels in the raw image, assigning a predetermined value to each invalid pixel, identifying an intensity value range that includes the intensity values ​​of a majority of the plurality of pixels excluding the invalid pixels, and generating a processed image of the sample based on the identified intensity value range.

[0022] This method has the same advantages as the fluorescence microscope system described above. In particular, this method can be supplemented using the features of the dependent claims directed to the fluorescence microscope system.

[0023] In the following, specific embodiments will be described with reference to the drawings. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a schematic diagram of a fluorescence microscope system according to one embodiment. [Figure 2] 2 is a flowchart of a method for generating a processed image of a sample by the fluorescence microscope system according to FIG. 1; [Figure 3a] FIG. 1 is a schematic diagram of a raw image. [Figure 3b] FIG. 2 is a schematic diagram of a processed image. [Figure 4] This is a brightness histogram of the raw image. DETAILED DESCRIPTION OF THE INVENTION

[0025] FIG. 1 is a schematic diagram of a fluorescence microscope system 100 according to one embodiment.

[0026] The fluorescence microscope system 100 is configured to capture a raw image 300 (see FIG. 3a) of a sample 102 by fluorescence microscopy. The illumination system 104 of the fluorescence microscope system 100 is configured to generate excitation light for exciting fluorophores located within the sample 102. The optical detection system 106 of the fluorescence microscope system 100 is configured to generate a raw image of the sample 102 based on fluorescence emitted by the excited fluorophores. The optical detection system 106 includes an objective lens 108 directed toward the sample 102 and a detection element 110. The objective lens 108 receives the fluorescence emitted by the excited fluorophores and directs the fluorescence toward a detection beam path 112. In this embodiment, a beam splitter 114 is disposed at the intersection of the illumination beam path and the detection beam path 112, which are perpendicular to each other in this embodiment. The beam splitter 114 is configured to direct the excitation light toward the sample 102 via the objective lens 108. Beam splitter 114 is further configured to direct the fluorescent light received by objective lens 108 towards detector element 110 .

[0027] The fluorescence microscope system 100 further includes a processor 116, an input unit, and an output unit. The processor 116 is connected to the illumination system 104 and the light detection system 106 and is configured to control the illumination system 104 and the light detection system 106 to acquire a raw image 300 of the sample 102. The processor 116 is connected to the input unit 118 and the output unit 120 and is configured to receive user input via the input unit 118. In this embodiment, the input unit 118 is illustratively configured to be a computer keyboard. Alternatively, another input unit, such as a computer mouse, joystick, or trackball, can be used. In particular, the fluorescence microscope system 100 is set up to support many different input devices. This allows a user to choose the input device that is most comfortable for them.

[0028] The control unit is further configured to implement a method for generating a processed image of the sample 102 from the raw image 300 and to output the raw image 300 and the processed image 306 (see Figure 3b) to a user via the output unit 120. The method is described below with reference to Figures 2 to 4.

[0029] FIG. 2 is a flowchart of a method for generating a processed image 306 of a sample 102 by the fluorescence microscope system 100 described above.

[0030] The process begins at step S200. In step S202, the processor 116 controls the illumination system 104 to emit excitation light to excite fluorophores located within the sample 102. The processor 116 also controls the light detection system 106 to capture fluorescence emitted by the excited fluorophores and generate a raw image 300 from the captured fluorescence. The raw image 300 includes a plurality of pixels, each having an intensity value. Step S202 can be triggered by user input. In step S204, the processor 116 identifies invalid pixels in the raw image 300. The invalid pixels may be saturated pixels, the result of a calculation error, or the result of a detection error. In either case, the invalid pixels do not represent actual data. In step S206, the processor 116 assigns each invalid pixel a predetermined intensity value. The predetermined value may be a maximum value, a minimum value, or another data type such as NaN, uniquely marking the pixel as an invalid pixel. All other pixels in the raw image 300 are hereinafter referred to as valid pixels. The result of step S206 is a modified raw image 300.

[0031] In optional step S208, the processor 116 determines a luminance histogram 400 (see FIG. 4 ) of the modified raw image 300. The processor 116 can output the luminance histogram 400 to a user via the output unit 120. In step S210, the processor 116 determines a luminance value range 410 (see FIG. 4 ) having luminance values ​​of the majority of the pixels of the modified raw image 300, excluding invalid pixels. When the luminance histogram 400 is determined in step S208, the determination of the luminance value range 410 may be based on the luminance histogram 400. In particular, the determination of the luminance value range 410 may be based on analyzing a plot of the luminance histogram 400, for example, by analyzing the luminance histogram 400 for peaks. The determination of the luminance value range 410 based on the luminance histogram 400 will be described in more detail below with reference to FIG. 4 .

[0032] In step S212, the processor 116 generates the processed image 306 of the sample 102 from the modified raw image 300 based on the luminance value range 410. Specifically, the processor 116 applies a lookup table to the modified raw image 300, which assigns a color value, preferably of a single hue, to each luminance value in the luminance value range 410. This generates a false color image. In this embodiment, the lookup table is selected so that the luminance value range 410 extends to the entire dynamic range of the output unit 120. This means that invalid pixels and unused luminance values, i.e., luminance values ​​that correspond to no or only a few pixels in the modified raw image 300, are discarded. This allows the actual data to be more visible to the user. Optionally, the lookup table can assign predetermined color values ​​to predetermined luminance values, i.e., invalid pixels. Preferably, the hue of the color value used to mark the invalid pixels is the complement of the hue of the color value used to color the valid pixels. This allows the invalid pixels to be clearly marked and visible to the user. In optional step S214, the user can adjust the brightness value range 410. If the user readjusts the brightness value range 410, the processor 116 repeats step S212 using the adjusted brightness value range 410. The process ends in step S216.

[0033] FIG. 3 a is a schematic diagram of a raw image 300 of the sample 102 .

[0034] The raw image 300 has a first region 302 consisting of saturated pixels in the center of the sample 102. The saturated pixels are indicated by solid lines in FIG. 3a. The saturated pixels in the raw image 300 correspond to pixels of the detector elements 110 that detected a fluorescence intensity that exceeded the capabilities of the respective detector pixel. The actual fluorescence intensity may be much higher than the maximum value assigned to the saturated pixel. The high fluorescence intensity may be due, for example, to a large concentration of fluorophores in the respective region of the sample 102. The actual fluorescence intensity is outside the dynamic range of the detector elements 110, so the actual fluorescence intensity cannot be measured. Therefore, the saturated pixels do not represent actual data.

[0035] The raw image 300 also includes a second region 304 of very dark pixels around the center of the sample 102. The dark pixels correspond to a weak fluorescent signal and are indicated by a dotted line in FIG. 3a. In the raw image 300, saturated pixels are brighter than the dark pixels, making them nearly invisible. For example, dark pixels may have an intensity value between 0 and approximately 200, whereas the full range of intensity values ​​is 0 to 65536. To make the structures of the sample 102 corresponding to the dark pixels visible, the intensity of the raw image 300 must be rescaled. To achieve this, the processor 116 performs the method described above with reference to FIG. 2. The resulting processed image 306 is described below with reference to FIG. 3b. In the example described above, the processor 116 identifies the intensity value range as 0 to 200 and stretches this intensity value range 410 to the full range to generate the processed image 306.

[0036] FIG. 3 b is a schematic diagram of the processed image 306 .

[0037] In the processed image 306, the dark pixels are clearly visible. This is represented by the dark pixels being shown as solid lines in Fig. 3b. The predetermined value at which the invalid pixels are marked is outside the brightness value range. Therefore, no saturated pixels are visible in the processed image 306.

[0038] FIG. 4 is a luminance histogram 400 of the raw image 300.

[0039] The abscissa 402 of the luminance histogram 400 represents the luminance value. The ordinate 404 of the luminance histogram 400 represents the number of pixels per luminance value. A single peak 406 on the right side of the luminance histogram 400 represents an invalid pixel that has been assigned the maximum luminance value as a predetermined value. The actual data is represented by a collection of peaks and valleys 408 on the left side of the luminance histogram 400.

[0040] The luminance value range 410 identified in step S208 of the method described above with reference to Figure 2 is indicated by a double-headed arrow in Figure 4. The luminance range can be identified from the luminance histogram 400, for example, as a range having a predetermined percentage of all valid pixels. Alternatively, a maximum luminance value can be identified for which the number of pixels having that luminance value exceeds a certain threshold. In this case, this maximum luminance value can be set as the upper limit of the luminance value range 410.

[0041] The same reference numerals are used throughout the figures to refer to the same, similar or identically functioning elements. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0042] Both individual features of the embodiments and all combinations of features with one another are considered to be disclosed. Furthermore, individual features of the embodiments are considered to be disclosed in combinations of individual features or groups of features in the preceding description and / or in combinations of individual features or groups of features in the claims.

[0043] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus. [Explanation of symbols]

[0044] 100 Fluorescence Microscope System 102 samples 104 Lighting System 106 Optical detection system 108 Objective Lens 110 Detector element 112 Detection beam path 114 Beam Splitter 116 processors 118 Input Units 120 output units 300 statues 302,304 areas 306 statue 400 Luminance Histogram 402 Horizontal axis 404 Ordinate 406 Point 408 Gathering 410 brightness value range

Claims

1. A fluorescence microscope system (100) having a light detection system (106) and a processor (116), the optical detection system (106) is configured to capture a raw image (300) of the sample (102), the raw image (300) having a plurality of pixels, each having a brightness value; The processor (116) Identifying invalid pixels in the raw image (300); assigning a predetermined value to each invalid pixel; identifying a luminance value range (410) having the luminance values ​​of most of the pixels excluding the invalid pixels; generating a processed image (306) of the sample (102) based on the identified luminance value range (410); It is configured as follows: A fluorescence microscope system (100).

2. the processor (116) is configured to assign a different color value to each luminance value in the luminance value range (410) and generate the processed image (306) as a false color image based on the assignments. The fluorescence microscope system (100) of claim 1.

3. the processor (116) is configured to assign predetermined color values ​​to the predetermined values ​​and generate the processed image (306) as a false color image based on the assignments. The fluorescence microscope system (100) of claim 2.

4. the predetermined color value is a complementary color to a hue of a range of color values; The fluorescence microscope system (100) of claim 3.

5. the fluorescence microscope system (100) includes a memory element having at least one lookup table, the lookup table correlating brightness values ​​to color values, and the processor (116) is configured to generate the processed image (306) as a false color image based on the lookup table. The fluorescence microscope system (100) of claim 2.

6. the processor (116) is configured to identify whether pixels of the raw image (300) are saturated or not, and to identify each saturated pixel as one of the invalid pixels; The fluorescence microscope system (100) of claim 1.

7. the processor (116) is configured to determine whether the luminance values ​​of pixels of the raw image (300) are the result of a calculation error and / or a detection error, and to identify each of the pixels as one of the invalid pixels. The fluorescence microscope system (100) of claim 1.

8. the processor (116) is configured to determine a confidence value for each pixel of the raw image (300) and to identify each pixel having a confidence value below a predetermined threshold as one of the invalid pixels; The fluorescence microscope system (100) of claim 1.

9. the processor (116) is configured to determine a luminance histogram (400) of the raw image (300) after the predetermined value has been assigned by the processor (116) to each invalid pixel, the luminance histogram (400) having a number of pixels for each luminance value, and the processor (116) is configured to determine the luminance value range (410) based on the luminance histogram (400). The fluorescence microscope system (100) of claim 1.

10. the predetermined value is a minimum luminance value or a maximum luminance value; The fluorescence microscope system (100) of claim 1.

11. The fluorescence microscope system (100) has an illumination system (104) configured to emit excitation light for exciting at least one fluorophore located in the sample (102), and the light detection system (106) is configured to generate the raw image (300) based on fluorescence emitted by the excited fluorophore, and the brightness values ​​of the raw image (300) correspond to fluorescence intensity. The fluorescence microscope system (100) of claim 1.

12. the fluorescence microscope system (100) is configured for fluorescence wide-field microscopy; The fluorescence microscope system (100) of claim 1.

13. the fluorescence microscope system (100) is configured for confocal laser scanning microscopy; The fluorescence microscope system (100) of claim 1.

14. The fluorescence microscope system (100) comprises an output unit (120) configured to display the raw image (300) and / or the processed image (306). The fluorescence microscope system (100) of claim 1.

15. 1. A method for generating a processed image (306) of a sample (102) with a fluorescence microscope system (100), the method comprising: capturing a raw image (300) of the sample (102) by an optical detection system (106) of the fluorescence microscope system (100), the raw image (300) having a plurality of pixels, each having an intensity value; Identifying invalid pixels in the raw image (300); assigning a predetermined value to each invalid pixel; identifying a luminance value range (410) that includes the luminance values ​​of most of the pixels excluding the invalid pixels; generating a processed image (306) of the sample (102) based on the identified luminance value range (410); A method having the following.

Images