Microscope and method for forming microscopic images with extended depth of field
By acquiring microframes line by line and combining them, using roller shutter shutter and deformable optical lens technology, the problems of low efficiency and image distortion of extended depth of field microscopy in the prior art are solved, and fast and efficient microscopy images are achieved Acquisition and image sharpness improve.
Patent Information
- Application Number
- CN202210238707.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-03-17
- Filing Date
- 2022-03-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-03-10
AI Technical Summary
The prior art is less efficient when acquiring microscopic images with extended depth of field, requiring more time and effort, and the image distortion problem caused by the global shutter cannot be effectively solved.
通过使用数字显微镜,逐线获取多个显微帧,改变焦点位置,利用卷帘快门和可变形光学透镜技术,逐步或周期性地改变焦点位置,组合显微帧以形成具有扩展景深的图像,优选使用CMOS图像传感器以提高读出速度。
It realizes rapid and high sensitivity to acquire microscopic images with extended depth of field, reducing image processing time, avoiding image distortion caused by global shutters, and improving image sharpness and efficiency.
Smart Images

Figure CN115113382B_ABST
Abstract
Description
Technical Field
[0001] The invention firstly relates to a method for producing a microscopic image with an extended depth of field by means of a microscope. The invention also relates to a microscope having an objective lens for optically imaging a sample. Background Art
[0002] In the January 2019 white paper “Things to Consider When Selecting a Microscope Camera” by Takeo Ogama of the Scientific Solutions Division of Olympus Corporation of the Americas, camera factors that affect microscopic imaging quality are discussed. For example, the rolling shutter is explained. Image distortion caused by the rolling shutter is a side effect of the fast readout characteristics of the CMOS sensor. For fast-moving samples, a global reset function after the global shutter is an ideal solution that can help suppress distortion.
[0003] In the October 2017 brochure "Zeiss Axiocam Series - A Technical Guide to Zeiss Microscope Cameras" version 1.0 from Carl Zeiss Microscopy GmbH, microscope cameras are described. With regard to CMOS technology, it is stated that the widely used rolling shutter architecture can lead to geometric distortions of moving objects.
[0004] In the paper "Flexible Depth Photography" by Sujit Kuthirummal, Hajime Nagahara, Changyin Zhou, and Shree K. Nayar, in Proceedings of the 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, pages 60-73, a prototype camera is described that uses a microactuator to translate the detector along the optical axis during image integration. Three applications of flexible DOF are presented. Extended DOF is shown, where a large depth range is captured using a very wide aperture, but with a defocus blur that is almost independent of depth. Applying deconvolution to the captured image results in an image with extended DOF but with a high SNR. Furthermore, the capture of images with discontinuous DOF is shown. For example, objects close and far away can be imaged sharply, while objects in the middle are severely blurred. Furthermore, a camera is described that captures images with tilted DOF without tilting the image detector.
[0005] G. in Optical Communications, Vol. 6, No. 1, September 1972, pp. 38-42. In the paper "Method for increasing depth of focus by two-step image processing", a method for obtaining increased depth of field is described. The method comprises two steps. The first step is to generate a modified incoherent image of a three-dimensional object which is degraded but has the same degradation for all object planes. The second step is to filter this modified image in a coherent image processor to obtain a non-degraded image with increased depth of focus.
[0006] DE 197 33 193 A1 teaches a microscope with adaptive optics. The emission wavefront modulator is located between the objective and the tube lens. The microscope can be used for confocal microscopy, laser-assisted microscopy, conventional microscopy or analytical microscopy.
[0007] US 7,269,344 B2 teaches an optical device having an imaging optical system equipped with a deformable mirror and an electronic zoom function. The light deflection function of the deformable mirror changes according to the change of the object area corresponding to the image to be used. This solution aims at high sharpness of the captured image even when the variable magnification is increased.
[0008] WO 2005 / 119331 A1 teaches a variable focal length lens, which includes a plurality of micro mirrors with two degrees of freedom rotation and one degree of freedom translation. The two degrees of freedom rotation and one degree of freedom translation of the micro mirrors are controlled to change the focal length of the lens and meet the same phase condition of the light. The lens is a diffractive Fresnel lens.
[0009] WO 2007 / 134264 A2 shows a three-dimensional imaging system with a variable focal length micromirror array lens. The micromirror array lens includes a plurality of micromirrors, wherein each micromirror is controlled to change the focal length of the micromirror array lens. The imaging system also includes an imaging unit and an image processing unit, and the image processing unit generates three-dimensional image data using an image captured by the imaging unit and focal length information of the micromirror array lens.
[0010] US 2014 / 0185462 A1 discloses a microscope having a first motorized drive in the z direction for positioning a unit comprising an objective and a camera, and having a second motorized drive in the z direction for positioning a sample stage for receiving a sample. The first motorized drive enables acquisition of images with an extended depth of field.
[0011] US 8,581,996 B2 discloses an imaging device with which a sample of a large area can be acquired and digitized and an image with an extended depth of field can be output. The imaging device comprises a movable sample stage for receiving the sample and a unit for changing the focus position. The imaging device comprises, among other things, a camera and a unit for generating an image with an extended depth of field. The image with the extended depth of field is an all-focus image.
[0012] US 2015 / 0185465 A1 teaches a digital microscope for acquiring and generating images with an extended depth of field. The microscope is configured to perform asynchronous and parallel positioning, image acquisition, and image processing in the z-direction to enable faster acquisition and generation of images with an extended depth of field.
[0013] In US 2015 / 0185464 A1, a solution is described which aims to save time in investigating the field of view. To this end, the magnification factor of the objective lens and the positioning in the z direction are adjusted, among other things.
[0014] The digital microscopes of the VHX2000 and VHX5000 types from the manufacturer Keyence make it possible to acquire microscopic images with an extended depth of field. The VHX5000 digital microscope manufactured by Keyence makes it possible to acquire the topology of a sample with a measured height of approximately 138 μm by means of a stack of 12 images in a period of approximately 9 seconds.
[0015] The SmartZoom5 microscope from the manufacturer Carl Zeiss Microscopy GmbH enables the acquisition of a microscopic image of a sample with a height of 10 mm in about 25 seconds with the aid of an image stack of about 60 images, and then requires about 19 seconds to calculate the microscopic image with extended depth of field. Thus, a total of about 44 seconds are required to provide an image with extended depth of field.
[0016] DE 10 2017 123 511 A1 relates to a method for producing a microscopic image with an extended depth of field by means of a microscope. In one step of the method, a plurality of microscopic frames of a sample are acquired from different focus positions. In another step, the plurality of microscopic frames are processed to form a microscopic image with an extended depth of field. The focus position is changed continuously during the acquisition of at least some of the microscopic frames at a variable speed or at a variable acceleration.
[0017] DE 10 2019 119 310 A1 teaches a method for generating a microscopic image with an extended depth of field using a microscope. The focus position of the microscope can be manually changed by manually changing the distance between at least one objective lens of the microscope and the sample to be photographed. In one step of the method, it is identified whether a user of the microscope manually changes the focus position according to a periodic function. After identifying that the user is changing the focus position according to a periodic function, several microscopic individual images of the sample with different focus positions are recorded. The multiple microscopic individual images are processed into a microscopic image with an extended depth of field. Summary of the invention
[0018] Starting from the prior art, it is an object of the invention to enable microscopic images with an extended depth of field to be produced more quickly and with less effort.
[0019] The above-mentioned objects are achieved by the method according to the invention and the microscope according to the invention.
[0020] The method according to the invention is used to generate a microscopic image with an extended depth of field (EDoF), which is also referred to as an EDoF image. For this purpose, a microscope, in particular a digital microscope, is used. The microscope comprises an objective lens and an image sensor for converting an image reflected directly or indirectly from the objective lens onto the image sensor.
[0021] In one step of the method according to the invention, a plurality of microscopic frames of a sample are acquired with the microscope while changing the focal position of the microscope. Microscopic frames are acquired from different focal positions so that each individual area of the sample is sharply imaged in at least one microscopic frame. The different focal positions are particularly preferably achieved by different actuations of a microsystem having a movable mirror. As a preferred alternative, the different focal positions are achieved by different actuations of a deformable optical lens. As a preferred alternative, the different focal positions are formed by changing the distance between the sample and the microscope objective. The distance between the sample and the microscope objective can also be described as a z-coordinate. Therefore, the focal position can be described by the value of the z-coordinate. The z-coordinate is usually in a vertical direction referred to as the z-direction. The microscopic frames form a stack. The stacked frames differ from each other in the z-coordinates they acquire, so they can also be referred to as z-stacks.
[0022] The image sensor comprises pixels arranged in a matrix. The matrix consists of lines, i.e. the matrix comprises a plurality of lines. These lines are horizontal lines or vertical lines of the image sensor. A microscopic frame of the matrix is acquired line by line. After acquiring one of the horizontal lines or vertical lines of one of the microscopic frames, the next line of the microscopic frame is acquired. Thus, the next acquired line represents a sample at a later point in time than the previously acquired line.
[0023] The focus position is changed during the acquisition of an individual of the microscopic frames. Thus, the next acquired line is acquired at another focus position than the previously acquired line. The lines of a single microscopic frame or a part of the lines of a single microscopic frame are acquired at different focus positions. The next acquired line may contain partial or completely different z-information than the previously acquired line or the lines acquired after it.
[0024] Preferably, the x- and y-coordinates of the sample position are not changed, while the focus position representing the z-coordinate is changed. The x-coordinate is measured in the x-direction and the y-coordinate is measured in the y-direction. The x- and y-directions are perpendicular to the z-direction. Preferably, the x- and y-coordinates of the sample position are changed slower than the focus position. Preferably, the x- and y-coordinates of the sample position are changed slower than reading out the image sensor in the x-direction.
[0025] In the next step, portions of the acquired microscopic frame are selected. These portions should be suitable for forming a target image with an extended depth of field. Thus, individual portions of the acquired lines are identified, wherein these portions sharply image the sample. These portions at least to a minimal extent sharply image the sample. In a preferred embodiment, these portions are individual pixels of the microscopic frame. Thus, individual groups of individual pixels of the acquired lines are identified, wherein these pixels sharply image the sample.
[0026] The described step of selecting parts of the acquired microscopic frames allows for artifact-free image processing in the case where the x- and y-coordinates of the sample position do not change, but the focal position of the microscope is changed. Preferably, a plurality of microscopic frames are combined together, wherein for each pixel showing the same x-coordinate and the same y-coordinate, the dates between microscopic frames can be analyzed in order to search for global maxima and / or local maxima; or the data can be sorted, for example from largest to smallest, in order to identify areas with maximum intensity and lowest intensity or noise level.
[0027] In a next step, the identified portions of the line of the microscopic frame are synthesized to form a microscopic image of the sample, wherein the microscopic image displays an extended depth of field. The microscopic image contains only sharp portions as much as possible. Synthesizing the identified portions of the line preferably depends on the experimental requirements, including synthesis speed, predefined noise suppression, a predefined degree of regional integrity of the target microscopic image with an extended depth of field, and / or other image-related analyses. If the portions of the line are pixels of an acquired microscopic frame, the identified pixels are combined to form a microscopic image of the sample, wherein the microscopic image displays an extended depth of field.
[0028] A particular advantage of the method according to the invention is that microscopic frames can be acquired very quickly with high sensitivity, since they are acquired line by line. The acquisition preferably includes the acquisition of other image sensor architecture specific functions for signal processing, such as dynamic range adjustment and offset application. In contrast, using a global shutter to acquire one entire microscopic frame at one point in time helps to ensure that in the case of a static actuator the information comes from the same focus position, or that in the case of a dynamic z-actuator the information in the z-direction will be blurred to an extent limited by the movement speed in the z-direction and the frame acquisition speed. Furthermore, global shutter image sensors tend to be slower, more expensive, and require more attention to handle in system integration.
[0029] In a preferred embodiment, multiple microscopic frames are acquired by using a rolling shutter. A rolling shutter is an image capture method in which frames are captured by scanning across the field of view vertically or horizontally. The rolling shutter can be mechanical or electronic. Preferably, the rolling shutter is electronic. In particular, the rolling shutter is implemented as a routine for reading out an image sensor. The image sensor is preferably a CMOS image sensor. The readout speed of a CMOS image sensor is faster than that of a global shutter. Multiple microscopic frames are acquired by using an image sensor with a rolling shutter.
[0030] In a preferred embodiment, the focal position is varied by changing the distance between the sample and the objective of the microscope or by changing the focal position along the spatial extension of the sample. For example, the focal position is preferably varied by controlling a microsystem with a movable mirror or by deforming a deformable optical lens of the objective.
[0031] Preferably, the focus position is changed continuously during acquisition of an individual of the microscopic frames.
[0032] In a further preferred embodiment, the focus position is changed stepwise or according to a predefined function. The predefined function may be a random function. If the focus position is changed stepwise, the step duration is preferably 10 ms or less. In a further preferred embodiment, the focus position is changed periodically. Preferably, the focus position is changed periodically between a minimum value of the focus position and a maximum value of the focus position. Preferably, the focus position is changed according to a sine wave. Preferably, the frequency of the periodic change is at least 100 Hz. More preferably, the frequency of the periodic change is at least 1 kHz. The frequency may also exceed 10 kHz.
[0033] In a preferred embodiment, a plurality of microscopic frames are acquired during at least ten cycles of the periodic change of the focus position. Preferably, the focus position is changed at a high frequency as described above. Due to this rapid change, many change cycles are experienced until a plurality of microscopic frames are acquired. More preferably, a plurality of microscopic frames are acquired during at least 100 cycles of the periodic change of the focus position. Preferably, each microscopic frame is acquired during a time period of at least 10 cycles of the periodic change of the focus position.
[0034] In a preferred embodiment, the plurality of microscopic frames comprises at least 20 microscopic frames. More preferably, the plurality of microscopic frames comprises at least 100 microscopic frames. This number defines a z-stack.
[0035] In a preferred embodiment of the method, the acquisition of multiple microscopic frame images continues until a stop condition is met, in particular until a predefined quality factor is achieved. These conditions preferably include the degree of image area integrity of the microscopic image with an extended depth of field, the absence of predefined artifacts, and / or the presence of predefined features. Predefined artifacts preferably include noise, lost data, and / or dark images. Predefined features preferably include sharp edges and / or flat surfaces. The quality factor is preferably related to image acquisition conditions, which preferably include exposure time, noise level, data transfer rate, and / or illumination intensity. The quality factor is preferably related to sample-specific parameters, which preferably include color, monochrome, prior information, and / or sample classification data. The quality factor is preferably related to image information content, which preferably includes the degree of z-information estimation over the area, the degree of artifacts, the degree of lost data, and / or the degree of underexposed or overexposed areas.
[0036] A preferred embodiment of the method comprises a further step, wherein a value of the focus position is recorded for each portion of a line of each microscopic frame. In order to form a microscopic image of the sample with an extended depth of field, the composition of the identified portions of the line is based on the recorded values of the focus position assigned to the identified portions of the line. The value of the focus position is preferably formed by the z position. The value of the focus position is used to process the identified portions of the line of the microscopic frame to form the microscopic image with an extended depth of field, in particular to correct the magnification of each portion, thereby producing a microscopic image with an extended depth of field based on a uniform magnification. If the identified portions of the line are pixels, a value of the focus position is preferably recorded for each pixel of the line of each microscopic frame.
[0037] A preferred embodiment of the method comprises a further step, in which height values of the microscopic image with extended depth of field are determined based on the recorded values of the focus position assigned to the identified part of the line. The height values are assigned to individual pixels or groups of individual pixels of the microscopic image with extended depth of field. The height values are sample-dependent. These values describe the sample height at a certain point of the sample photographed at the assigned pixel or group of assigned pixels. The height values describe the size of the sample in the z-direction. The height values define a height map of the sample. The height map is matched to the microscopic image with extended depth of field.
[0038] In a preferred embodiment of the method, the height value in the z direction corresponds to a pixel having a height value in the z direction defined by the corresponding position of the actuator for changing the focus position. The relationship is preferably based on time. For this purpose, the timing signal from the actuator in the form of a timestamp or a trigger signal is preferably related to the timing of reading out the image sensor in the form of a timestamp and / or in the form of a trigger signal related to a pixel readout or a pixel line readout or a frame readout defined for a complete frame or a predefined region of interest.
[0039] A preferred embodiment of the method comprises a further step in which a microscopic three-dimensional image of the sample is determined based on the microscopic image with extended depth of field and based on the height values. Since the height values describe the size of the sample in the z direction, these values can be used to generate a three-dimensional image.
[0040] In a preferred embodiment, the focus position is controlled to change the focus position. This is usually done by an electronic controller that controls an actuator to change the focus position. The actuator preferably comprises a microsystem for mechanically moving the micromirror. Therefore, the focus position is controlled by controlling the microsystem for mechanically moving the micromirror. The focus position is controlled based on a command value for the focus position. The command value is generated by the desired time course for changing the focus position. The command value depends on time. The command value is usually calculated by a controller. The command value is preferably used to record the value of the focus position for each portion of the line of each microscopic frame. This can be done in this way because the actual time course for changing the focus position is close to the desired time course for changing the focus position if no malfunction occurs. Since each pixel of each microscopic frame is assigned to a time point, the pixel is also assigned to the command value of the focus position. Therefore, each portion of each microscopic frame is assigned to a time point and, therefore, the portion is also assigned to the command value of the focus position. Each microscopic frame contains portions of a line that originate from positions having a defined x-coordinate and a defined y-coordinate but from different z-coordinates.
[0041] In order to identify the individual parts of the acquired lines that image the sample sharply, the sharpness of the acquired microscopic frame must be analyzed. Preferably, the parts of the acquired lines are identified by applying a sharpness function to the signal of the image sensor over time. For example, the parts of the acquired lines are identified by searching for the maximum (maxima) of the signal of the image sensor over time. The signal represents the individual parts of the line showing the same x-coordinate and the same y-coordinate but different z-coordinates. The maximum values are searched in the various segments of the signal, which are individually related to the local position of the corresponding part of the line, in particular to the xy position within the frame as the position of the pixel to be identified. These maxima are preferably global maxima in the corresponding segments of the time signal. The time signal is generated by acquiring the frame line by line, in particular by using a rolling shutter. The time signal of the image sensor represents the content of the microscopic frame. The maximum values represent those parts of the line of the microscopic frame at the corresponding local position that are the sharpest. The parts of the line are identified, that is, selected so as to be combined to form a microscopic image with an extended depth of field.
[0042] In a further preferred embodiment, for each local position of a respective portion of a line, in particular for an xy position within a frame as the position of a pixel to be identified, more than one portion of the acquired line is identified. Thus, a plurality of one maxima are searched for within each segment of the signal associated with a local position of a respective portion of a line, in particular for an xy position within a frame as the position of a pixel to be identified. These maxima preferably include global maxima and local maxima in the respective segment of the time signal. For each local position, more than one identified portion of the acquired line is preferably combined into more than one microscopic image of the sample with an extended depth of field. This results in a plurality of microscopic images with an extended depth of field.
[0043] In a preferred embodiment, the plurality of microscopic frames are acquired at a frame rate of at least 10 frames per second. More preferably, the plurality of microscopic frames are acquired at a frame rate of at least 50 frames per second. More preferably, the plurality of microscopic frames are acquired at a frame rate of at least 100 frames per second.
[0044] The above-mentioned step for forming a microscopic image with an extended depth of field is preferably repeated periodically to form a sequence of images with an extended depth of field. Preferably, the images with an extended depth of field are formed at a rate of at least 10 images per second. Preferably, the images with an extended depth of field are formed at a rate of at least 50 images per second. In the above-mentioned embodiment of determining a microscopic three-dimensional image of the sample, the step of determining the microscopic three-dimensional image is preferably repeated periodically to form a sequence of microscopic three-dimensional images. Preferably, the microscopic three-dimensional image is formed at a rate of at least 10 images per second.
[0045] The microscope according to the invention is digital and comprises an objective lens for magnifying optical imaging of a sample on an image plane. The objective lens comprises an optical assembly for magnifying optical imaging of a sample on an image plane. The optical assembly is specifically instantiated by an optical lens and optionally also by one or more mirrors, shutters and filters.
[0046] The microscope preferably comprises an image sensor for converting an image reflected directly or indirectly by the objective lens onto the image sensor into an electrical signal.
[0047] The microscope comprises at least one actuator for changing the focal position of the microscope. In a simple embodiment, the actuator is preferably designed to change the distance between the sample and the objective lens. For this purpose, the actuator preferably comprises a motor for moving a sample stage carrying the sample or for moving the objective lens. The actuator is preferably designed to change the focal length of the objective lens. The actuator is preferably used to actuate an active optical element that can be used to adjust the focus. Particularly preferably, the actuator of the active optical element is a microsystem for mechanically moving a micromirror and / or a microlens. The micromirror preferably forms a lens. By changing the position of the micromirror, the focal length of the lens can be changed very quickly. This rapid change of focal length enables the focus position to be adjusted very quickly. As a preferred alternative, the actuator is designed to deform a deformable optical lens.
[0048] The microscope further comprises a control and image processing unit for controlling the actuator and processing the microscopic frame. The control and image processing unit is configured to perform the method according to the invention. The control and image processing unit is preferably configured to perform one of the preferred embodiments of the method according to the invention. Furthermore, the microscope preferably also has the features specified in conjunction with the method according to the invention and its preferred embodiments.
[0049] The microsystem for mechanically moving the micromirror is preferably a mirror array lens system. Such a mirror array lens system is provided under the trademark MALS. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Other advantages, details and improvements of the present invention will become apparent from the following description of preferred embodiments of the present invention with reference to the accompanying drawings. There is shown:
[0051] Figure 1 : A graph showing signal strength over time according to a preferred embodiment of the method of the present invention;
[0052] Figure 2 : represents a signal strength diagram at coordinate z according to the method of the prior art;
[0053] Figure 3 : A diagram of acquiring a microscopic frame according to a preferred embodiment of the method of the present invention; and
[0054] Figure 4 : Figure of a microscopic frame acquired according to the prior art. DETAILED DESCRIPTION
[0055] Figure 1A diagram representing a signal intensity A over time t representing a preferred embodiment of the method according to the invention is shown. The signal intensity A is measured at the output of an image sensor (not shown) of a microscope (not shown). The signal intensity A is related to the local position of a pixel of the image sensor. The signal intensity A depends on the time t, because a plurality of microscopic frames are acquired over time t. The focus position at coordinate z of the microscope (not shown) is changed, for example, periodically or randomly. Thus, time t and coordinate z are associated. The image sensor (not shown) operates according to the method of a rolling shutter. The image sensor (not shown) is preferably a CMOS sensor.
[0056] The signal intensity A shows a global maximum at z=z_1. At z_1, the pixels of the corresponding frame acquired by the image sensor (not shown) are sharp. Due to the rolling shutter, there are also local maxima, for example, max_loc. At these local maxima max_loc, the pixels of the corresponding frame acquired by the image sensor (not shown) are also sharp. The sharp pixels at the global maximum constitute a microscopic image with an extended depth of field. Optionally, the sharp pixels at the local maximum max_loc constitute a further microscopic image with an extended depth of field.
[0057] Figure 2 A diagram representing the signal intensity A at coordinate z of a method according to the prior art is shown. The signal intensity A is measured at the output of an image sensor (not shown) of a microscope (not shown). The signal intensity A is related to the local position of the pixel of the image sensor. The signal intensity A depends on the time t, because multiple microscopic frames are acquired over time t. The focus position at coordinate z of the microscope (not shown) changes periodically. The image sensor (not shown) operates according to the method of the global shutter. The image sensor (not shown) is, for example, a CCD sensor. Due to the global shutter, the signal intensity A only shows a maximum value at z=z_1. At z_1, the pixels of the corresponding frame acquired by the image sensor (not shown) are sharp.
[0058] Figure 3 A diagram showing the acquisition of microscopic frames according to a preferred embodiment of the method of the present invention is shown. The four microscopic frames are numbered N, N+n, N+m and N+p. Four microscopic frames N, N+n, N+m, N+p are acquired at t_0, t_n, t_m and t_p over time t. As described above with reference to Figure 1 As explained, the focus position at coordinate z is varied randomly, for example. Since the image sensor (not shown) operates according to the rolling shutter method, the lines and parts of the lines of each individual microscopic frame N, N+n, N+m, N+p are obtained at different z_0, z_1, z_2 and z_3 values of the focus position at coordinate z, respectively.
[0059] Figure 4A diagram of obtaining microscopic frames according to the prior art is shown. The four microscopic frames are numbered N, N+1, N+2 and N+3. Four microscopic frames N, N+1, N+2, N+3 are obtained at t_0, t_1, t_2 and t_3 over time t. As shown above with reference to Figure 2 As explained, the image sensor (not shown) operates according to the method of a global shutter. Due to the global shutter, each individual microscopic frame N, N+1, N+2, N+3 is acquired at a single value of the focus position at coordinate z, namely at z_0, z_1, z_2 and z_3, respectively.
[0060] In an exemplary embodiment of the method according to the invention, a stack of 20 to 60 microscopic frames of the sample is acquired while periodically changing the focus position at a frequency of about 100 Hz. Microscopic frames are acquired at a rate of 60 frames per second. The pixels of the acquired lines of the frames that sharply image the sample constitute a microscopic image with an extended depth of field.
[0061] Reference Symbols List
[0062] A Signal Strength
[0063] t time
[0064] z Focus position
[0065] max Maximum value
[0066] The focal position at the maximum value of z_1
[0067] max_loc local maximum
Claims
1. A method for forming a microscopic image with an extended depth of field by means of a microscope comprising an image sensor, wherein: The image sensor comprises pixels arranged in a matrix formed by lines, wherein the method comprises the following steps: - acquiring a plurality of microscopic frames of the sample while varying the focus position (z), wherein the microscopic frames are acquired line by line, and wherein the focus position (z) is varied during the acquisition of an individual of the microscopic frames; - identifying individual portions of the acquired lines that sharply image the sample; - said identified portions constituting said line so as to form a microscopic image of said sample with an extended depth of field; and - recording the value of said focus position for each said portion of said line of each said microscopic frame; Therein, in order to form a microscopic image of the sample with an extended depth of field, the composition of the identified portion of the line is based on the recorded values of the focus position (z) assigned to the identified portion of the line.
2. The method according to claim 1, characterized in that The plurality of microscopy frames are acquired by using a rolling shutter.
3. The method according to claim 1 or 2, characterized in that: The focus position (z) is changed by changing the distance between the sample and the objective of the microscope.
4. The method according to claim 1 or 2, characterized in that: The focus position (z) is periodically varied, wherein the frequency of the periodic variation is at least 100 Hz.
5. The method according to claim 4, characterized in that The plurality of microscopy frames are acquired during at least ten periods of the periodic change of the focus position.
6. The method according to claim 1 or 2, characterized in that: The focus position (z) is changed stepwise.
7. The method according to claim 1 or 2, characterized in that: The plurality of microscopic frames includes at least 20 microscopic frames.
8. The method according to claim 1, characterized in that The following steps are also included: - determining a height value of the microscopic image with extended depth of field based on the recorded values of the focus position (z) assigned to the identified portion of the line.
9. The method according to claim 8, characterized in that The following steps are also included: - determining a microscopic three-dimensional image of the sample based on the microscopic image with extended depth of field and based on the height values.
10. The method according to claim 1, characterized in that The focus position (z) is controlled so as to be changed based on a command value of the focus position, wherein the command value is used to record the value of the focus position (z) for each portion of the line of each of the microscopic frames.
11. The method according to claim 1 or 2, characterized in that: The portion of the line of the microscopic frame that is acquired is identified by searching for a maximum (max) of the signal of the image sensor over time.
12. The method according to claim 1 or 2, characterized in that: The plurality of microscopic frames are acquired at a frame rate of at least 10 frames per second.
13. The method according to claim 1 or 2, characterized in that: The steps are repeated periodically to form a sequence of said images having an extended depth of field, wherein said images having an extended depth of field are formed at a rate of at least ten images per second.
14. A microscope comprising: - an objective lens for optically imaging the sample; - at least one actuator for changing the focal position (z) of the microscope; as well as - A control and image processing unit configured to perform the method according to any of claims 1 to 13.
15. The microscope according to claim 14, characterized in that The at least one actuator further comprises a microsystem for mechanically moving the micromirror and / or the microlens.
Citation Information
Patent Citations
Microscope and method for producing a microscopic image with extended depth of field
DE102017123511A1
Microscope and method for producing a microscopic image with extended depth of field
DE102019119310A1
microscope with adaptive optics
DE19733193A1
Troubleshooting method and apparatus
US20140185462A1
Magnifying Observation Apparatus, Magnified Image Observing Method And Computer-Readable Recording Medium
US20150185464A1
