Microscope system and corresponding control system, method and computer program

Abstract

To provide an improved concept for determining a position of a movable sample stage.SOLUTION: Examples relate to a microscope system and to a corresponding control system, method and computer program. The microscope system comprises a microscope configured to generate images of a sample being arranged on a sample stage. The microscope system comprises a sample stage configured to carry the sample. The sample stage comprises a visual marker. The method comprises an optical imaging sensor configured to provide image data of an image showing the visual marker of the sample stage. The method comprises a control system configured to determine a position of the sample stage based on the image data of the image showing the visual marker.SELECTED DRAWING: Figure 1a

Description

Technical Field

[0001] The examples relate to a microscope system and corresponding control system, method and computer program, and more specifically, but not exclusively, to a concept for determining the position of a movable sample stage of a microscope system.

Background Art

[0002] Microscopes for industrial use often have a movable sample stage that is movable along two axes (hereinafter referred to as the X-axis and the Y-axis). In order to perform imaging over various positions of the sample stage, accurate X-Y positioning of the stage may be required for object alignment and image stitching. In many cases, optical or magnetic linear encoders are used to determine the X-Y position of the stage. Image stitching can use both the measured X-Y position and (sample) image landmarks to perform image alignment and collation.

[0003] However, linear encoders may have a different coefficient of thermal expansion than the sample stage, which causes inaccurate position readings. Further, the stage may have to be stopped in order to capture an image at each desired X-Y position. If the X-Y position is inaccurate and the object has weak visual contrast or a repeating pattern, image stitching can be difficult or inaccurate.

[0004] There is a desire to provide an improved concept for determining the position of a movable sample stage.

Summary of the Invention

Means for Solving the Problems

[0005] The above problems are solved by the subject matter of the independent claims.

[0006] Various examples of this disclosure are based on the finding that, instead of using a linear encoder to determine the position of a movable sample stage, a visual approach using a visual marker displayed on the top or bottom of the sample stage may be chosen. Since the visual marker is part of the sample stage, the coefficient of thermal expansion may be the same as, or at least similar to, that of the sample stage and the visual marker. An optical imaging sensor is used to generate image data of an image showing the visual marker, and this image data is analyzed to determine the position of the sample stage.

[0007] Various examples of this disclosure relate to a microscope system. The microscope system includes a microscope configured to produce an image of a sample placed on a sample stage. The microscope system includes a sample stage configured to support a sample. The sample stage includes a visual marker. The microscope system includes an optical imaging sensor configured to provide image data of an image showing the visual marker on the sample stage. The microscope system includes a control system configured to determine the position of the sample stage based on the image data of an image showing the visual marker. The visual marker is part of the sample stage and therefore also follows the thermal expansion coefficient of the sample stage, which would result in higher accuracy.

[0008] In various examples, the sample stage is configured to move along two lateral dimensions. The control system can be configured to determine the position of the sample stage with respect to these two lateral dimensions. In other words, the XY position of the sample stage can be determined.

[0009] For example, the sample stage may be a motor-driven sample stage. A motor-driven sample stage can be used for the automatic recording of large samples across multiple positions on the sample stage.

[0010] In some cases, the imaging throughput of a microscope can be improved by continuously moving the sample stage. For example, instead of stopping the sample stage for each image being captured, the sample stage can be moved continuously, and the position of the sample stage can be determined while it is moving using the proposed concept. For example, a motor-driven sample stage may be configured to move continuously while the microscope is generating multiple images at multiple time points. An optical imaging sensor may be configured to generate image data of images showing visual markers while the motor-driven sample stage is moving. A control system may be configured to determine the position of the motor-driven sample stage at multiple time points based on the image data of images showing visual markers that are being generated while the motor-driven sample stage is moving. This can improve the throughput of the microscope system.

[0011] Strobe light may be used to avoid or reduce motion artifacts when the sample stage moves continuously. The microscope system may include at least one illumination element to illuminate the visual marker and / or the sample with a light strobe. The control system may be configured to trigger the illumination element to emit a light strobe at multiple points in time. The light strobe may be brighter than the ambient light, and therefore, the majority of the light recorded by the microscope's optical imaging sensor and / or by the optical imaging sensor used to determine the position of the sample stage is caused by the light strobe.

[0012] For example, the control system may be configured to trigger an illumination element to emit a light strobe so that each pixel of the image generated by the microscope and / or each pixel of the sample in the image data of the image showing the visual markers is exposed to the reflection of the same light strobe. For example, the light strobe may be timed so that each optical imaging sensor is read row by row or block by block so that all rows / blocks of the optical imaging sensors used to generate the image frame are exposed to light from the same strobe.

[0013] In some examples, the visual marker is positioned at the bottom of the sample stage, and the optical imaging sensor is positioned below the sample stage. This arrangement can prevent the visual marker from being obscured by large samples. Furthermore, it allows for handling the microscope without obstructing the optical imaging sensor, for example, when the microscope is tilted.

[0014] Alternatively, the visual marker can be positioned above the sample stage, and the optical imaging sensor can be positioned above the sample stage. In this case, the same illumination element for light strobe emission can be used for both the sample and the visual marker.

[0015] In some examples, the microscope system includes two or more optical imaging sensors that provide image data of two or more images showing at least one of two or more visual markers on the sample stage. The control system may be configured to determine the position of the sample stage based on the image data of two or more images showing at least one of two or more visual markers. By using multiple cameras and visual markers, it becomes possible to use smaller markers and shorter distances between the optical imaging sensors and the visual markers, while lateral movement may still remain large.

[0016] For example, the lateral spread of the visual marker may be at least the same size as the maximum lateral movement of the sample stage. In this case, the size of the visual marker may allow for the determination of the sample stage's position.

[0017] In some examples, the visual marker has a two-dimensional visual pattern. This two-dimensional pattern may be used to determine the position of the sample stage in two lateral dimensions.

[0018] Alternatively or additionally, the sample stage may include a first visual marker aligned in a first lateral movement direction of the sample stage and a second visual marker aligned in a second lateral movement direction of the sample stage. Each of the first and second markers may have a one-dimensional visual pattern. In this case, the first visual marker may be used to determine the position of the sample stage along the first lateral movement direction, and the second visual marker may be used to determine the position of the sample stage along the second lateral movement direction.

[0019] In various examples, the control system may be configured to determine the position of the sample stage relative to the microscope. In this way, the position of the sample stage, and therefore the position of the recorded sample, can be recorded using the image captured by the microscope. For example, the control system may be configured to annotate the image captured by the microscope with the position of the sample stage at the time the image was generated.

[0020] Various examples of this disclosure relate to corresponding control systems for microscope systems. A microscope system includes a microscope configured to produce an image of a sample placed on a sample stage, and a sample stage configured to support the sample. The sample stage includes a visual marker. A control system includes one or more processors and one or more storage devices. The system is configured to acquire image data of an image showing the visual marker on the sample stage from the optical imaging sensor of the microscope system. The system is configured to determine the position of the sample stage based on the image data of the image showing the visual marker.

[0021] Various examples of this disclosure relate to corresponding methods for microscope systems. A microscope system includes a microscope configured to produce an image of a sample placed on a sample stage, and a sample stage configured to support the sample. The sample stage includes a visual marker. The method includes acquiring image data of an image showing the visual marker on the sample stage. The method includes determining the position of the sample stage based on the image data of the image showing the visual marker.

[0022] Various examples of the present disclosure relate to corresponding computer programs having program code for performing the above when the computer program is executed on a processor.

[0023] Some examples of apparatus and / or methods are described below, for illustrative purposes only, with reference to the attached drawings. [Brief explanation of the drawing]

[0024] [Figure 1a] This is a schematic diagram of an example microscope system. [Figure 1b] This is a block diagram of an example control system for a microscope system. [Figure 1c]It is a schematic diagram of an example of a microscope system including a control system. [Figure 2] It is a figure showing an example of a timing diagram of stroboscopic light relative to the readout of an optical imaging sensor. [Figure 3] It is a figure showing a flowchart of an example of a method for a microscope system. [Figure 4] It is a schematic diagram of a system including a microscope and a computer system.

Embodiments for Carrying Out the Invention

[0025] Next, various examples will be described in more detail with reference to the accompanying drawings in which several examples are shown. For clarity, in the figures, the thickness of lines, layers, and / or regions may be exaggerated.

[0026] FIG. 1a shows a schematic diagram of an example of a microscope system 100. The microscope system 100 includes a microscope 110 configured to generate an image of a sample 120 disposed on a sample stage. Generally, a microscope is an optical instrument suitable for inspecting an object that is too small to be inspected visually (alone) by a human. For example, a microscope can provide an optical magnification of a sample, such as sample 120. In recent microscopes, the optical magnification is often provided for a camera or imaging sensor, such as the optical imaging sensor of microscope 110. Microscope 110 may further include one or more optical magnifying components, such as an objective lens (i.e., a lens), used to magnify the field of view on the sample.

[0027] There are various types of microscopes. When a microscope is used in the medical or biological field, the sample observed through the microscope may be, for example, a sample of organic tissue placed in a petri dish or present in a part of a patient's body. For example, microscope system 100 may be a microscope system for use in a laboratory, and may be a microscope that can be used, for example, to examine a sample of organic tissue in a petri dish. Alternatively, microscope system 100 may be a microscope system used in materials science and / or quality control. In particular, microscope system 100 may be a microscope system for generating extremely detailed images of large samples by generating multiple images and stitching them together. Although embodiments have been described in relation to microscopes, they may be applied in a more universal manner to any optical device.

[0028] The microscope 110 of the microscope system 100 is used with several additional optional or required components. Therefore, the term “microscope system” is used to cover parts of the system used in conjunction with the microscope, such as the sample stage, additional optical imaging sensors, and control systems, as will be shown below, although these are not parts of the actual microscope (which include optical components and are therefore also referred to as the “optical carrier”).

[0029] The microscope system further includes a sample stage (e.g., an XY stage) 130 configured to support a sample. The sample stage includes a visual marker 140 (located in a patterned area of ​​the sample stage). The microscope system further includes an optical imaging sensor (also referred to as an XY camera) 150 configured to provide image data of an image showing the visual marker on the sample stage.

[0030] The microscope system further includes a control system configured to determine the position of the sample stage based on image data of images showing visual markers. Figure 1b shows a block diagram of an example control system 160 for the microscope system, for example, the microscope system in Figure 1a. As shown in Figure 1b, for example, the control system 160 includes one or more processors 164 and one or more storage devices 166. Optionally, the control system includes one or more interfaces 162. One or more processors are coupled with one or more storage devices and the optional interfaces. Generally, the functions of the control system are provided by one or more processors in association with, for example, an optional interface (for exchanging information) and / or one or more storage devices (for storing data). The control system 160 is configured to acquire image data of images showing visual markers on the sample stage from the optical imaging sensor 150 of the microscope system (for example, via interface 162). The control system is configured to determine the position of the sample stage based on image data of images showing visual markers.

[0031] Figure 1c shows a schematic diagram of an example of a microscope system 100 including a control system 160. The microscope system 100 in Figure 1c shows further optional components such as at least one illumination element 170.

[0032] Various examples of this disclosure provide concepts for detecting the position of a microscope sample stage to support image stitching of images produced by the microscope. The proposed microscope system is equipped with a sample stage 130, the position of which is determined by a control system. Thus, the sample stage can move relative to the microscope 110 (and is therefore movable). Furthermore, the sample stage can move relative to an optical imaging sensor 150. In other words, the optical imaging sensor may remain in a fixed position while the sample stage moves relative to the optical imaging sensor. In practice, if the sample stage is moved, the positions of the microscope 110 and the optical imaging sensor 150 may shift relative to the sample stage 130. In some examples, the sample stage can be moved manually. Alternatively, the sample stage may be a motor-driven sample stage. In other words, the sample stage may include a motor for moving the sample stage. For example, the sample stage may be configured to move relative to the microscope according to a predetermined pattern, for example, according to a Z-shaped pattern or a meandering pattern.

[0033] In various examples, the sample stage may be moved along one or two transverse dimensions relative to the microscope 110. In particular, in some examples, the sample stage may be configured to move along two transverse dimensions (one transverse dimension being perpendicular to the other). Thus, since the sample stage can be moved along the X transverse dimension and the Y transverse dimension, the sample stage may be referred to as the XY stage. In some examples, the sample stage may also be movable along a third dimension (i.e., the vertical dimension or Z dimension). In this context, the transverse dimension (i.e., horizontal dimension) is defined perpendicular to the direction of gravity, and the vertical dimension is defined parallel to gravity.

[0034] The sample stage includes a visual marker 140 used to determine the position of the sample stage. For example, the visual marker may be printed on the sample stage, or it may be etched into the sample stage using, for example, a laser-based etching method. Alternatively, the visual marker may be specific to the material of the sample stage, for example, resulting from the mold or tools used to manufacture the sample stage. Alternatively, the visual marker may be attached to the sample stage using an adhesive, or it may be attached to the sample stage using another mechanical means for fixing the visual marker to the sample stage, such as screws or rivets. In various examples, the visual marker may be designed so that specular reflection is reduced or minimized.

[0035] In some examples, as shown in Figures 1a and 1c, the visual marker 140 is located on the bottom side of the sample stage (e.g., printed on the bottom side, etched within the bottom side, or mounted on the bottom side), and the optical imaging sensor is located below (vertically) the sample stage. Alternatively, the visual marker may be located on the top side of the sample stage (e.g., printed on the top side, etched within the top side, or mounted on the top side), and the optical imaging sensor may be located above the sample stage (not shown). In this context, the term “top side” of the sample stage may refer to the side of the sample stage facing away from the Earth’s center of gravity, while the “bottom side” of the sample stage may refer to the side of the sample stage facing the Earth’s center of gravity. Similarly, an optical imaging sensor located above the sample stage means that the optical imaging sensor is located further away from the Earth’s center of gravity than the sample stage, or at least further away than the surface of the sample stage containing the visual marker. Therefore, an optical imaging sensor positioned below the sample stage means that the optical imaging sensor is positioned closer to the Earth's center of gravity than the sample stage, or at least closer than the surface of the sample stage containing the visual markers. These definitions assume that the microscope system is oriented to allow microscope operation with the sample positioned on the sample stage.

[0036] The position of the sample stage is determined by generating and analyzing image data of the visual marker on the sample stage. As the sample stage moves, the visual marker moves with it. Correspondingly, and because the sample stage moves relative to the optical imaging sensor, the portion of the visual marker that is visible in the image showing the visual marker also changes as the sample stage moves. When the sample stage moves along one or two lateral dimensions, the visual marker also moves, and this movement of the visual marker is indicated in the image showing the visual marker.

[0037] This affects the spread of the visual marker to determine the position of the sample stage over its entire range of movement, and a portion of the visual marker may be shown in the image data generated by the optical imaging sensor at any (lateral) position of the sample stage. As a result, the lateral spread of the visual marker (e.g., the maximum lateral distance along one or two lateral dimensions between the two outermost portions of the visual marker) may be at least the same magnitude as the maximum lateral movement of the sample stage. In other words, the visual marker may be large enough so that a portion of the visual marker can be shown in the image data generated by the optical imaging sensor at any lateral position of the sample stage. In some examples, this may be achieved by using a single connected visual marker. Alternatively, the sample stage may include two or more visual markers, which may be recorded by two or more optical imaging sensors. In other words, the microscope system may include two or more optical imaging sensors to provide image data of two or more images showing at least one of the two or more visual markers of the sample stage. Alternatively, a single optical imaging sensor may be used to provide image data of an image showing two or more visual markers on a sample stage, or two or more optical imaging sensors may be used to provide image data of an image showing a single connected visual marker.

[0038] In general, visual markers may use any visual pattern that provides sufficient contrast for subsequent analysis. For example, visual markers may include (pseudo) random patterns, such as random patterns based on dots, rectangles, triangles, polygons, etc. Alternatively, visual markers may be the inherent texture of the material surface. In some examples, two-dimensional visual patterns may be used so that movement along two lateral dimensions can be reliably tracked. In other words, visual markers may have two-dimensional visual patterns. Alternatively, two visual markers may be used, each having a one-dimensional visual pattern such as a barcode. In other words, the sample stage may include a first visual marker aligned in a first lateral movement direction of the sample stage and a second visual marker aligned in a second lateral movement direction of the sample stage, each of the first and second markers having a one-dimensional (or two-dimensional) visual pattern. Specular reflections due to patterns can be reduced or minimized so that the movement of the sample stage, and consequently the visual marker, can be visually tracked.

[0039] In the proposed concept, the optical imaging sensor 150 is used by a control system to generate image data of a visual marker, which is then evaluated by the control system to determine the position of the sample stage. Thus, the optical imaging sensor 150 is configured to generate image data. For example, the optical imaging sensor 150 and / or the optical imaging sensor of the microscope 110 may include, or be, an APS (Active Pixel Sensor) or CCD (Charge-Coupled-Device) type imaging sensor. For example, in an APS type imaging sensor, light is recorded at each pixel using a photodetector and an active amplifier for each pixel. APS type imaging sensors are often based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology. In a CCD type imaging sensor, incident photons are converted into electronic charges at the semiconductor-oxide interface, and then moved between capacitive bins within the imaging sensor by the control circuit of the imaging sensor to perform imaging. System 160 is configured to acquire (i.e., receive or read) image data from optical imaging sensors. Optionally, System 160 may be configured to further acquire image data from the optical imaging sensors of the microscope 110. Image data may be acquired by receiving image data from each optical imaging sensor (e.g., via interface 162), by reading image data from the memory of each optical imaging sensor (e.g., via interface 162), or by reading image data from the storage device 166 of the control system 160 after the image data has been written to the storage device 166 by each optical imaging sensor or another system or processor.

[0040] The image data from the optical imaging sensor 150 includes images showing the visual markers on the sample stage. Therefore, the optical imaging sensor 150 may be oriented toward the visual markers on the sample stage, i.e., toward the sample stage. When the visual markers are moved (together with the sample stage) above or below the optical imaging sensor, the movement of the visual markers is represented by the image data. In other words, at a given point in time, the image data includes images showing the visual markers, for example, a portion of the visual markers, according to the current position of the sample stage. For example, at each position on the sample stage, the images included in the image data may show a distinct portion of the visual markers.

[0041] In various examples, the position of the sample stage may be tracked over multiple time points. For example, the sample stage may be continuously movable, during which images of the sample are generated at multiple time points by the microscope (or rather, the microscope's optical imaging sensor). In other words, images of the sample and visual markers may be captured while the sample stage is moving. Accordingly, the image data may include multiple images showing the visual marker (or rather, multiple parts of the visual marker) representing the position of the sample stage at multiple time points. For example, the sample stage may move continuously, with the exception of changes in direction of movement. However, even if there are such discontinuities in the movement, the movement of the sample stage may be considered continuous.

[0042] Subsequently, the control system is used to analyze the image data and determine the position of the sample stage. Generally, the position of the sample may be defined with respect to one or two transverse dimensions. In other words, the control system may be configured to determine the position of the sample stage with respect to one or two transverse dimensions. For example, the control system may be configured to determine the position of the sample stage with respect to a one-dimensional or two-dimensional coordinate system representing one or two transverse dimensions. For example, the position of the sample stage may be determined by the control system for each of the one or two dimensions.

[0043] In some examples, the position of the sample stage may be determined relative to the microscope. In other words, the control system is configured to determine the position of the sample stage relative to the microscope. For example, the microscope may have a fixed position in a one-dimensional or two-dimensional coordinate system, thereby defining the position of the sample stage relative to the microscope in that one-dimensional or two-dimensional coordinate system.

[0044] In general, patterns shown in images representing visual markers may be analyzed to determine the location of the sample stage. For example, at each location of the sample stage, the image may show a portion of a visual marker, where each portion has a unique pattern. For example, a control system may be configured to compare a portion of a visual marker shown in an image with the (overall) pattern of the visual marker to determine the location of this portion within the visual marker, and then determine the location of the sample stage based on the location of this portion within the visual marker. Alternatively or additionally, the location of the sample stage may be encoded within the pattern of the visual marker. For example, a visual marker may include multiple position markers encoded within the pattern of the visual marker, where these position markers are embedded at multiple predetermined locations of the visual marker. For example, a control system may be configured to decode one or more position markers shown in an image and determine the location of the sample stage based on these decoded position markers. For example, if one or more position markers are shown in an image, the locations of one or more position markers in the image may be used to determine the location of the sample stage.

[0045] As described above, in some cases, two or more optical imaging sensors may be used in combination with two or more visual markers positioned at different locations on the sample stage. In this case, the same approach may be used depending on which of the two or more visual markers is visible at a given time. For example, the control system may be configured to determine the position of the sample stage based on image data from two or more images showing at least one of the two or more visual markers. At any given time, at least one of the two or more visual markers may be shown in at least one of the two or more images. Therefore, the control system may be configured to determine the position of the sample stage based on at least one of the two or more visual markers shown in at least one of the two or more images.

[0046] In general, the microscope systems described in relation to Figures 1a and 1c may be used as microscopes for generating (highly detailed) composite images of a sample by subsequently generating multiple (partially overlapping) images of the sample using the microscope's optical imaging sensor, and then stitching these multiple images together to generate a composite image. By synchronizing the XY camera (i.e., optical imaging sensor 150), the microscope's main camera (i.e., the microscope's optical imaging sensor), and their respective (strobe) light sources, it becomes possible to take photographs with precise XY positional information while the stage is moving. The sample stage may be used to move the sample relative to the microscope when imaging. In other systems, this process can be time-consuming because the sample stage moves to an intermediate position, stops, an image is taken, the position where the image was taken is determined, and then it moves to the next position. In various examples of this disclosure, this process can be accelerated using the proposed concept.

[0047] For example, instead of stopping the sample stage each time an image is captured by the microscope's optical imaging sensor, the sample stage may be moved continuously. In other words, the motor-driven sample stage may be configured to move continuously without being stopped, for example, for the purpose of exposing the microscope's optical imaging sensor while the sample is in a fixed position relative to the microscope, while the microscope is generating multiple images at multiple time points. Therefore, the optical imaging sensor 150 may be configured to generate image data of an image indicating a visual marker while the motor-driven sample stage is moving. In other words, while the sample stage is moving, one image, or rather multiple images stitched together, may be generated at multiple time points by the microscope's optical imaging sensor to form a composite image of the sample, and the corresponding images may be generated at multiple time points by the optical imaging sensor used to determine the position of the sample stage. The control system may be configured to determine the position of the motor-driven sample stage at multiple time points based on the image data of the image indicating a visual marker that is being generated while the motor-driven sample stage is moving. For example, at each of multiple time points, an image may be generated by the optical imaging sensor of the microscope, and by the optical imaging sensor used to determine the position of the sample stage. The position of the sample stage at multiple time points may be determined based on images captured by the optical imaging sensor 150 at multiple time points.

[0048] Motion artifacts may be observed in the respective images if the sample stage moves while the optical imaging sensor used to determine the position of the microscope's optical imaging sensor and the sample stage are exposed to light reflected from the sample and visual markers, respectively. Furthermore, optical imaging sensors often use a so-called "rolling shutter." Generally, optical imaging sensors may include a two-dimensional grid of pixels containing multiple pixel rows and multiple pixel columns. The contents of the two-dimensional grid of pixels are read out to generate frames of image data. Often, only a subset of pixels in the two-dimensional grid of pixels is read out at a time, for example, row by row or block by block. To ensure that all pixels are exposed to the same (or at least similar) amount of light, the exposure duration of each pixel may be (substantially) the same. Since some pixels are read out earlier than others, these pixels are also exposed to light earlier to maintain the same exposure duration. The slower a pixel is read out, the later its exposure begins. This concept is illustrated in relation to Figure 2.

[0049] Figure 2 shows an example of a strobe timing diagram relative to the readout of an optical imaging sensor. For example, the timing of the strobe light is synchronized with the exposure timing of the optical imaging sensor. In Figure 2, the x-axis represents time. Four frames are shown, namely the first frame 210, the second frame 220, the third frame 230, and the fourth frame 240. As can be seen from Figure 2, the exposure of pixel rows to light and the subsequent readout of pixel rows are staggered in time. Figure 2 shows a line 211 indicating the start of exposure for the first row, and a further line 212 indicating the start of exposure for the last row of the first frame 210. Figure 2 further shows a line 214 indicating the end of exposure for the first row and a line 215 indicating the end of exposure for the last row. As can be seen from Figure 2, the end of exposure for the last row of the first frame 210 may coincide with the start of exposure for the first row (or rather, most rows) of the second frame 220. However, there is a time interval during which all pixels of the optical imaging sensor are exposed to the same light (between the start of exposure for the last row 212 and the end of exposure for the first row 214).

[0050] At this moment, the sample (and / or visual marker) may be illuminated using strobe light 213, and so the light recorded by the optical imaging sensor is primarily based on the reflection of the strobe light, and any light recorded separately from the strobe reflection is extinguished. In other words, in order to reduce or minimize rolling shutter distortion and for consistent illumination of the visual pattern, the optical imaging sensor and / or the optical imaging sensor of the microscope used to determine the position of the sample stage may have a synchronized strobe light 170 mounted next to or around the lens (as shown in Figure 1c). The main optical imaging sensor and light source of the microscope may similarly be synchronized with the optical imaging sensor and light source used to determine the position of the sample stage to enable continuous capture while the XY stage is moving. In other words, the microscope system may include at least one illumination element 170 for illuminating the visual marker and / or sample with a light strobe. The control system may be configured to trigger the illumination element to emit a light strobe at multiple points in time, for example, during frame generation, before exposure of the first row of the pixel grid is stopped and after exposure of the last row of the pixel grid is started. If the sensor is exposed row by row, for example, during frame generation, before exposure of the first row of the pixel grid is stopped and after exposure of the last row of the pixel grid is started, the control system may be configured to trigger the illumination element to emit a light strobe so that each pixel of the image data sample of the image representing the visual markers produced by the microscope is exposed to the reflection of the same light strobe. Thus, the light / illumination element may emit a short but strong burst of light after exposure of the last row of each optical imaging sensor is started, but before exposure of the first row is stopped. In practice, in cases where an image sensor with an electronic rolling shutter is used, a strobe light source can be used to eliminate image distortion during stage movement.

[0051] The resulting image position resolution is determined by the combination of magnification and signal-to-noise ratio (SNR). For example, assuming the SNR is high enough for accurate 1 / 4 pixel precision motion estimation, the image sensor has a pixel pitch of 1.55 μm, and the required XY precision is 0.4 μm. A lens with a magnification of 1.55 / 0.4 / 4 = 0.96875x or higher may be sufficient. The maximum speed of the stage may be limited by the camera's field of view and frame rate, so that the stage can have image content that overlaps sufficiently between adjacent frames for motion calculation.

[0052] Generally, the position of the sample stage is used to stitch together images generated by the microscope's optical imaging sensor to produce a composite image. For example, information about the position of the sample stage at multiple time points in time may be stored in the control system's storage device 166 or provided to an external entity. Alternatively or additionally, information about the position of the sample stage may be stored together with the images generated by the microscope's optical imaging sensor. For example, the control system may be configured to annotate images captured by the microscope with the position of the sample stage at the time the image was generated. In other words, the control system may be configured to further acquire image data from the microscope system's optical imaging sensor, including images captured by the microscope's optical imaging sensor, and annotate the images captured by the optical imaging sensor with positional information, which is done, for example, by generating a corresponding data unit for each captured image, for example, a file representing the position of the sample stage, and storing or outputting the positional data together with the captured image file. Alternatively, positional information may be included in the metadata of each image captured by the microscope's optical imaging sensor.

[0053] Therefore, various examples of this concept provide a concept for determining the position of a sample stage. For example, a fragment of a pattern (i.e., a visual marker) may be positioned either above or below a sample stage, which may be an XY stage movable along two lateral dimensions X and Y. The fragment of the pattern may, advantageously, exhibit properties related to thermal expansion, similar to those of the sample stage itself. Furthermore, the dimensions of the visual marker may be at least the same as the range of XY movement of the stage. Alternatively, multiple fragments of the pattern may be associated with the same number of cameras and distributed across different areas of the stage, thereby reducing the stage size. The pattern may be random dots or an inherent texture of the material surface. Specular reflection may be reduced to allow for visual tracking of movement. A digital camera, e.g., an optical imaging sensor used to determine the position of the sample stage, is directed at the pattern and used to determine the position of the sample stage. The digital camera sends image data (e.g., video) to a processor to analyze and track the movement of the stage. Since the pattern is located on the stage itself, its movement accurately represents the movement of the sample stage, and consequently, the (XY) position of the sample stage.

[0054] One or more interfaces 162 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, one or more interfaces 162 may include interface circuits configured to receive and / or transmit information. In embodiments, one or more processors 164 may be implemented using one or more processing units, one or more processing units, or any means for processing, such as a processor, computer, or programmable hardware component, which are operable with appropriately adapted software. In other words, the described functions of one or more processors 164 may be implemented in software, in which case the software runs on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, and the like. In at least some embodiments, one or more storage devices 166 may include at least one element of a group of computer-readable storage media, such as a magnetic storage medium or an optical storage medium, which may be, for example, a hard disk drive, flash memory, a solid-state drive (SSD), a floppy disk, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or network storage.

[0055] Further details and embodiments of the microscope system and control system are referred to in relation to the proposed concept or one or more examples described above or below (e.g., Figure 3). The microscope system and / or control system may include one or more additional optional functions corresponding to the proposed concept or one or more embodiments of one or more examples described above or below.

[0056] Figure 3 shows a flowchart of an example of a corresponding method for a microscope system, for example, the microscope system 100 in Figures 1a-1c. As shown in relation to Figures 1a-1c, the microscope system has a microscope 110 configured to produce an image of a sample 120 placed on a sample stage, and a sample stage 130 configured to support the sample 120, the sample stage including a visual marker 140. The method includes acquiring image data 310 of an image showing the visual marker on the sample stage. The method includes determining the position of the sample stage 320 based on the image data of the image showing the visual marker.

[0057] As described above, the features described in relation to the microscope system 100, control system 160, sample stage 130, microscope 110, and at least one illumination element 170 in Figures 1a to 1c can similarly be applied to the method in Figure 3.

[0058] As used herein, the term "and / or" includes all possible combinations of one or more of the related items and may be abbreviated as " / ".

[0059] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus.

[0060] Some embodiments relate to microscopes that include systems such as those described in relation to one or more of Figures 1a to 3. Alternatively, the microscope may be part of a system such as those described in relation to one or more of Figures 1a to 3, or may be connected to a system such as those described in relation to one or more of Figures 1a to 3. Figure 4 shows a schematic diagram of a system 400 configured to carry out the methods described herein. The system 400 includes a microscope 410 and a computer system 420. The microscope 410 is configured to take images and is connected to the computer system 420. The computer system 420 is configured to carry out at least a portion of the methods described herein. The computer system 420 may be configured to run machine learning algorithms. The computer system 420 and the microscope 410 may be separate entities but can be integrated into a single common housing. The computer system 420 may be part of the central processing system of the microscope 410, and / or the computer system 420 may be part of a dependent component of the microscope 410, such as a sensor, actuator, camera, or lighting unit of the microscope 410.

[0061] The computer system 420 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed in various locations, which are local clients and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which may be of any type. As used herein, a processor may mean any type of computing circuit of a microscope or microscope component (e.g., a camera) or any other type of processor or processing circuit. This is not limited to, but includes, for example, microprocessors, microcontrollers, composite instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, graphics processors, digital signal processors (DSPs), multicore processors, and field-programmable gate arrays (FPGAs). Other types of circuits that may be included in the computer system 420 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (communication circuits, etc.) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 420 may include one or more storage devices that may include one or more memory elements suited to a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, and digital video discs (DVDs).The computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that enables a system user to input information into and receive information from the computer system 420.

[0062] Some or all of the steps may be performed by a hardware device (or by using a hardware device), such as a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most critical steps may be performed by such a device.

[0063] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation may be carried out using a non-temporary storage medium such as a digital storage medium, which may be a floppy disk, hard disk drive (HDD), solid-state drive (SSD), DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, where electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to carry out each method. Thus, the digital storage medium may be computer-readable.

[0064] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so as to carry out any of the methods described herein.

[0065] In general, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to perform one of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.

[0066] Other embodiments include a computer program stored on a machine-readable carrier for carrying out any of the methods described herein.

[0067] Therefore, in other words, embodiments of the present invention are computer programs having program code for carrying out any of the methods described herein when the computer program is executed on a computer.

[0068] Therefore, another embodiment of the present invention is a storage medium (or data carrier or computer-readable medium) containing a computer program stored therein for carrying out any of the methods described herein when executed by a processor. A data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transient. Another embodiment of the present invention is the apparatus described herein, comprising a processor and a storage medium.

[0069] Therefore, another embodiment of the present invention is a data stream or signal sequence representing a computer program for carrying out any of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0070] Another embodiment includes processing means, for example, a computer or programmable logic device configured or adapted to carry out any of the methods described herein.

[0071] Another embodiment includes a computer having a computer program installed thereon for carrying out any of the methods described herein.

[0072] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for carrying out any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0073] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the method described herein. In some embodiments, a field-programmable gate array can cooperate with a microprocessor to carry out any of the methods described herein. Generally, the method is advantageously carried out by any hardware device. [Explanation of symbols]

[0074] 100 Microscope Systems 110 Microscopes 120 samples 130 Sample Stages 140 Visual Markers 150 Optical Imaging Sensors 160 Control System 162 Interfaces 164 processors 166 Storage device 170 lighting elements 210 First Frame 211 Start of exposure for the first row 212 Start of exposure for the last row 213 Strobe 214 End of exposure for the first row 215 End of exposure for the last row 220 Second Frame 230 Third Frame 240 The fourth frame 310 Acquisition of image data 320 Determination of the sample stage position 400 System 410 Microscope 420 Computer Systems

Claims

1. A microscope system (100; 400) comprising: a microscope (110; 410) configured to generate an image of a sample (120) disposed on a sample stage; a sample stage (130) configured to support the sample, the sample stage (130) including a visual marker (140); an optical imaging sensor (150) configured to provide image data of an image indicative of the visual marker of the sample stage; a control system (160; 420) configured to determine a position of the sample stage based on the image data of the image showing the visual marker; A microscope system (100; 400) comprising:

2. the sample stage is configured to move along two lateral dimensions; the control system is configured to determine the position of the sample stage with respect to the two lateral dimensions. The microscope system according to claim 1 .

3. the sample stage is a motor-driven sample stage; The microscope system according to claim 1 .

4. the motorized sample stage is configured to move continuously while the microscope generates a plurality of images at a plurality of time points; the optical imaging sensor is configured to generate the image data of the image indicative of the visual marker while the motorized sample stage is moving; the control system is configured to determine the position of the motorized sample stage at the plurality of times based on the image data of images indicative of the visual markers being generated while the motorized sample stage is moving. The microscope system according to claim 3 .

5. the microscope system includes at least one illumination element (170) for illuminating the visual marker and / or the sample with a light strobe; the control system is configured to trigger the lighting elements to emit light strobes at the multiple times.

5. The microscope system according to claim 4.

6. the control system is configured to trigger the illumination elements to fire the light strobes such that each pixel of an image produced by the microscope and / or each pixel of a sample of the image data of the image showing the visual marker is exposed to a reflection of the same light strobe. The microscope system according to claim 5 .

7. the visual marker is disposed on a bottom side of the sample stage, and the optical imaging sensor is disposed below the sample stage; The microscope system according to claim 1 .

8. the visual marker is disposed above the sample stage, and the optical imaging sensor is disposed above the sample stage; The microscope system according to claim 1 .

9. the microscope system including two or more optical imaging sensors for providing image data of two or more images indicative of at least one visual marker of the sample stage; the control system is configured to determine the position of the sample stage based on the image data of the two or more images showing the at least one visual marker of the two or more visual markers. The microscope system according to claim 1 .

10. the lateral extent of the visual markers is at least as large as the maximum lateral movement of the sample stage; The microscope system according to claim 1 .

11. the visual marker having a two-dimensional visual pattern; The microscope system according to claim 1 .

12. the sample stage includes a first visual marker aligned with a first lateral direction of movement of the sample stage and a second visual marker aligned with a second lateral direction of movement of the sample stage, each of the first visual marker and the second visual marker having a one-dimensional visual pattern; The microscope system according to claim 1 .

13. the control system is configured to determine the position of the sample stage relative to the microscope. The microscope system according to claim 1 .

14. the control system is configured to annotate the image captured by the microscope with the position of the sample stage at the time the image was generated. The microscope system according to claim 1 .

15. A control system (160; 420) for a microscope system according to any one of claims 1 to 14, the microscope system comprising: a microscope (110; 410) configured to generate an image of a sample (120) disposed on a sample stage (130); a sample stage (130) configured to support the sample, the sample stage (130) including a visual marker (140); Contains The control system includes one or more processors (164) and one or more memory devices (166); The control system includes: acquiring image data from an optical imaging sensor (150) of the microscope system for an image indicative of the visual marker of the sample stage; determining a position of the sample stage based on the image data of the image showing the visual marker; It is configured as follows: Control system (160; 420).

16. A method for a microscope system according to any one of claims 1 to 14, comprising: The microscope system includes a microscope configured to generate an image of a sample disposed on a sample stage, and a sample stage configured to support the sample; the sample stage includes a visual marker; The method comprises: acquiring (310) image data of an image indicative of the visual marker of the sample stage; determining (320) a position of the sample stage based on the image data of the image showing the visual marker; A method comprising:

17. A computer program comprising: The computer program comprises a program code for performing the method according to claim 16 when the computer program is executed on a processor. Computer program.

Images