Optical imaging device for microscope

Abstract

An optical imaging device for a microscope includes: a first and a second optical system, respectively configured to form first and second optical images corresponding to first and second regions of a sample according to first and second imaging modes, wherein the first and second regions spatially coincide with a target region, and the first and second imaging modes are different from each other; a memory storing first and second distortion correction data and transformation data suitable for correcting misalignment in position between the first and second optical images, the distortion correction data being suitable for correcting optical distortion caused by the optical system in the optical images; and a processor configured to process first image data based on the first distortion correction data to generate first distortion-corrected image data, process second image data based on the second distortion correction data to generate second distortion-corrected image data; and combine the first and second distortion-corrected image data based on the transformation data to generate combined image data.

Description

Field of the Invention

[0001] The present invention relates to an optical imaging device for a microscope. Furthermore, the present invention relates to a method for imaging a sample using a microscope and a method for calibrating an optical imaging device of a microscope. Background Art

[0002] In the field of microscopes, a variety of different imaging modes are used to generate optical images of samples. Each of these imaging modes has advantages and disadvantages in terms of image quality, spatial resolution, imaging speed, exposure, etc. For example, while confocal imaging can achieve higher spatial resolution, it has the disadvantage of a long image acquisition time because the sample must be scanned point by point to create an image. In contrast, the wide-field imaging mode is advantageous in terms of the time required for image acquisition. However, the image resolution may be significantly reduced.

[0003] Therefore, there is a need to combine different modes when imaging the same sample. However, since different imaging modes may use different optical paths, it is a challenge to make two images generated in different modes coincide so that a properly combined image can be displayed on, for example, a monitor. In the case where the user manually registers or aligns the images, similar image structures on which registration can be performed are required. This is a cumbersome task and in many cases precise image fusion is not possible.

[0004] In this context, reference is made to the document EP 2 322 969 B1, which discloses a microscope that includes a plurality of observation optical systems capable of acquiring images of the same sample in different modes. This microscope allows for making full use of the respective fields of view of the plurality of observation optical systems, thereby improving work efficiency. However, the aforementioned document does not address automatically merging images generated by applying different imaging modes into a combined image. Summary of the Invention

[0005] The object of the present invention is to provide an optical imaging device and method that are capable of precisely combining images generated according to different imaging modes. Furthermore, the object is to provide a method for calibrating an optical imaging device of a microscope such that the optical imaging device is capable of precisely combining images.

[0006] The aforementioned object is achieved by the subject matter of the independent claims. Advantageous embodiments are defined in the dependent claims and the following description.

[0007] According to an embodiment, an optical imaging device for a microscope includes a first optical system and a second optical system. The first optical system is configured to form a first optical image corresponding to a first region of a sample according to a first imaging mode. The second optical system is configured to form a second optical image corresponding to a second region of the sample according to a second imaging mode. The first region and the second region spatially coincide with a target region of the sample, and the first imaging mode and the second imaging mode are different from each other. The optical imaging device further includes a memory that stores first distortion correction data suitable for correcting a first optical distortion caused by the first optical system in the first optical image, second distortion correction data suitable for correcting a second optical distortion caused by the second optical system in the second optical image, and transformation data suitable for correcting a positional misalignment between the first optical image and the second optical image. The optical imaging device further includes a processor configured to process first image data representing the first optical image based on the first distortion correction data to generate first distortion-corrected image data. The processor is further configured to process second image data representing the second optical image based on the second distortion correction data to generate second distortion-corrected image data. The processor is configured to combine the first distortion-corrected image data and the second distortion-corrected image data based on the transformation data to generate combined image data representing a combined image corresponding to the target region of the object.

[0008] The optical imaging device includes two optical systems, where each of these optical systems can have an image sensor adapted to a specific imaging mode. It is considered that each optical system may cause an optical distortion in the optical image generated by that optical system. The first optical system and the second optical system can use different optical paths for imaging, such that the optical distortions caused by the first optical system and the second optical system can be independent of each other. Therefore, the first distortion correction data and the second distortion correction data are also not related to each other. The distortion correction data can be determined and stored independently for each optical system during assembly. Typically, the optical distortion caused by the corresponding optical system represents an aberration that causes image blurring or distortion, such that the correct alignment of the image is adversely affected. This is even more true since the corresponding optical distortions are significantly different from each other due to different imaging modes. Any such adverse effects can be avoided by storing the first distortion correction data and the second distortion correction data, and these distortion correction data will be automatically taken into account when the first optical image and the second optical image are combined into a combined image.

[0009] The memory of the optical imaging device also stores transformation data suitable for correcting the misalignment of positions between the first optical image and the second optical image. Compared with the first distortion correction data and the second distortion correction data that can be determined independently for each optical system, the transformation data represents data considering both optical systems, especially considering the positional relationship between the optical systems that may cause the misalignment of positions between the optical images.

[0010] The first optical system and the second optical system are used to image the first region and the second region of the sample respectively, where these regions spatially coincide with the target region of the sample. The first region and the second region of the sample can coincide spatially in different ways. For example, the first sample region and the second sample region can be the same as each other, so that the target region itself is also the same as the corresponding region. Alternatively, one of the sample regions can be completely included in the other region, so that the target region is formed by the included sample region. In other alternatives, the first region and the second region can partially overlap. In this case, the target region is formed by the overlapping region common to the two sample regions.

[0011] The optical imaging device can include a display unit on which a combined image corresponding to the imaging target region of the sample is displayed. Thus, the user can observe the target region of the sample based on the synthetic image that benefits from the advantages of both imaging modes.

[0012] Preferably, the first imaging mode is a wide-field mode and the second imaging mode is a confocal imaging mode. The synthetic image based on these fundamentally different imaging modes provides the user with image information far beyond standard image information.

[0013] In a preferred embodiment, the transformation data represents the misalignment of positions between the first optical reference image formed by the first optical system according to the first imaging mode and the second optical reference image formed by the second optical system according to the second imaging mode. Optical reference images can be generated using a reference object suitable for imaging properly in both imaging modes. Alternatively, real-time images generated during actual imaging can be used as reference images.

[0014] Preferably, a calibration mode is provided in which the processor is configured to generate transformation data and store the transformation data in the memory before forming the first optical image and the second optical image. In this calibration mode, the processor can also be configured to generate the first distortion correction data and the second distortion correction data and store this data in the memory. Preferably, the calibration mode is applied during the manufacturing and assembly process, so that the distortion correction data and the transformation data are already stored in the finished product. Therefore, the user does not need to worry about any calibration. Instead, the user is allowed to focus entirely on experiments including sample preparation, adjusting imaging parameters, etc.

[0015] The processor may be configured to determine correlation data representing the correlation between a first optical reference image and a second optical reference image, and generate transformation data based on the correlation data. For example, an algorithm that determines a correlation coefficient based on the recognition of structural features in an image may be applied. Based on this information, an iterative optimization process may be used to determine the required coordinate transformation.

[0016] The positional misalignment represented by the transformation data may include translation, rotation, scaling, shearing, mirroring, and / or distortion.

[0017] In a preferred embodiment, the processor is configured to update the transformation data and store the updated transformation data in a memory. By updating the transformation data, the user is enabled to react to changes that occur during an experiment. For example, changes caused by drift in a sample, structural modifications, dynamic processes, etc. may be compensated for by re-determining the transformation data stored in the memory.

[0018] As an example, the processor may be configured to cause a first optical system to generate a first sequence of optical images according to a first imaging mode, and cause a second optical system to generate a second sequence of optical images according to a second imaging mode. To re-determine the transformation, the processor may also be configured to determine a first tracking marker within one of the first sequence of optical images, and determine a second tracking marker within one of the second sequence of optical images. In this case, the processor is configured to perform tracking of the first tracking marker and the second tracking marker, and update the transformation data based on the tracking. By automatically defining so-called fiducials in the form of the tracking markers in the initial images mentioned above, and tracking these fiducials over time, the transformation can be re-calibrated without having to use a calibration standard in the form of a reference object again.

[0019] The processor may also be configured to combine first distortion-corrected image data and second distortion-corrected image data based on the transformation data, such that one of the first optical image and the second optical image is mapped to a reference frame defined by the other of the first optical image and the second optical image, or such that both optical images are mapped to a common reference frame. In other words, a coordinate transformation is applied through which the coordinate system of one optical image is transformed to the coordinate system of the other image, or the coordinate systems of both optical images are transformed to a new common coordinate system.

[0020] The first optical system and the second optical system may be mounted in a fixed positional relationship with respect to each other. Preferably, two different imaging modes are adapted to each other as closely as possible by physically aligning the corresponding optical elements. This alignment allows for minimizing the required transformation of image points. Therefore, interference caused by interpolation, rotation, etc. can be significantly reduced.

[0021] According to another aspect, a method of imaging a sample using a microscope is provided. The imaging method includes the following steps: forming a first optical image corresponding to a first region of the sample according to a first imaging mode through a first optical system; forming a second optical image corresponding to a second region of the sample according to a second imaging mode through a second optical system, wherein the first region and the second region spatially coincide with a target region of the sample and the first imaging mode and the second imaging mode are different from each other; obtaining first distortion correction data suitable for correcting a first optical distortion caused by the first optical system in the first optical image; obtaining second distortion correction data suitable for correcting a second optical distortion caused by the second optical system in the second optical image; obtaining transformation data suitable for correcting a misalignment in position between the first optical image and the second optical image; processing first image data representing the first optical image based on the first distortion correction data for generating first distortion-corrected image data; processing second image data representing the second optical image based on the second distortion correction data for generating second distortion-corrected image data; and combining the first distortion-corrected image data and the second distortion-corrected image data based on the transformation data for generating combined image data representing a combined image corresponding to the target region of the object.

[0022] According to another aspect, a method of calibrating an optical imaging device of a microscope is provided, the optical imaging device including: a first optical system configured to form a first optical image corresponding to a first region of a sample according to a first imaging mode; a second optical system configured to form a second optical image corresponding to a second region of the sample according to a second imaging mode, wherein the first region and the second region spatially coincide with a target region of the sample and the first imaging mode and the second imaging mode are different from each other; a processor; and a memory. The method includes the following calibration steps: obtaining first distortion correction data suitable for correcting a first optical distortion caused by the first optical system in the first optical image; obtaining second distortion correction data suitable for correcting a second optical distortion caused by the second optical system in the optical image; obtaining transformation data suitable for correcting a misalignment in position between the first optical image and the second optical image; and storing the first distortion correction data, the second distortion correction data, and the transformation data in the memory for access by the processor.

[0023] The calibration method can be performed during the manufacturing and assembly process such that an assembled product can be provided to a user having the required calibration data already stored therein.

[0024] According to a preferred embodiment, a first optical reference image of a reference object is formed by a first optical system according to a first imaging mode. A second optical reference image of the reference object is formed by a second optical system according to a second imaging mode. Transformation data is determined based on the misalignment of the positions of the first reference image and the second reference image. By using a single calibration standard in the form of the aforementioned reference object, the transformation data can be determined in a simple and reproducible manner. The calibration standard can also be used to determine first distortion correction data and second distortion correction data.

[0025] By way of example only, the reference object may include a grid formed by a plurality of points. The grid is adapted to the first optical system and the second optical system such that each optical system can image at least two of the plurality of points within the available magnification range. Such a grid is used to ensure that the calibration standard contains sufficient structural information to achieve the required accuracy. In particular, the grid is formed to represent the same structure in both imaging modes. In addition, the grid can be imaged with both transmitted light and fluorescence. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Hereinafter, specific embodiments will be described with reference to the accompanying drawings, wherein:

[0027] Figure 1 is a schematic diagram showing an optical imaging device for a microscope according to an embodiment;

[0028] Figure 2 is a diagram illustrating an example of the positional relationship between a first region and a second region of a sample to be imaged by the optical imaging device;

[0029] Figure 3 is a diagram showing a reference object for calibration according to an embodiment;

[0030] Figure 4 is a diagram illustrating a tracking process for updating transformation data;

[0031] Figure 5 is a flowchart showing a method for calibrating an optical imaging device according to an embodiment;

[0032] Figure 6 is a flowchart showing a method for imaging a sample using a microscope according to an embodiment; and

[0033] Figure 7 is a diagram illustrating the transformation of coordinate systems respectively assigned to a first optical image and a second optical image to a common reference coordinate system. DETAILED DESCRIPTION

[0034] Figure 1A microscope 100 is shown including an optical imaging device 102 configured to image a sample 104 on a microscope stage 106 according to different imaging modes. For example, the optical imaging device 102 can be used to image the sample 104 in a wide-field imaging mode and a confocal imaging mode. Needless to say, these imaging modes should only be understood as examples. Any other mode can be applied as long as these modes are adapted to provide image information that can be reasonably combined to generate a synthetic image.

[0035] The optical imaging device 102 includes a first optical system 108 configured to form a first optical image corresponding to a first region 210 of the sample 104 (see Figure 2 ) according to a first imaging mode, which is a wide-field imaging mode in this embodiment. Similarly, the optical imaging device 102 includes a second optical system 112 configured to form a second optical image corresponding to a second region 214 of the sample 104 (see Figure 2 ) according to a second imaging mode, which is a confocal imaging mode in this embodiment. As Figure 1 schematically illustrated, the first optical system 108 and the second optical system 112 use different optical paths 116 and 118 respectively, and the detection light emitted from the sample 104 propagates along the optical paths 116 and 118 through the optical systems 108, 112 to the image sensors 120, 122 respectively coupled to the optical systems 108, 112. The first image sensor 120 assigned to the first optical system 108 can be formed by a camera suitable for wide-field imaging. The second image sensor 122 assigned to the second optical system 112 can be formed by a sensor suitable for confocal imaging, such as a point detector.

[0036] The optical imaging device 102 further includes a processor 124 that can be used to control the overall operation of the optical imaging device 102. In particular, the processor 124 is configured to process first image data representing the first optical image and second image data representing the second optical image, wherein the first optical image and the second optical image are generated on the image sensors 120, 122 by the first optical system 108 and the second optical system 112 respectively. For this purpose, the processor 124 is connected to the optical systems 108, 112 and the image sensors 120, 122 via control lines 126, 128, 130, 132.

[0037] The optical imaging device further includes a memory 134 connected to the processor 124 via a control line 136. In addition, a display unit 138 connected to the processor 124 via a control line 140 can be provided.

[0038] In addition, in order to provide different optical paths 116, 118 towards the first optical system 108 and the second optical system 112, a beam splitter or any other suitable light deflector 142 may be included in the optical imaging device 102.

[0039] As mentioned above, the optical imaging device 102 is capable of operating in a wide-field imaging mode and a confocal imaging mode in order to image a first region 210 and a second region 214 of the sample 104. The first region 210 and the second region 214 spatially coincide with the target region of the sample 104, which is illustrated by the shaded region 242 in Figure 2 The spatial coincidence can be achieved in different ways. For example, the second region 214 assigned to the confocal imaging mode may be completely included in the first region, as shown in Figure 2 (a). Since the target region 242 is formed by the overlap of the first region 210 and the second region 214, the target region 242 is the same as the second region 214 in the example shown in Figure 2 (a). In addition, Figure 2 (b) shows an example where the first region 210 and the second region 214 are the same such that the target region 242 is likewise the same as the corresponding region. In the example shown in Figure 2 (c), the first region 210 and the second region 214 partially overlap, such that the target region 242 is formed by the overlapping region common to the two regions 210, 214. Needless to say, the spatial coincidence between the first region 210 and the second region 214 is not limited to the Figure 2 shown examples.

[0040] In the Figure 1 illustrated embodiment, a memory 134 is provided for storing first distortion correction data and second distortion correction data. The first distortion correction data is adapted to correct a first optical distortion (e.g., optical aberration) caused by the first optical system 108 when generating a first optical image of the first region 210 of the sample 104 according to the wide-field imaging mode. Similarly, the second distortion correction data is used to correct a second optical distortion caused by the second optical system 112 when generating a second optical image of the second region 214 of the sample 104 according to the confocal imaging mode. Since the first optical system 108 and the second optical system 112 use different optical paths 116, 1,18 to image the corresponding regions 210, 214, the optical distortions caused by the first optical system 108 and the second optical system 112 are independent of each other. Therefore, the first distortion correction data and the second distortion correction data can be determined independently and stored in the memory 134.

[0041] In addition, the memory 134 stores transformation data that can be used to correct misalignment in the positions between a first optical image and a second optical image created respectively in a wide-field imaging mode and a confocal imaging mode. Although the first distortion correction data and the second distortion correction data can be independently assigned to each of the optical systems 108, 112, the transformation data stored in the memory 134 reflects the positional relationship between the optical systems 108, 112.

[0042] The processor 124 utilizes the first distortion correction data, the second distortion correction data, and the transformation data stored in the memory 134 to provide a combined image corresponding to the target region 242 of the object 104. This combined image provides image information derived from both wide-field imaging and confocal imaging. To create the combined image, the processor generates first distortion-corrected image data by processing first image data representing the first optical image based on the first distortion correction data. Similarly, the processor 124 generates second distortion-corrected image data by processing second image data representing the second optical image based on the second distortion correction data. Then, based on the transformation data, the processor combines the first distortion-corrected image data and the second distortion-corrected image data so as to create combined image data representing the combined image to be displayed on the display unit 134.

[0043] The optical imaging device 102 can provide a calibration mode in which the processor 124 generates the transformation data and stores this data in the memory 134. This calibration mode is preferably applied during the manufacturing and assembly process of the optical imaging device 102 such that the calibration mode can be automatically used at a later stage when the user operates the microscope 100 to image the sample 104. The first distortion correction data and the second distortion correction data independently assigned to the respective optical systems 108, 112 can similarly be generated by the processor 124 in the calibration mode.

[0044] To calibrate the optical imaging device 102, a reference object can be used. By way of example only, such a reference object can be formed by a grid 350 as Figure 3 shown.

[0045] The grid 350 includes a plurality of points 352 arranged in a rectangular array. The grid 350 is adapted to the first optical system 108 and the second optical system 112 such that, despite the fact that the optical systems 108, 112 apply different imaging modes, the two optical systems 108, 112 are capable of imaging at least two of the points 352 within the available magnification range of the microscope 100.

[0046] For calibration purposes, the first optical system 108 generates a first optical reference image of the grid 350 in a wide field of view imaging mode. Correspondingly, the second optical system 112 generates a second optical reference image of the grid 350 in a confocal imaging mode. Subsequently, the processor 124 generates first distortion correction data and second distortion correction data, and stores this data in the memory 134. In addition, the processor determines a positional misalignment between the first reference image and the second reference image representing the grid 350. Based on this misalignment, the processor 124 generates transformation data and stores this data in the memory 134.

[0047] It should be understood that using a reference object as shown in Figure 3 for calibrating the optical imaging device 102 is merely an example. Thus, transformation data can also be generated based on real-time images created when the user operates the microscope 100 to image the sample 104. In addition, real-time images can also be used to update the initial transformation data stored in the memory 134, so as to compensate for drifts, structural modifications, dynamic processes, etc. in the sample that occur during the experiment. For example, the processor 124 can cause each of the optical systems 108, 112 to generate a sequence of optical images according to the corresponding imaging mode. For each sequence, the processor determines a tracking marker 452 as shown in Figure 4 in the initial image of the sequence, and tracks the tracking marker 452 over time, i.e., on multiple images generated after the initial image. Using the tracking marker 452 enables the processor 124 to generate updated transformation data to recalibrate the transformation without having to use a reference object as shown in Figure 3

[0048] To generate transformation data, the processor 124 can be configured to determine correlation data representing the correlation between reference images. As explained above, images representing reference objects or real-time images can be used.

[0049] Figure 5 The flowchart of

[0050] illustrates a method for calibrating the optical imaging device 102 according to an embodiment. Figure 3 In step S2, the first optical system 108 images a reference object as shown in

[0051] ​In step S8, the second optical system 112 forms a second reference image, and the processor 124 stores the corresponding image data in the image memory. In step S10, the processor 124 determines the second distortion correction data. In step S12, the processor 124 stores the second distortion correction data in the memory 134. Steps S8, S10, and S12 for generating and storing the second distortion correction data are performed in the same manner as steps S2, S4, and S6 with respect to the first distortion correction data.

[0052] In step S14, as explained above, the processor 124 determines the transformation data based on the first reference image and the second reference image. Finally, in step S16, the processor 124 stores the transformation data in the memory 134.

[0053] Figure 6 The flowchart shown illustrates a method for imaging a sample 104 according to an embodiment.

[0054] In step S12, the first optical system 108 forms a first optical image of the first target region 210 according to a wide field of view imaging mode. Correspondingly, in step 14, the second optical system 112 forms a second optical image of the second region 214 according to a confocal imaging mode. As referenced above Figure 2 as explained, the first and second regions of the sample 104 spatially coincide with the target region 242. In steps 12 and 14, the first image data and the second image data representing the first optical image and the second optical image, respectively, are stored in the image memory.

[0055] After the first image and the second image have been generated, in step S16 the processor 124 reads the first distortion correction data from the memory 134. Similarly, in step S18, the processor 124 reads the second distortion correction data from the memory 134. In step S20, the processor 124 reads the transformation data from the memory 134.

[0056] In step S22, the processor 124 processes the first image data representing the first optical image based on the first distortion correction data to create first distortion correction image data. Correspondingly, in step 24, the processor 124 processes the second image data representing the second optical image based on the second distortion correction data to create second distortion correction image data. The first distortion correction image data and the second distortion correction image data are stored in the image memory.

[0057] Finally, in step S26, the processor 124 combines the first distortion-corrected image data and the second distortion-corrected image data using the transformation data read from the memory 134. Thus, combined image data representing the combined image is created. The combined image corresponding to the target region 242 of the sample 104 can be displayed on the display unit 138. In addition, the combined data can be stored in an image memory not shown in the figure.

[0058] In Figure 7 is schematically illustrated a specific transformation performed by the processor 124 for combining the first optical image and the second optical image. According to Figure 7 the example of, the processor 124 uses two mapping operations T' and T” for transforming the first coordinate system KS1 assigned to the first image and the second coordinate system KS2 assigned to the second image into a third coordinate system KS3. The third coordinate system KS3 represents a common reference system assigned to the combined image formed by the first image and the second image.

[0059] Needless to say Figure 7 the transformation shown in is only an example. Thus, the processor 124 can combine the first distortion-corrected image data and the second distortion-corrected image data based on the transformation data such that the first coordinate system KS1 is mapped to the second coordinate system KS2 and vice versa.

[0060] Although some aspects have been described in the context of a device, it is clear that these aspects also represent a description of the corresponding method, where the blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of the corresponding blocks or items or features of the corresponding device. Some or all of the method steps can be performed by (or using) hardware devices such as processors, microprocessors, programmable computers or electronic circuits. In some embodiments, one or more of the most important method steps can be performed by such a device.

[0061] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, which has electronically readable control signals stored thereon that cooperate (or are capable of cooperating) with a programmable computer system such that the corresponding method is performed. Thus, the digital storage medium can be computer-readable.

[0062] Some embodiments according to the present invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system such that one of the methods described herein is performed.

[0063] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when the computer program product is run on a computer, is operable to perform one of the methods. The program code can be stored, for example, on a machine-readable carrier.

[0064] Other embodiments include a computer program stored on a machine-readable carrier for performing one of the methods described herein.

[0065] In other words, thus, an embodiment of the present invention is a computer program that, when run on a computer, has program code for performing one of the methods described herein.

[0066] Thus, other embodiments of the present invention are a storage medium (or data carrier, or computer-readable medium) that includes a computer program stored thereon for performing one of the methods described herein when executed by a processor. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory. Other embodiments of the present invention are a device as described herein that includes a processor and a storage medium.

[0067] Thus, other embodiments of the present invention are a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence can be configured, for example, to be transmitted via a data communication connection (such as the Internet).

[0068] Other embodiments include a processing device (such as a computer or a programmable logic device) configured or adapted to perform one of the methods described herein.

[0069] Other embodiments include a computer having a computer program installed thereon for performing one of the methods described herein.

[0070] Other embodiments according to the present invention include a device or system configured to (e.g., electronically or optically) transmit a computer program for performing one of the methods described herein to a receiver. The receiver can be, for example, a computer, a mobile device, a storage device, etc. The device or system can include, for example, a file server for transmitting the computer program to the receiver.

[0071] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0072] List of Reference Signs

[0073]

[0074]

Claims

1. An optical imaging device (102) for a microscope (100), comprising: a first optical system (108) configured to form a first optical image corresponding to a first region (210) of the sample (104) according to a first imaging mode, a second optical system (112) configured to form a second optical image corresponding to a second region (214) of the sample (104) according to a second imaging mode, wherein the first region and the second region (214) spatially coincide with a target region (242) of the sample (104) and the first imaging mode and the second imaging mode are different from each other, a memory (134) storing first distortion correction data adapted to correct a first optical distortion caused by the first optical system (108) in the first optical image, second distortion correction data adapted to correct a second optical distortion caused by the second optical system (112) in the second optical image, and transformation data adapted to correct a positional misalignment between the first optical image and the second optical image, and A processor (124), the processor being configured to: processing first image data representing the first optical image based on the first distortion correction data to generate first distortion corrected image data, processing second image data representing the second optical image based on the second distortion correction data to generate second distortion corrected image data; as well as The first distortion-corrected image data and the second distortion-corrected image data are combined based on the transformation data to generate combined image data representing a combined image corresponding to the target area (242) of the sample (104); wherein the processor (124) is configured to combine the first distortion-corrected image data and the second distortion-corrected image data based on the transformation data so that one of the first optical image and the second optical image is mapped to a reference frame (KS1, KS2) defined by the other of the first optical image and the second optical image, or so that the two optical images are mapped to a common reference frame (KS3).

2. The optical imaging device (102) according to claim 1, wherein the first imaging mode is a wide field imaging mode and the second imaging mode is a confocal imaging mode.

3. The optical imaging device (102) according to claim 1, wherein the transformation data represents a positional misalignment between a first optical reference image formed by the first optical system (108) according to the first imaging mode and a second optical reference image formed by the second optical system (112) according to the second imaging mode.

4. The optical imaging device (102) according to claim 1, wherein a calibration mode is provided, in which the processor (124) is configured to generate the transformation data before forming the first optical image and the second optical image and store the transformation data in the memory (134).

5. The optical imaging device (102) of claim 3, wherein the processor (124) is configured to determine correlation data representing a correlation between the first optical reference image and the second optical reference image, and to generate the transformation data based on the correlation data.

6. The optical imaging device (102) according to claim 1, wherein the positional misalignment represented by the transformation data includes translation, rotation, scaling, shearing, mirroring and / or distortion.

7. The optical imaging device (102) according to claim 1, wherein the processor (124) is configured to update the transformation data and store the updated transformation data in the memory (134).

8. The optical imaging device (102) according to claim 7, wherein the processor (124) is configured to cause the first optical system (108) to generate a first optical image sequence according to the first imaging mode, and to cause the second optical system (112) to generate a second optical image sequence according to the second imaging mode, in, The processor (124) is configured to determine a first tracking marker (452) within one of the first sequence of optical images and to determine a second tracking marker within one of the second sequence of optical images, and The processor (124) is configured to perform tracking of the first tracking mark and the second tracking mark (452) and update the transformation data based on the tracking.

9. The optical imaging device (102) according to claim 1, wherein the first optical system (108) and the second optical system (112) are mounted in a fixed positional relationship with each other.

10. A method for imaging a sample (104) using a microscope (100), comprising the steps of: forming a first optical image corresponding to a first region (210) of the sample (104) by a first optical system (108) according to a first imaging mode, forming a second optical image corresponding to a second region (214) of the sample (104) by a second optical system (112) according to a second imaging mode, wherein the first region and the second region (214) spatially coincide with a target region (242) of the sample (104) and the first imaging mode and the second imaging mode are different from each other, obtaining first distortion correction data suitable for correcting first optical distortion caused by the first optical system (108) in the first optical image, obtaining second distortion correction data suitable for correcting second optical distortion caused by the second optical system (112) in the second optical image, obtaining transformation data suitable for correcting positional misalignment between the first optical image and the second optical image, processing first image data representing the first optical image based on the first distortion correction data to generate first distortion corrected image data; processing second image data representing the second optical image based on the second distortion correction data to generate second distortion corrected image data, and combining the first distortion-corrected image data and the second distortion-corrected image data based on the transformation data to generate combined image data representing a combined image corresponding to the target region (242) of the sample (104); The first distortion-corrected image data and the second distortion-corrected image data are combined based on the transformation data so that one of the first optical image and the second optical image is mapped to a reference system (KS1, KS2) defined by the other of the first optical image and the second optical image, or so that the two optical images are mapped to a common reference system (KS3).

11. A method for calibrating an optical imaging device (102) of a microscope (100), the optical imaging device (102) comprising: a first optical system (108) configured to form a first optical image corresponding to a first region (210) of the sample (104) according to a first imaging mode, a second optical system (112) configured to form a second optical image corresponding to a second region (214) of the sample (104) according to a second imaging mode, wherein the first region and the second region (214) spatially coincide with a target region (242) of the sample (104) and the first imaging mode and the second imaging mode are different from each other, a processor (124), and Memory (134), The method comprises the following calibration steps: obtaining first distortion correction data suitable for correcting first optical distortion caused by the first optical system (108) in the first optical image, obtaining second distortion correction data suitable for correcting second optical distortion caused by the second optical system (112) in the second optical image, obtaining transformation data suitable for correcting positional misalignment between the first optical image and the second optical image, and storing the first distortion correction data, the second distortion correction data, and the transformed data in the memory (134) so as to be accessible to the processor (124); processing first image data representing the first optical image based on the first distortion correction data to generate first distortion corrected image data, processing second image data representing the second optical image based on the second distortion correction data to generate second distortion corrected image data; as well as combining the first distortion-corrected image data and the second distortion-corrected image data based on the transformation data to generate combined image data representing a combined image corresponding to the target region (242) of the sample (104); The first distortion-corrected image data and the second distortion-corrected image data are combined based on the transformation data, so that one of the first optical image and the second optical image is mapped to a reference system (KS1, KS2) defined by the other of the first optical image and the second optical image, or so that the two optical images are mapped to a common reference system (KS3).

12. The method according to claim 11, wherein a first optical reference image of a reference object (350) is formed by the first optical system (108) according to the first imaging mode, forming a second optical reference image of the reference object according to the second imaging mode by the second optical system (112), and The transformation data is determined based on a positional misalignment of the first optical reference image and the second optical reference image.

13. The method of claim 12, wherein the reference object comprises a grid (350) formed of a plurality of points (352), the grid (350) being adapted to the first and second optical systems (112) such that each optical system (108, 112) is capable of imaging at least two of the plurality of points (352) within a usable magnification range.

14. A computer program product comprising computer program instructions for executing the method according to any one of claims 10 to 13 when the computer program instructions are run on a processor.

Images