Optical imaging system with scanning imaging device, method, system and computer program

Abstract

Embodiments of the present invention relate to an optical imaging system including a scanning imaging device, as well as a method, system, and computer program for such an optical imaging system. The method includes receiving (130) sensor data from a detector of the scanning imaging device. The sensor data includes a representation of a pattern recorded by the detector. The method further includes identifying (140) a characteristic geometric shape of the representation of the pattern. The method further includes comparing (150) the characteristic geometric shape with a reference geometric shape to identify a comparison result. The method further includes identifying (180) at least one calibration parameter for calibrating at least one drive control unit for moving a beam guide element of the scanning imaging device based on the comparison result. The method further includes operating (190) the at least one drive control unit based on the at least one calibration parameter.

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION Embodiments of the present invention relate to optical imaging systems that include scanning imaging devices, and to methods, systems, and computer programs for such optical imaging systems. [Background technology]

[0002] Confocal microscopy, an example of scanning microscopy (derived from the English word "raster scanning"), is based in many implementations on a two-dimensionally rotatable mirror. The two mirror movements are preferably precisely coordinated with one another so that the imaged area of ​​each pixel point is the same size in x and y (i.e., along the two lateral dimensions). Otherwise, the representation would contain significant distortions. Depending on the algorithm used, this distorted imaging can affect, for example, the point spread function used in the unfolding operation for processing the confocal microscope's sensor data. Distorted imaging is also referred to as anisotropic imaging. Isotropy of imaging becomes even more important when confocal imaging is to be recorded by multiple additional modalities, such as wide-field images provided by the microscope's camera sensor.

[0003] Confocal microscopes are typically calibrated using calibration standards built into the scanner. These standards are used to set the drive controls for the confocal microscope's galvanometer scanner (or "galvos" for short). Using calibration standards allows technical services to guarantee an isotropic representation. Double-cross calibration targets with no periodic features are often used here.

[0004] Such double-cross calibration targets, as often used, are positioned outside the camera's beam path and therefore cannot be represented by the camera of a confocal microscope. This represents a limitation in optical imaging systems that include not only a scanning imaging device, such as a confocal microscope, but also a camera-based imaging device, for example, to provide wide-field representation. In some such optical imaging systems, seamless switching between the scanning imaging device and the camera-based imaging device may be desirable. To this end, efforts are made to obtain knowledge of the isotropy of the imaging, as well as the identical field of view (i.e., identical imaging region, also referred to as field of view in English) of the two imaging devices and the scaling of the imaging device's image data, which may enable the generation of an appropriate optical overlap that can be overlaid on the imaging device's image data. However, without a common calibration target, it is difficult to ensure that the fields of view, i.e., the imaging regions of the two modalities, are identical.

[0005] Therefore, currently, many systems undergo a laborious manual calibration procedure, which attempts to simultaneously and isotropically set the field of view via galvo control (via the parameters x-scaling, y-scaling, x-offset, and y-offset). Quality assessment is performed solely visually (using an overlaid representation of a technical sample). Simultaneous setting and assessment of the four parameters listed above is necessary because these parameters cannot be observed independently of one another. Manual calibration is time-consuming, complex, and potentially error-prone. It is an iterative process because offset and scaling cannot be visually assessed independently of one another. Here, manual calibration checks the isotropy of the confocal scan only relative to the wide-field image, not independently (using an overlaid representation).

[0006] Possible errors in calibration can lead to representational deviations that are extremely difficult to identify and, at worst, can lead (unnoticed) to a misinterpretation of the image data. Deviations can also negatively affect subsequent alignments and their quality. For example, dyes in confocal scans can appear in different locations and, in some cases, mark altered shapes when compared to IMC (Imaging Mass Cytometry) images, thereby eliminating existing colocalizations.

[0007] There is a need for improved concepts for calibrating scanning imaging devices, especially scanning imaging devices in optical imaging systems having multiple imaging devices, for example to match the fields of view of two imaging devices. Summary of the Invention [Means for solving the problem]

[0008] This need is taken into account by the subject matter of the independent claims.

[0009] Various embodiments of the present disclosure are based on the recognition that geometric analysis of a calibration pattern is suitable for determining appropriate calibration parameters for a scanning mirror to achieve isotropy in the image data of a scanning imaging device, such as a confocal microscope. This geometric analysis can also be used, for example, to match the fields of view of the two imaging devices when applied to both the image data of the scanning imaging device and that of a separate (camera-based) optical imaging device. This avoids time-consuming manual calibration of the optical imaging system, achieving higher calibration accuracy and reducing susceptibility to errors. Furthermore, calibration can be repeated in the field, without the need for a technician, ensuring isotropy and field of view match even after a relatively long period of use.

[0010] Various aspects of the present disclosure relate to a method for an optical imaging system including a scanning imaging device. The method includes receiving sensor data from a detector of the scanning imaging device. The sensor data includes a representation of a pattern recorded by the detector. The method further includes identifying a characteristic geometric shape of the representation of the pattern. The method further includes comparing the characteristic geometric shape with a reference geometric shape to identify a comparison result. The method further includes identifying at least one calibration parameter for calibrating at least one drive control unit for moving a beam guide element of the scanning imaging device based on the comparison result. The method further includes operating the at least one drive control unit based on the at least one calibration parameter. By identifying the characteristic geometric shape and comparing it with the reference geometric shape, it is possible to easily mathematically ascertain how different the representation of the pattern is from the reference representation of the pattern (e.g., with respect to isotropy). Furthermore, if desired, it is possible to ascertain how different the field of view of the scanning imaging device is from the reference field of view. This confirmation can identify one or more suitable calibration parameters that can be used to drive and control the beam guide elements so that the scanning imaging device will generate sensor data in the future that corresponds to the definition of the reference geometry.

[0011] Basically, there are several options for obtaining the reference geometry. The reference geometry can be generated, for example, by an optical imaging system. Correspondingly, the method may include a step of identifying the reference geometry. As previously mentioned, the inventive concept is suitable, for example, for matching the field of view of a camera-based imaging device with the field of view of a scanning imaging device. In this case, for example, the camera-based optical imaging device can be used to identify the reference geometry. Accordingly, the method may further include a step of receiving another sensor data of another optical imaging sensor of the optical imaging system. In this case, the another sensor data includes another representation of the pattern recorded by the other optical imaging sensor. The method may include a step of identifying the reference geometry based on the another representation of the pattern. Correspondingly, at least one calibration parameter can be determined here so that the representation of the scanning imaging device is adapted to the representation of the other optical imaging sensor.

[0012] By comparing the characteristic geometry with the reference geometry, it can first be determined that an appropriate scaling factor is used to calibrate at least one drive control unit. When this scaling factor is applied to the representation, an adapted representation with the correct scaling can be calculated. Correspondingly, the method can include a step of adapting, e.g., scaling, the representation based on the comparison result. Accordingly, x- and y-offsets (hereinafter referred to as difference values) between the adapted representation and another representation generated by another optical imaging sensor can also be determined. The difference values ​​can also be used to shift the field of view of the scanning imaging device, thereby matching the fields of view. The method can, for example, include a step of determining a difference value between the adapted representation and the other representation, and a step of determining at least one calibration parameter based on the difference value.

[0013] Additionally or alternatively, the difference value can be obtained after applying at least one calibration parameter based on newly acquired image data of the scanning imaging device. For example, the method can include receiving new sensor data having a representation of the pattern after applying the at least one calibration parameter, determining a difference value between the newly received representation and another representation, and further determining at least one calibration parameter based on the difference value. This allows for determining and (further) reducing the offset between the fields of view of the two imaging devices.

[0014] Alternatively, the reference geometry can be determined based on sensor data of the scanning imaging device, for example, to enable automatic new calibration after extended use of the imaging system. The method can include, for example, determining the reference geometry based on sensor data of the scanning imaging device after a factory calibration of the scanning imaging device.

[0015] Alternatively, the reference geometry can be preset by the manufacturer or by another scanning imaging device, for example, a reference geometry defined by the manufacturer or a geometry determined using another scanning imaging device and stored in a memory device of the optical imaging system, thereby achieving a uniform calibration across devices.

[0016] As previously described, one possible goal is to match the field of view of the scanning imaging device with the field of view of the camera-based imaging device. Thus, for example, at least one calibration parameter can be determined such that, after applying the at least one calibration parameter, the field of view of the scanning imaging device corresponds, within a tolerance, to the field of view of another optical imaging sensor in the optical imaging system. This allows for seamless switching between image data from the scanning imaging device and the optical imaging sensor when operating the optical imaging system.

[0017] This can be achieved by matching a characteristic geometry of the scanning imaging device with a reference geometry. For example, the at least one calibration parameter can be determined after applying the at least one calibration parameter, receiving new sensor data, and identifying a characteristic geometry that corresponds, within a tolerance, to the characteristic geometry of the reference geometry. This can be used as an indication that the calibration was successful.

[0018] The accuracy of the calibration can potentially be improved by iteratively repeating the proposed procedure, whereby the difference between the characteristic geometry and the reference geometry is iteratively reduced, e.g., comparison results can be iteratively determined and at least one calibration parameter can be iteratively adapted.

[0019] In some cases, aging-related effects may prevent the desired control of the beam guiding elements to match the field of view (or achieve isotropy). In this case, an alarm can be output to signal to a user of the optical imaging system that the respective component needs to be repaired and / or replaced. For example, the method may further include preparing an alarm if, after repeated adaptation of at least one calibration parameter, the characteristic geometry does not correspond to the reference geometry within a tolerance. This can prevent operation of the optical imaging system with the erroneous component.

[0020] In the present concept, a characteristic geometric shape is calculated and compared with a reference geometric shape. One option for identifying this characteristic geometric shape is to identify the spacing between the elements of the pattern. This is again facilitated by the pattern being a periodic pattern, i.e., a pattern in which elements appear repeatedly according to a predetermined periodicity. The pattern may, for example, be a periodic pattern, and the characteristic geometric shape may include the periodicity of the representation of the periodic pattern. The periodicity is a numerical value that can, on the one hand, be easily compared with the periodicity of the reference geometric shape and, on the other hand, be used (directly) to set the scaling mentioned above. This periodicity can, for example, be calculated by calculating an autophase correlation with little computational effort and without segmenting the sensor data. Therefore, the periodicity can be determined by calculating the autophase correlation.

[0021] In scanning imaging devices, the object to be scanned is usually scanned according to a two-dimensional sampling pattern, whereby the beam guide element is correspondingly moved in two dimensions that can be calibrated separately from each other. Thus, the pattern can be a two-dimensional periodic pattern. The characteristic geometry can be a periodic representation of the two-dimensional periodic pattern in the two dimensions.

[0022] Correspondingly, the beam guiding element may also be movable in two dimensions by the drive control unit. The at least one calibration parameter may include at least a first scaling factor for scaling the movement of the beam guiding element in the first dimension and a second scaling factor for scaling the movement of the beam guiding element in the second dimension. This allows for calibration of the scanning imaging device for scanning according to a two-dimensional sampling pattern.

[0023] Calibration can occur at various times. For example, the characteristic geometry and comparison results can be performed at the beginning of start-up of the scanning imaging device. For example, an automatic calibration can be performed at the (or each) start-up of the scanning imaging device, thereby continuously ensuring measurement accuracy (and agreement with the field of view of the optical imaging sensor).

[0024] Alternatively or additionally, calibration can be performed sporadically, for example after vibrations of the optical imaging system. Thus, characteristic geometric shapes and comparison results can be determined after detecting vibrations of the optical imaging system. This is recommended, since vibrations can be expected to cause, for example, changes in the characteristics of the movement of the beam guide elements and / or changes in the field of view of the optical imaging sensor.

[0025] Additionally or alternatively, characteristic geometries and comparison results can be identified according to a pre-set schedule, again ensuring continuous measurement accuracy (and matching the field of view of the optical imaging sensor).

[0026] Some aspects of the present disclosure relate to a system for an optical imaging system including a scanning imaging device. The system includes one or more processors and one or more storage devices. The system is configured to perform the method presented above. Some aspects of the present disclosure also relate to an optical imaging system including the scanning imaging device and the system presented above.

[0027] In the present disclosure, the term scanning imaging device is used. Examples of scanning imaging devices are raster scanning electron microscopes and two-photon microscopes. However, as mentioned at the beginning, the scanning imaging device may in particular be a confocal imaging device, such as a confocal microscope (in English: Confocal Laser Scanning Microscope, CLSM, Confocal Laser Raster Scanning Microscope).

[0028] There are various locations where the pattern can be placed. For example, the pattern can be imaged onto a sample carrier, and the sample carrier can be inserted into a sample holder of an optical imaging system for calibration, or can be placed on a sample table of an optical imaging system. Alternatively, the sample table of the optical imaging system can include the pattern, allowing calibration to be performed without the user having to insert a sample carrier with the pattern.

[0029] Alternatively, the optical imaging system may include a housing, and the pattern may be disposed within the housing and outside a field of view of the scanning imaging device that is accessible to a user of the optical imaging system. This also enables automatic calibration without the cooperation of a user of the optical imaging system. In this case, the system may be configured to control the drive control unit so that the pattern is detected by a detector of the scanning imaging device outside a field of view that is accessible to a user of the optical imaging system. This allows a pattern outside the field of view of the scanning imaging device that is accessible to a user of the optical imaging system to be recorded for calibration, without the pattern interfering with normal use of the scanning imaging device.

[0030] Some aspects of the present disclosure further relate to computer programs comprising program code for performing the methods set forth above.

[0031] Some embodiments of the apparatus and / or methods will now be described in detail, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0032] [Figure 1a] FIG. 1 shows a flowchart of an embodiment of a method for an optical imaging system with a scanning imaging device. [Figure 1b] FIG. 10 is another flowchart of another example of a method for an optical imaging system with a scanning imaging device. [Figure 2a] 1 is a schematic diagram of an embodiment of an optical imaging system including a scanning imaging device. [Figure 2b] FIG. 2 is another schematic diagram of an embodiment of an optical imaging system including a scanning imaging device. [Figure 3] 1 is a flow chart of one embodiment of a calibration method flow. [Figure 4] FIG. 1 is a schematic diagram of the effect of calibration. [Figure 5] 1 is a schematic diagram of a system including an optical imaging device and a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0033] Some embodiments will be described in more detail with reference to the accompanying drawings. However, possible alternative embodiments are not limited to the features of these embodiments described in detail. Furthermore, the terms used to describe particular embodiments herein should not be considered limiting to possible alternative embodiments.

[0034] When two elements A and B are combined using "or", this should be understood to disclose all possible combinations, i.e., A only, B only, and A and B, unless clearly stated otherwise in individual cases. "At least one of A and B" or "A and / or B" may also be used as alternative expressions for the same combination. This applies equally to combinations of more than two elements. The term "and / or" includes any and all combinations of one or more of the associated listed items and may also be replaced with " / ".

[0035] Where the singular forms, e.g., the indefinite articles "ein," "eine," and the definite articles "der," "die," and "das," are used, and where the use of only one individual element is not bindingly specified, either expressly or implicitly, alternative embodiments may use multiple elements to implement the same functionality. Where a function is described below as being implemented using multiple elements, alternative embodiments may implement the same functionality using a single element or a single processing entity.

[0036] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0037] 1a and 1b show a flowchart of an embodiment of a method for an optical imaging system 200a; 200b including a scanning imaging device 220 (reference numerals for the optical imaging system relate to FIGS. 2a and 2b). The method includes a step 130 of receiving sensor data of a detector 222 of the scanning imaging device 220. The sensor data includes a representation of a pattern 10; 20; 30 (shown in FIG. 2a or 2b) recorded by the detector. The method further includes a step 140 of identifying a characteristic geometric shape of the representation of the pattern. The method further includes a step 150 of comparing the characteristic geometric shape with a reference geometric shape and identifying a comparison result. The method further includes a step 180 of identifying at least one calibration parameter for calibrating at least one drive control unit 224 for moving a beam guide element 226 of the scanning imaging device based on the comparison result. The method further includes a step 190 of operating the at least one drive control unit based on the at least one calibration parameter.

[0038] A basic first version of the method is shown in Figure 1a, which may further include optional features, which are shown as dashed blocks in Figure 1b and are described in the subsequent description of Figures 1a-2b.

[0039] The method of Figures 1a and 1b relates to an optical imaging system. Figures 2a and 2b show schematic diagrams of examples of such optical imaging systems 200a and 200b that include a scanning imaging device. Each of the optical imaging systems 200a and 200b includes a scanning imaging device 220 and a system 210 configured to implement the method of Figures 1a and / or 1b. The system 210 may be implemented as a computer system, for example, including one or more processors and one or more storage devices 216. The system 210 may optionally further include one or more interfaces 212. The one or more processors 214 are connected to the one or more storage devices and the one or more interfaces 212. The one or more processors 214 are here configured to provide the functionality of the system 210 in cooperation with one or more interfaces 212 (to exchange information with other components of the optical imaging system, such as the detector 222 of the scanning imager, the optical imaging sensor 230 (shown in FIG. 2b), or a display (not shown) of the optical imaging system) and one or more memory devices 216 (for storing and retrieving information, e.g., machine-readable instructions, including program code, for the one or more processors 214). Generally, the functionality of the one or more processors 214 can be implemented by the one or more processors 214 through execution of machine-readable instructions by the one or more processors. Correspondingly, functionality attributed to the one or more processors 214 may be defined by one or more instructions from a plurality of machine-readable instructions. The machine-readable instructions in the system 210 may be included in, for example, the one or more memory devices 216.

[0040] In the following, features of the method, system, and corresponding computer program of Figures 1a and 1b, and the optical imaging system 200a; 200b of Figures 2a and 2b, will be mainly described in relation to the present method and the present optical imaging system. It is clear that features described in relation to the present method can also be transferred to the corresponding system and computer program, because the present system is configured to perform the present method, or the computer program is an implementation of the method. Features described in relation to the optical imaging system 200; 200b also apply to the present method, system, and computer program.

[0041] The present concept relates to an optical imaging system including a scanning imaging device. These terms are used here because the present concept is applicable to many different optical imaging systems. For example, the imaging system may be a microscope system or an exoscopic system. Unlike a microscope system, an exoscopic system is an optical imaging system that is used exclusively via a display or a head-mounted display (a display worn similar to glasses). Furthermore, exoscopic systems are often used from a relatively large distance. In this case, the term optical imaging system is used to clearly indicate that, on the one hand, this is a system with optical components, and, on the other hand, this system includes other components in addition to the optical components, such as system 210. In addition to the optical components and system 210, the optical imaging system may also include other components, such as an input device (e.g., a touch-sensitive display, keyboard, or operation buttons), a display, a sample table, a housing, etc.

[0042] The optical imaging system includes at least one optical component, in this case, a scanning imaging device 220. The optical imaging system may further include another optical imaging sensor 230, as shown in Figure 2b.

[0043] A scanning imaging device is an optical component configured to generate image data by scanning (raster) multiple positions on a sample. An example of a scanning imaging device is a confocal microscope. In other words, a scanning imaging device may be a confocal imaging device, such as a confocal microscope. Other examples of scanning imaging devices are two-photon microscopes and raster scanning electron microscopes. In all cases, a single beam is deflected by a beam guiding element to the aforementioned multiple positions on the sample. One or more detectors are used to detect the interaction of the beam with the sample, e.g., reflection, photon emission (in the case of fluorescence), emitted beam, or electrons. In the case of confocal microscopy and two-photon microscopy, the beam is a laser beam, while in the case of raster scanning electron microscopy, it is an electron beam. This beam is deflected by a beam guiding element to various positions on the sample (according to a raster scan). In the case of confocal and two-photon microscopy, mirrors or other reflective elements are often used (as shown in FIG. 2a), while in the case of raster scanning electron microscopy, coils are used to deflect the electron beam. The present disclosure addresses the control of this beam guide element using a drive control unit. In the case of confocal and two-photon microscopy, the drive control unit may include, for example, a so-called galvo motor or a microelectromechanical system (MEMS) that moves the beam guide element. In the case of raster scanning electron microscopy, the drive control unit may include a control circuit for controlling the coil.

[0044] 2a, for example, shows the characteristic structure of a confocal microscope. On the left side, a laser emitter 228 is shown emitting laser light onto a mirror 226, which is moved by a drive control unit 224 to raster scan the pattern 10. In some other scanning imaging devices, instead of a single mirror, multiple mirrors are used, each controlled by a drive control unit.

[0045] The calibration of the drive control of the beam guiding elements by the drive control unit is based on an evaluation of sensor data of a detector (which detects the interaction of the beam with the sample) of the scanning imaging device, in which a characteristic geometry of a representation of the pattern is determined and compared with a reference geometry, and based on this comparison, calibration parameters for calibrating the drive control unit are determined.

[0046] In the following, a confocal microscope is used as the scanning imaging device by way of example, however, the basic principle can be correspondingly transferred to other scanning imaging devices.

[0047] The sensor data of the detector is evaluated to identify a characteristic geometric shape of the representation of the pattern. For example, image data including the representation of the pattern can be generated from the sensor data. The image data is analyzed using image processing to identify the characteristic geometric shape based on the representation of the pattern included in the image data. The characteristic geometric shape can correspond to, for example, an absolute or relative size of one or more geometric elements of the pattern in the representation of the pattern, which can be determined using image processing.

[0048] Preferably, however, the characteristic geometric shape corresponds to or includes the periodicity of the elements of the pattern. For example, the pattern may be a periodic pattern, i.e., a pattern in which one or more elements (e.g., dots, lines, triangles, squares, etc.) are repeated at regular intervals along at least one horizontal dimension of the pattern. As shown in FIG. 3, the pattern (here, calibration standard 310) may be, for example, a two-dimensional periodic pattern, e.g., a two-dimensional periodic pattern of dots. Thus, the characteristic geometric shape may be the periodicity of the representation of the two-dimensional periodic pattern in two dimensions. The periodicity corresponds to the spacing between two adjacent elements of the repeated representation of the pattern, e.g., the spacing between the centers of two horizontally or vertically adjacent dots in the pattern shown in FIG. 3. Such periodicity can be calculated, for example, by calculating an autophase correlation. Correspondingly, the method may include a step 145 of calculating the autophase correlation, as shown in FIG. 1b. To calculate the autophase correlation, a 2D Fourier transform is applied to the image data, and the cross power spectrum between a 2D Fourier transformed version of the image data and a 2D Fourier transformed version of the image data (i.e., the autophase correlation between the same 2D Fourier transformed versions of the image data) is calculated. This result is then inverted using an inverse Fourier transform. Periodicity can be determined by calculating the spacing between peaks in the inverted version.

[0049] These characteristic geometries (e.g., determined periodicities in one or two lateral dimensions) are compared with a reference geometry, which may likewise correspond to the absolute or relative size of one or more geometric elements of the pattern, or preferably to a reference periodicity in one or two lateral dimensions, to determine a comparison result.

[0050] The reference geometry may come from a variety of sources. For example, the reference geometry may be a reference geometry defined by the manufacturer or a geometry determined using another scanning imaging device, and this geometry is stored in the memory device 216 of the optical imaging system (i.e., in the memory device 216 of one or more memory devices of the system). This geometry may be pre-defined by the manufacturer for many optical imaging systems, or may be created for a population of similar optical imaging systems by one optical imaging system in the optical imaging system user's laboratory.

[0051] Alternatively, the reference geometry can be determined using an optical imaging system. Accordingly, the method may include step 120 of identifying the reference geometry, as further shown in FIG. 1b. For example, a scanning imaging device can be manually calibrated and, in the calibrated state, can be used to identify the reference geometry. In other words, the method may include step 120 of identifying the reference geometry based on sensor data of the scanning imaging device after factory calibration of the scanning imaging device. To this end, as described above, a characteristic geometry can be identified and subsequently used as the reference geometry. For this purpose, the reference geometry can be stored in memory 216 of the optical imaging system.

[0052] As already explained above, this concept can be used in particular not only to ensure the isotropy of the image data that can be obtained from the sensor data of a scanning imaging device, but also to adapt the field of view of a scanning imaging device to the field of view of another imaging device (of another optical imaging sensor 230 in Figure 2b).

[0053] FIG. 2b shows an optical imaging system including a first scanning imaging device (confocal microscope 220) and a second optical imaging device (formed by another optical imaging sensor 230). To achieve intuitive operation of the optical imaging system with two imaging devices, it may be desirable for the fields of view of the two imaging devices to correspond to each other. This is made possible by the proposed concept by defining a reference geometry using the sensor data of the second optical imaging device. Here, the sensor data of the optical imaging sensor 230 is used for this purpose. This is because the field of view of the optical imaging sensor cannot be corrected to the field of view of the scanning imaging device without compromising image quality, but the reverse is possible. Furthermore, ensuring isotropy is a relatively small issue for the sensor / image data of the optical imaging sensor compared to the sensor data of the scanning imaging device.

[0054] 1b, the method may further include a step 110 of receiving another sensor data from another optical imaging sensor 230 of the optical imaging system. Here, the another sensor data includes another representation of the pattern recorded by the other optical imaging sensor. Based on this another sensor data, which may correspond to image data, a characteristic geometric shape of the another representation of the pattern may be similarly identified (as previously described), used as a reference geometric shape, and optionally stored. Correspondingly, the method may include a step 120 of identifying a reference geometric shape based on the another representation of the pattern. A detailed example of this is discussed, for example, in connection with FIGS. 3 and 4.

[0055] The identification of the characteristic or reference geometric shapes is based on the recording of the pattern by the respective imaging devices. The pattern can be located at different locations in the optical imaging system. Two locations are shown in FIG. 2b by way of example. To make the pattern "visible" by the two imaging devices, the pattern 30 can be imaged, for example, on a sample carrier. The sample carrier itself can be inserted into a sample holder (not shown) of the optical imaging system for calibration, or it can be placed on a sample table 250 of the optical imaging system. Alternatively, the sample table 250 can include the pattern 30, i.e., the pattern can be printed or glued on the sample table.

[0056] Alternatively, for example, if no reference information is to be determined by the separate optical imaging sensor 230, the pattern 20 may be located within the housing 240 and outside a field of view available to the user, as further shown in FIG. 2b. To this end, the beam guide element may be tilted further for calibration, thereby increasing the field of view of the scanning imaging device 220 during calibration. The method may include controlling the drive control unit to cause the pattern to be detected by a detector of the scanning imaging device outside a field of view accessible to a user of the optical imaging system.

[0057] Once the characteristic geometry and, optionally, the reference geometry have been determined, a step 150 of comparing the characteristic geometry with the reference geometry can be performed, thereby determining a comparison result. For example, a relationship, e.g., a periodicity relationship, between the characteristic geometry and the reference geometry can be formed. This relationship can be determined separately for, for example, two lateral dimensions. The comparison result can thus include a relationship between the characteristic geometry and the reference geometry (in two dimensions). This relationship can again be used to determine at least one calibration parameter.

[0058] The use of the relationship between the geometries is due to the nature of the movement of the beam guiding element. One essential aspect of the calibration is the scaling of the movement, i.e., how much the mirror moves for a given input value. The beam guiding element can therefore be moved by the drive control unit, for example, in two dimensions. The at least one calibration parameter may include at least one first scaling factor for scaling the movement of the beam guiding element in a first (lateral) dimension and a second scaling factor for scaling the movement of the beam guiding element in a second (lateral) dimension orthogonal to the first dimension. These scaling factors determine how much the mirror moves for a given input value. The reference geometry is the target of the calibration, i.e., if the characteristic geometry corresponds to the reference geometry, no further calibration is required, so the relationship between the characteristic geometry and the reference geometry can determine how the scaling should be adapted, respectively.

[0059] If the purpose of the calibration is to match the field of view of a scanning imaging device with that of another optical imaging sensor, attention must be paid not only to the scaling factor but also to the lateral offset between the fields of view. This can be seen, for example, in FIG. 4, where the elements of the pattern representation 410; 420 not only have different scaling but are also offset laterally in two dimensions. This (two-dimensional) offset can be calculated, on the one hand, by adapting (i.e., scaling) the pattern representation according to the determined scaling factor and determining the offset based on the adapted version. For example, the method can further include a step 160 of adapting the representation based on the comparison result, as shown in FIG. 1b, and a step 165 of determining a difference value (e.g., lateral offset) between the adapted representation and the other representation. On the other hand, the offset can be determined by applying the scaling parameters, receiving new sensor data of the optical imaging device with a new representation of the pattern, and comparing the characteristic geometry of the new representation with a reference geometry. In other words, the method may include receiving 130 new sensor data having a representation of the pattern after applying at least one calibration parameter, and determining 165 a difference value between the newly received representation and another representation. The at least one calibration parameter may further be determined based on the difference value (step 180).

[0060] At least one calibration parameter, including scaling factors and offsets in the two lateral dimensions, is ideally specified so that after calibration, the (newly determined) characteristic geometry corresponds to the reference geometry. In other words, the at least one calibration parameter can be determined so that after applying the at least one calibration parameter, receiving new sensor data, and specifying the characteristic geometry, the characteristic geometry corresponds to the reference geometry within a suitable tolerance. If this is the case, and especially when offsets are also taken into account, the field of view of the scanning imaging device and the field of view of the other optical imaging sensor should match. In other words, the at least one calibration parameter is specified so that after applying the at least one calibration parameter, the field of view of the scanning imaging device corresponds to the field of view of the other optical imaging sensor of the optical imaging system within a suitable tolerance. The respective tolerances depend on the application and the selected reference quantity.

[0061] In some cases, at least one calibration parameter may not be determined in a single pass so that the result falls within the aforementioned tolerance range. This may be due, for example, to measurement accuracy and nonlinearity in controlling the movement of the beam guide element. Therefore, at least one calibration parameter may be iteratively adapted until the desired accuracy is reached. In other words, a comparison result may be iteratively determined (step 150) (by receiving new sensor data, identifying a characteristic geometry, and comparing it with a reference geometry), and at least one calibration parameter may be iteratively adapted (step 185) (based on the adapted comparison result). However, such an iterative method may not always be successful, for example, if the fields of view are excessively different or if the galvo motor operates inaccurately due to increased aging. In this case, as shown in FIG. 1b, the method may further include step 170 of preparing an alarm if, after iteratively adapting at least one calibration parameter, the characteristic geometry does not correspond to the reference geometry within the tolerance range.

[0062] However, if at least one calibration parameter is required to ensure a sufficiently accurate calibration, the at least one calibration parameter may be utilized during operation.

[0063] The calibration described herein can be used in many ways. In particular, if the user of the optical imaging system does not need to manually insert a pattern, the calibration can be performed automatically and regularly, for example, according to a preset schedule or upon start-up of the scanning imaging device (or each). The system can trigger the calibration at start-up or according to a schedule, respectively. If a pattern is attached to a sample carrier, the user may need to insert the sample carrier or place it on the sample table. Alternatively or in addition to regular calibration, calibration can also be performed as needed. This need exists, in particular, when a new optical imaging sensor 230 is attached or its position is changed, or when the optical imaging system is transported or shaken by a collision. In this case, the vibration can be detected by the system 210, and the system can perform a calibration in response to the vibration.

[0064] In the proposed optical imaging system, an optical imaging sensor 230, also referred to as a camera in some imaging systems, is used. Correspondingly, the optical imaging sensor can be configured to generate separate sensor data, which is imaging sensor data. For example, one optical imaging sensor of a stereoscopic imaging device can include or correspond to an APS (Active Pixel Sensor)-based or CCD (Charge-Coupled Device)-based image sensor. In an APS-based image sensor, for example, light at each pixel is detected using a photodetector and a pixel active amplifier. APS-based image sensors are often based on CMOS (Complementary Metal-Oxide-Semiconductor) technology or S-CMOS (Scientific CMOS) technology. In a CCD-based image sensor, photons arriving at a semiconductor-oxide interface are converted into electron charges, which are then transferred between capacitance bins (sinks) in the image sensor by the image sensor's circuitry for imaging.

[0065] The system 210 can be configured to receive (i.e., receive or read) sensor data of the detector 222 from the detector of the scanning imaging device 220 and / or to receive (i.e., receive or read) other sensor data of the optical imaging sensor 230 from the optical imaging sensor 230, for example via the interface 212.

[0066] The one or more interfaces 212 of the system 210 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may exist as digital (bit) values ​​according to a particular code within a module, between modules, or between modules of different units. The one or more interfaces 212 may include, for example, interface circuitry configured for receiving and / or transmitting information.

[0067] The one or more processors 214 of the system 210 can be implemented by one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer, or a programmable hardware component operable by correspondingly adapted software. In other words, the described functionality of the one or more processors 214 can also be implemented in software, in which case the software is executed in one or more programmable hardware components. Such hardware components may include general-purpose processors (e.g., central processing units), digital signal processors (DSPs), microcontrollers, etc.

[0068] The one or more storage devices 216 of the system 210 may include at least one element from the group of computer-readable media, e.g., magnetic or optical storage media, such as a hard disk drive, flash memory, floppy disk, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPPROM), or network storage.

[0069] Further details and aspects of the present methods, systems, corresponding computer programs, present optical imaging systems, and scanning imaging devices are described in connection with the concepts and embodiments described above or below (e.g., in connection with FIGS. 3-5). The present methods, systems, computer programs, present optical imaging systems, and scanning imaging devices may include one or more optional features corresponding to one or more aspects of the proposed concepts or described embodiments, as described above or below.

[0070] Various aspects of the present disclosure relate to an apparatus for automatically calibrating the field of view of an optical imaging device or system, such as a microscope or microscope system, using image processing. Some embodiments specifically address the automatic calibration of the fields of view of multiple imaging devices.

[0071] For this purpose, a calibration target common to both modalities (e.g., the pattern used in connection with FIGS. 1a-2b) is used. The calibration target may be designed, for example, as a sample slide. For example, a calibration target with a periodic pattern (in the x and y dimensions) can be used. This periodicity can be determined, for example, via autophase correlation. The periodicity can correspond to the characteristic geometric shapes discussed in connection with FIGS. 1a-2b or represent an aspect of the characteristic geometric shapes. The advantage of using autophase correlation is its low technical cost. In some cases, segmentation can be omitted. Furthermore, it is less susceptible to noise compared to conventional targets and algorithms.

[0072] The same target can then be recorded in widefield and confocal (by a camera-based imaging device, such as another optical imaging sensor 230 in FIG. 2b). The deviation between the confocal and widefield image regions can be determined. For example, the widefield image can be used as a reference for the scaling to be achieved by the confocal scanner (with respect to scaling in the x and y dimensions), since the camera sensor (e.g., a CCD sensor) cannot be altered.

[0073] This allows the confocal image to be confined to the same image area (of the wide-field camera), while at the same time ensuring isotropy of the recording, or alternatively set to be at least identical to the wide-field sensor.

[0074] A flow chart of an example of the flow of a calibration procedure is shown in Figure 3. In this example of Figure 3, first, a calibration standard 310 (which may be, for example, a periodic point grid, such as the pattern discussed in connection with Figures 1a-2b) is inserted. Next, a wide-field recording is made, which is used as reference imaging 320. The microscope modality is then switched to confocal, with the calibration standard 310 still inserted. A further recording of the slide is made by a confocal scanner to obtain confocal imaging (also called CLSM imaging, where CLSM stands for Confocal Laser Scanning Microscope) 350.

[0075] The periodicity of the samples in the x and y dimensions is determined for the two recordings 320 and 350 using autophase correlation 330 and 360 and other morphological operations, respectively. In the following, the wide-field recording is used as the reference / target value 340, since the camera does not provide this type of parameterization and therefore cannot be changed. Due to the deviation in period in the confocal image compared to the wide-field image, the relevant settings of the galvo control can be adapted (step 370). For example, the x offset, y offset, x scaling, and y scaling are also calculated here. By adapting these values, the mirror movement is changed so that the calibration standard is imaged as well as possible confocally and in the wide-field identically. Consequently, the image area and scaling are set identical to those of the reference recording.

[0076] For verification purposes, after the values ​​have been set, a new confocal image is recorded and processed in the same manner, with deviations from the wide-field image only permitted within a predetermined tolerance range. Optionally, a further iteration step 380 can now be performed based on the confocal image.

[0077] Alternatively, when using the proposed concept, for example to compensate for aging phenomena or to set the image area anew when switching between different stands, it is also possible to use older confocal recordings to determine the reference 340.

[0078] The effect of calibration is shown diagrammatically in Figure 4, which shows, on the one hand, a reference recording 410 of the calibration standard and, on the other hand, a confocal recording 420. It can be seen in the original overlap 430 before calibration that the confocal recording has a relatively large scaling in the x-dimension and very little scaling in the y-dimension. Furthermore, an offset is present. After correcting the image area, an exact overlap 440 of the representation of the calibration standard is achieved.

[0079] The proposed method can further be used to test the focus scanner settings, e.g., isotropy, possible distortions caused by different scanning speeds, recording errors, etc. (isotropy should be maintained at different scanning speeds) based on the recorded images using periodicity.

[0080] This calibration can be performed automatically or can be used as an evaluation of the current settings and as support for manual adjustments. Compared to manual calibration, the added value comes from the independent evaluation of scaling and offset values. This can improve quality while saving significant time and reducing calibration costs in the production of optical imaging systems. Compared to purely visual manual evaluation, calibration validation can also be automated and quantified.

[0081] Here, we describe an embodiment of the present invention that applies the concept to microscope systems, including confocal and wide-field microscopes, but the concept is also applicable to other optical imaging systems, including other scanning imaging devices, such as scanning imaging devices and camera-based imaging devices.

[0082] Reference will now be made to further details and aspects of an apparatus for automatically calibrating a field of view in connection with concepts or embodiments described previously (e.g., FIGS. 1a-2b). The apparatus for automatically calibrating a field of view may include one or more additional, optional features corresponding to one or more aspects of the proposed concepts or described embodiments, as described previously or hereinafter.

[0083] Some embodiments relate to an optical imaging system or optical imaging device, such as a microscope system or microscope, including a system such as that described in connection with one or more of FIGS. 1-4. Alternatively, the optical imaging system, e.g., a microscope, may be part of a system such as that described in connection with one or more of FIGS. 1-4 or may be connected to a system such as that described in connection with one or more of FIGS. 1-4. FIG. 5 shows a schematic diagram of a system 500 configured to perform the methods described herein. The system 500 includes an optical imaging device 510, e.g., a microscope (e.g., a scanning microscope or a non-scanning microscope), and a computer system 520. The optical imaging device 510 is configured to capture images and is connected to the computer system 520. The computer system 520 is configured to perform at least a portion of the methods described herein. The computer system 520 may be configured to execute a machine learning algorithm. The computer system 520 and the optical imaging device 510 may be separate entities or may be integrated into a common housing. The computer system 520 may be part of the central processing system of the optical imaging device 510, and / or the computer system 520 may be part of a subordinate component of the optical imaging device 510, such as a sensor, actor, camera, or lighting unit of the optical imaging device 510.

[0084] The computer system 520 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed across various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 520 may include any circuit or combination of circuits. In one embodiment, the computer system 520 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuitry that may be included in computer system 520 may be custom circuitry, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 520 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.Computer system 520 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touch screen, voice recognition device, or any other device that allows a user of the system to input information to and receive information from computer system 520.

[0085] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry. In some embodiments, any one or more of the critical steps may be performed by such an apparatus.

[0086] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation may be performed by a non-transitory storage medium, such as a digital storage medium, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Therefore, the digital storage medium may be computer-readable.

[0087] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

[0088] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code that is operative to perform any of the methods when the computer program product is run on a computer, and that may be stored, for example, on a machine-readable carrier.

[0089] Further embodiments comprise the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0090] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein when the computer program runs on a computer.

[0091] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a recording medium.

[0092] A further embodiment of the present invention is, therefore, a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0093] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0094] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0095] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0096] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus. [Explanation of symbols]

[0097] 10;20;30 pattern 110 Receiving other sensor data 120 Step of identifying reference geometry 130 Receiving sensor data 140 Steps to identify characteristic geometric shapes 150. Comparing the characteristic geometry with the reference geometry 160 Steps to Match Pattern Expressions 165 Steps to identify differential values 170 Steps to Prepare an Alarm 180 identifying at least one calibration parameter 185. Adapting at least one calibration parameter 190 Activating at least one drive control 200a;200b Optical Imaging System 210 System 212 one or more interfaces 214 One or more interfaces 216 One or more storage devices 220 Scanning Imaging Device 222 detector 224 Drive Control Unit 226 Light guide element, mirror 228 Laser Emitter 230 Another Optical Imaging Sensor 240 Housing 250 Sample Table 310 Calibration Standards 320 Reference Imaging 330 Autophase Correlation 340 Standards 350 Confocal Imaging 360 Autophase Correlation 370 Steps to Adapt Galvo Control 380 Further iterative steps 410 Standard Record 420 Confocal Recording 430 Original overlap 440 Overlap after calibration 500 Systems 510 Optical Imaging Device 520 Computer Systems

Claims

1. A method for an optical imaging system (200a; 200b; 500) comprising a scanning imaging device (220; 510), said method comprising: receiving (130) sensor data of a detector (222) of said scanning imaging device (220), said sensor data comprising a representation of a pattern (10; 20; 30) recorded by said detector; Identifying (140) characteristic geometric shapes of said representation of said pattern; comparing (150) the characteristic geometric shape with a reference geometric shape to determine a comparison result; identifying (180) at least one calibration parameter for calibrating at least one drive control unit (224) for moving a beam guide element (226) of the scanning imaging device based on the comparison; activating (190) at least one of said drive control units based on at least one of said calibration parameters; A method comprising:

2. The method comprises: receiving (110) another sensor data of another optical imaging sensor (230) of the optical imaging system, the another sensor data comprising another representation of the pattern recorded by the another optical imaging sensor; Identifying (120) the reference geometric shape based on the alternative representation of the pattern; further comprising: The method of claim 1.

3. The method comprises: Adapting (160) the representation based on the comparison; determining (165) a difference value between the adapted representation and the alternative representation; further determining (180) at least one of the calibration parameters based on the difference value; further comprising: The method of claim 2.

4. The method comprises: receiving (130) the sensor data having the representation of the pattern after applying at least one of the calibration parameters; determining (165) a difference value between the newly received representation and the other representation; further determining (180) at least one of the calibration parameters based on the difference value; Including, 4. The method according to claim 2 or 3.

5. the reference geometry is a reference geometry defined by a manufacturer or a geometry determined using another scanning imaging device and stored in a storage device (216) of the optical imaging system; 5. The method according to any one of claims 1 to 4.

6. the at least one calibration parameter is determined such that, after applying the at least one calibration parameter, a field of view of the scanning imaging device corresponds, within a tolerance, to a field of view of another optical imaging sensor of the optical imaging system.

6. The method according to any one of claims 1 to 5.

7. and determining the at least one calibration parameter such that, after applying the at least one calibration parameter, receiving the new sensor data, and identifying the characteristic geometry, the characteristic geometry corresponds to the reference geometry within a tolerance.

7. The method according to any one of claims 1 to 6.

8. The comparison results are iteratively determined (150) and at least one of the calibration parameters is iteratively adapted (185).

8. The method according to any one of claims 1 to 7.

9. The method further comprises preparing (170) an alarm if, after iteratively adapting at least one of the calibration parameters, the characteristic geometry does not correspond to the reference geometry within a tolerance.

9. The method of claim 8.

10. the pattern is a periodic pattern and the characteristic geometric shape comprises a periodicity of the representation of the periodic pattern; 10. The method according to any one of claims 1 to 9.

11. The periodicity is determined by calculating (145) an autophase correlation. The method of claim 10.

12. the pattern is a two-dimensional periodic pattern and the characteristic geometric shape is a periodicity of the representation of the two-dimensional periodic pattern in two dimensions; 12. The method according to any one of claims 1 to 11.

13. the beam guiding element is movable by the drive control unit in two dimensions, and the at least one calibration parameter comprises at least a first scaling factor for scaling the movement of the beam guiding element in a first dimension and a second scaling factor for scaling the movement of the beam guiding element in a second dimension.

13. The method according to any one of claims 1 to 12.

14. the characteristic geometric shape and the comparison result are identified upon start-up of the scanning imaging device after detecting vibrations in the optical imaging system and / or according to a predetermined schedule.

14. The method according to any one of claims 1 to 13.

15. A system (210; 520) for an optical imaging system (200; 500) with a scanning imaging device (220; 510), comprising: The system includes one or more processors (214) and one or more storage devices (216), the system being configured to implement the method of any one of claims 1 to 14. System (210; 520).

Images