Optical imaging system comprising a scanning imaging device, method, system, and computer program

Abstract

Embodiments of the invention relate to an optical imaging system comprising a scanning imaging device, to a method, to a system, and to computer program for such an optical imaging system. The method involves obtaining (130) sensor data of a detector of the scanning imaging device, said sensor data comprising a representation of a pattern which has been captured by the detector, determining (140) a characteristic geometry of the representation of the pattern, comparing (150) the characteristic geometry with a reference geometry in order to determine a comparison result, determining (180) at least one calibration parameter in order to calibrate at least one control unit for moving a beam-conducting element of the scanning imaging device on the basis of the comparison result, and operating (190) the at least one control unit on the basis of the at least one calibration parameter.

Description

[0001] Optical imaging system with a scanning imaging device, method, system and computer program

[0002] Technical area

[0003] Embodiments of the present invention relate to an optical imaging system with a scanning imaging device, as well as to a method, a system and a computer program for such an optical imaging system.

[0004] background

[0005] Confocal microscopy technology, which is an example of scanning microscopy, is based in many implementations on a two-dimensionally pivoting mirror. The two mirror movements are preferably precisely coordinated so that the imaged area of ​​a single pixel is equal in x and y (i.e., along two lateral dimensions), otherwise the image contains severe distortions. Depending on the algorithm used, such a distorted image can, for example, influence a point spread function used in a deconvolution operation to process the confocal microscope's sensor data. A distorted image is also called a non-isotropic image.The isotropy of the image becomes even more important when confocal imaging is to be registered with other modalities, such as a wide-field image provided by a microscope camera sensor.

[0006] Generally, a calibration standard built into the scanner is used to calibrate confocal microscopes. This calibration standard is used to adjust the control of the confocal microscope's galvanometer scanners (galvos for short). By using the calibration standard, technical service can ensure an isotropic image. A double-cross calibration target, which does not exhibit periodic properties, is typically used for this purpose.

[0007] Such a double-cross calibration target, as is usually used, cannot be imaged using a confocal microscope camera because it is located outside the camera's beam path. This represents a limitation in optical imaging systems that, in addition to a scanning imaging device such as a confocal microscope, also have a camera-based imaging device, for example to provide a wide-field image. In some of these optical imaging systems, it may be desirable to be able to switch seamlessly between the scanning imaging device and the camera-based imaging device. To achieve this, in addition to the isotropy of the images, an identical field of view (i.e. an identical imaged area) of the two imaging devices is desired, as is knowledge of the scaling of the image data of the imaging device so that, if necessary,a suitable optical overlay can be generated, which is superimposed on the image data of the imaging device. However, without a common calibration target, it is difficult to ensure that the field of view, i.e., the imaged area of ​​the two modalities, is identical.

[0008] Currently, some systems in production require a laborious manual calibration procedure to attempt to set the field of view identically and isotropically using the galvo controller (using the parameters scaling in the x-dimension, scaling in the x-dimension, offset in the x-dimension, and offset in the y-dimension). Quality assessment is performed solely visually (using superimposed images of a technical sample). Simultaneous adjustment and assessment of the four aforementioned parameters is necessary because the parameters cannot be viewed independently of one another. Manual calibration is time-consuming, complicated, and potentially error-prone. It is an iterative process because offset and scaling cannot be assessed independently of one another visually.The manual calibration checks the isotropy of the confocal scan only relative to the wide-field image and not independently of it (using the overlay display).

[0009] Possible calibration errors lead to display deviations that are difficult to identify, which - if unnoticed - can, in the worst case, lead to incorrect interpretation of the image data. Deviations can also negatively impact subsequent alignment and its quality. For example, a confocally scanned dye may appear in a different location compared to an IMC (imaging mass cytometry) image and may cover a different shape, thus ruling out existing colocalization. There is a need for an improved concept for calibrating scanning imaging devices, particularly scanning imaging devices in optical imaging systems with multiple imaging devices, for example to harmonize the field of view of the two imaging devices.

[0010] Summary

[0011] This need is taken into account by the subject matter of the independent claims.

[0012] Various embodiments of the present disclosure are based on the finding that a geometric analysis of a calibration pattern is suitable for achieving isotropy in image data from a scanning imaging device, such as a confocal microscope, by determining suitable calibration parameters for the scanner mirror(s). Furthermore, this geometric analysis, for example when applied to both image data from the scanning imaging device and image data from another optical (camera-based) imaging device, can also be used to harmonize the fields of view of the two imaging devices. This avoids the time-consuming manual calibration of the optical imaging system, and achieves greater calibration accuracy and a lower susceptibility to errors.In addition, calibration can be repeated in the field without the need for a technician, so that isotropy and consistency of the fields of view can be guaranteed even after extended periods of use.

[0013] Various aspects of the present disclosure relate to a method for an optical imaging system having a scanning imaging device. The method comprises obtaining sensor data from a detector of the scanning imaging device. The sensor data comprise a representation of a pattern recorded by the detector. The method further comprises determining a characteristic geometry of the representation of the pattern. The method further comprises comparing the characteristic geometry with a reference geometry to determine a comparison result. The method further comprises determining at least one calibration parameter for calibrating at least one control unit for moving a beam-guiding element of the scanning imaging device based on the comparison result.The method further comprises operating the at least one control unit based on the at least one calibration parameter. By determining the characteristic geometry and comparing it with the reference geometry, it is possible to mathematically easily determine the extent to which the representation of the pattern differs from a reference representation of the pattern (for example, with regard to isotropy). Furthermore, if desired, it is possible to determine the extent to which the field of view of the scanning imaging device differs from a reference field of view. These determinations can be used to determine one or more suitable calibration parameters that can be used to control the beam-guiding element such that the scanning imaging device generates sensor data corresponding to the specifications of the reference geometry in the future.

[0014] In principle, there are several ways to obtain the reference geometry. For example, the reference geometry can be generated by the optical imaging system. Accordingly, the method can also comprise determining the reference geometry. As previously described, the present concept is suitable, for example, for harmonizing the fields of view of a camera-based imaging device and a scanning imaging device. In this case, the camera-based optical imaging device can be used, for example, to determine the reference geometry. The method can further comprise obtaining further sensor data from a further optical imaging sensor of the optical imaging system. The further sensor data comprises a further representation of the pattern recorded by the further optical imaging sensor.The method may include determining the reference geometry based on the further representation of the pattern. Accordingly, the at least one calibration parameter may be determined such that the representation of the scanning imaging device is adapted to the representation of the further optical imaging sensor.

[0015] By comparing the characteristic geometry with the reference geometry, it can primarily be determined that suitable scaling factors are used to calibrate at least one control unit. If these scaling factors are applied to the representation, an adapted representation with correct scaling can be calculated. Accordingly, the method can include adapting, for example scaling, the representation based on the comparison result. The x- and y-offset (hereinafter referred to as the difference value) between the adapted representation and the further representation generated by the additional optical imaging sensor can then also be determined. Using the difference value, the field of view of the scanning imaging device can now also be shifted so that the fields of view can be aligned.For example, the method may include determining a difference value between the adjusted representation and the further representation, and determining the at least one calibration parameter further based on the difference value.

[0016] Additionally, or alternatively, this difference value can also be obtained after applying the at least one calibration parameter based on newly acquired image data from the scanning imaging device. For example, after applying the at least one calibration parameter, the method can comprise re-obtaining the sensor data with the representation of the pattern, determining a difference value between the newly obtained representation and the further representation, and determining the at least one calibration parameter further based on the difference value. This allows the offset between the fields of view of the two imaging devices to be determined and (further) reduced.

[0017] Alternatively, the reference geometry can be determined based on sensor data from the scanning imaging device, for example, to enable automated recalibration after extended use of the imaging system. For example, the method can include determining the reference geometry based on sensor data from the scanning imaging device after a factory calibration of the scanning imaging device.

[0018] Alternatively, the reference geometry can be specified at the factory or by another scanning imaging device. For example, the reference geometry can be a factory-defined reference geometry or a geometry determined by another scanning imaging device and stored in a memory of the optical imaging system. This allows for consistent calibration across devices.

[0019] As previously stated, one possible goal is to align the field of view of the scanning imaging device with the field of view of a camera-based imaging device. For example, the at least one calibration parameter can be determined such that, after applying the at least one calibration parameter, a field of view of the scanning imaging device corresponds to a field of view of another optical imaging sensor of the optical imaging system within a tolerance range. This allows for seamless switching between image data from the scanning imaging device and image data from the optical imaging sensor during operation of the optical imaging system.

[0020] This can be achieved by aligning the characteristic geometry of the scanning imaging device with the reference geometry. For example, the at least one calibration parameter can be determined such that, after applying the at least one calibration parameter and reobtaining the sensor data and determining the characteristic geometry, the characteristic geometry corresponds to the reference geometry within a tolerance range. This can be used as an indicator that the calibration was performed successfully.

[0021] The accuracy of the calibration can be increased if necessary by iteratively repeating the proposed procedure, so that the difference between the characteristic geometry and the reference geometry is iteratively reduced. For example, the comparison result can be determined repeatedly and at least one calibration parameter can be repeatedly adjusted.

[0022] In some cases, it may happen that the beam-guiding element cannot be controlled as desired due to aging-related effects in order to align the fields of view (or achieve isotropy). In these cases, a warning can be issued to signal to the user of the optical imaging system that repair and / or replacement of the respective component is necessary. For example, the method can further comprise providing a warning if the characteristic geometry does not correspond to the reference geometry within a tolerance range after repeated adjustment of the at least one calibration parameter. This can prevent operation of the optical imaging system with a faulty component.

[0023] In the present concept, a characteristic geometry is calculated and compared with a reference geometry. One way to determine this characteristic geometry is to determine the distances between elements of the pattern. This is made easier by the fact that the pattern is a periodic pattern, i.e., a pattern in which elements repeatedly occur according to a predetermined periodicity. For example, the pattern can be a periodic pattern and the characteristic geometry can include a periodicity of the representation of the periodic pattern. The periodicity is a numerical value that can, on the one hand, be easily compared with a periodicity of the reference geometry, and, on the other hand, can be used (directly) to set the aforementioned scaling.For example, the periodicity can be calculated with little computational effort and without segmenting the sensor data by calculating an auto-phase correlation. Consequently, the periodicity can be determined by calculating an auto-phase correlation.

[0024] In scanning imaging devices, an object to be scanned is usually scanned according to a two-dimensional scanning pattern. The beam-guiding element is moved accordingly in two dimensions, which can be calibrated separately. Consequently, the pattern can be a two-dimensional periodic pattern. The characteristic geometry can be a periodicity of the representation of the two-dimensional periodic pattern in two dimensions.

[0025] Accordingly, the beam-guiding element can also be movable in two dimensions by the control unit. The at least one calibration parameter can comprise at least a first scaling factor for scaling the movement of the beam-guiding element in a first dimension and a second scaling factor for scaling the movement of the beam-guiding element in a second dimension. This allows the scanning imaging device to be calibrated for scanning according to the two-dimensional scanning pattern.

[0026] Calibration can be performed at various times. For example, the characteristic geometry and comparison results can be performed upon startup of the scanning imaging device. For example, automatic calibration can be performed at (or every) startup of the scanning imaging device, ensuring continuous measurement accuracy (and compliance with the field of view of the optical imaging sensor).

[0027] Alternatively or additionally, calibration can be performed sporadically, for example, after a vibration of the optical imaging system. Consequently, the characteristic geometry and the comparison result can be determined after detecting a vibration of the optical imaging system. This is recommended because the vibration is expected to change the characteristic movement of the beam-guiding element and / or the field of view of the optical imaging sensor.

[0028] Additionally, or alternatively, the characteristic geometry and the comparison result can be determined according to a predefined schedule. Here, too, measurement accuracy (and compliance with the field of view of the optical imaging sensor) can be continuously ensured.

[0029] Some aspects of the present disclosure relate to a system for an optical imaging system with a scanning imaging device. The system comprises one or more processors and one or more memory devices. The system is configured to perform the method presented above. Some aspects of the present disclosure further relate to an optical imaging system comprising a scanning imaging device and the system presented above.

[0030] In the present disclosure, the term "scanning imaging device" is used. Examples of scanning imaging devices include scanning electron microscopes and two-photon microscopes. In particular, however, as mentioned above, the scanning imaging device can be a confocal imaging device, such as a confocal microscope (confocal laser scanning microscope, CLSM).

[0031] There are various locations where the pattern can be placed. For example, the pattern can be imaged on a sample carrier, which can be inserted into a sample holder of the optical imaging system for calibration, or it can be placed on a sample stage of the optical imaging system. Alternatively, the sample stage of the optical imaging system can enclose the pattern. This allows calibration to be performed without requiring a user to insert a sample carrier containing the pattern.

[0032] Alternatively, the optical imaging system may comprise a housing, wherein the pattern is arranged within the housing and outside a field of view of the scanning imaging device accessible to users of the optical imaging system. This also enables automatic calibration without the involvement of a user of the optical imaging system. In this case, the system may be configured to control the control unit such that the pattern outside the field of view accessible to users of the optical imaging system is detected by the detector of the scanning imaging device. This allows the pattern to be recorded for calibration outside the field of view of the scanning imaging device accessible to users of the optical imaging system without interfering with normal use of the scanning imaging device.

[0033] Some aspects of the present disclosure also relate to a computer program having a program code for carrying out the method presented above.

[0034] Short character description

[0035] Some examples of devices and / or methods are explained in more detail below with reference to the accompanying figures. They show:

[0036] Figs. 1a and 1b show flowcharts of examples of a method for an optical imaging system with a scanning imaging device;

[0037] Figs. 2a and 2b show schematic drawings of examples of an optical imaging system with a scanning imaging device;

[0038] Fig. 3 shows a flowchart of an example of a flow of the calibration method;

[0039] Fig. 4 shows a schematic representation of the effect of calibration; and

[0040] Fig. 5 shows a schematic representation of a system with an optical imaging device and a computer system.

[0041] Description Some examples will now be described in more detail with reference to the accompanying figures. However, other possible examples are not limited to the features of these detailed embodiments. Furthermore, the terminology used herein to describe specific examples is not intended to be limiting of other possible examples.

[0042] When two elements A and B are combined using "or," this is to be understood as disclosing all possible combinations, i.e., only A, only B, and A and B, unless expressly defined otherwise in the individual case. Alternative wording for the same combinations may be "at least one of A and B" or "A and / or B." This applies equivalently to combinations of more than two elements. The term "and / or" encompasses all combinations of one or more of the related listed elements and can be abbreviated to " / ."

[0043] If a singular form is used, such as "a," "an," and "the," and the use of only a single element is neither explicitly nor implicitly defined as mandatory, further examples may also use multiple elements to implement the same function. If a function is described below as being implemented using multiple elements, further examples may implement the same function using a single element or a single processing entity.

[0044] Although some aspects have been described in the context of a device, it is clear that these aspects also represent a description of the corresponding method, with a block or device corresponding to a method step or a function of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block, element, or property of a corresponding device.

[0045] Figs. 1a and 1b show flowcharts of examples of a method for an optical imaging system 200a; 200b with a scanning imaging device 220 (the reference numerals of the optical imaging system refer to Figs. 2a and 2b). The method comprises obtaining 130 sensor data from a detector 222 of the scanning imaging device 220. The sensor data comprises a representation of a pattern 10; 20; 30 (shown in Figs. 2a and 2b, respectively) recorded by the detector. The method further comprises determining 140 a characteristic geometry of the representation of the pattern. The method further comprises comparing 150 the characteristic geometry with a reference geometry to determine a comparison result.The method further comprises determining 180 at least one calibration parameter for calibrating at least one control unit 224 for moving a beam-guiding element 226 of the scanning imaging device based on the comparison result. The method further comprises operating 190 the at least one control unit based on the at least one calibration parameter.

[0046] Fig. 1a shows a basic first version of the method. The method may further include optional additional features, which are shown in Fig. 1b as blocks with dashed lines and which will be explained in the course of the further description of Figs. 1a to 2b.

[0047] The method of Figs. 1a and 1b relates to an optical imaging system. Figs. 2a and 2b show schematic drawings of examples of such an optical imaging system 200a; 200b with a scanning imaging device. The optical imaging systems 200; 200b each comprise the scanning imaging device 220 and a system 210 configured to carry out the method of Figs. 1a and / or 1b. The system 210 can be implemented as a computer system. For example, the system 210 comprises one or more processors and one or more memory devices 216. Optionally, the system 210 can further comprise one or more interfaces 212. The one or more processors 214 are coupled to the one or more memory devices and to the one or more interfaces 212.The one or more processors 214 are configured to provide the functionality of the system 210 in interaction with the one or more interfaces 212 (for exchanging information with other components of the optical imaging system, such as the detector 220 of the scanning imaging device, an optical imaging sensor 230 (shown in Fig. 2b), or a screen of the optical imaging system (not shown)) and with the one or more storage devices 216 (for storing and retrieving information, such as machine-readable instructions comprising program code for the one or more processors 214). In general, the functionality of the one or more processors 214 can be implemented by the one or more processors 214 by the one or more processors 214 executing the machine-readable instructions.Accordingly, functionality attributed to the one or more processors 214 may be defined by one or more instructions from a plurality of machine-readable instructions. The system 210 may contain the machine-readable instructions, e.g., in the one or more storage devices 216.

[0048] In the following, the features of the method of Figs. 1a and 1b, of the system, of a corresponding computer program, and of the optical imaging systems 200a; 200b of Figs. 2a and 2b are explained primarily with reference to the method and the optical imaging system. It is obvious that features explained in connection with the method can also be transferred to the corresponding system and computer program, since the system is designed to carry out the method and the computer program represents an implementation of the method. Features explained with reference to the optical imaging systems 200; 200b also affect the method, the system, and the computer program.

[0049] The present concept relates to an optical imaging system with a scanning imaging device. These terms are used because the present concept can be applied to a variety of different optical imaging systems. For example, the imaging system can be a microscope system or an exoscope system, whereby an exoscope system is an optical imaging system that, in contrast to a microscope system, is used exclusively via a screen or a head-mounted display (a screen that is worn similar to glasses). Furthermore, an exoscope is usually used from a greater distance. The term "optical imaging system" is used to clarify that, on the one hand, it is a system with optical components, and on the other hand, in addition to the optical components, it includes other components, such as the system 210.The optical imaging system may include, in addition to the optical components and the system 210, other components such as an input device (such as a touchscreen, a keyboard, or control buttons), a display, a sample stage, a housing, etc.

[0050] The optical imaging system comprises at least one optical component. In the present case, the optical imaging system comprises at least the scanning imaging device 220 as an optical component. Furthermore, the optical imaging system can also comprise a further optical imaging sensor 230, as shown in Fig. 2b.

[0051] A scanning imaging device is an optical component configured to generate image data by scanning a plurality of positions on a sample. One example of a scanning imaging device is a confocal microscope. In other words, the scanning imaging device can be a confocal imaging device, such as a confocal microscope. Other examples of scanning imaging devices are the two-photon microscope and the scanning electron microscope. In all cases, a beam is directed by a beam-directing element to the aforementioned plurality of positions on the sample. One or more detectors are used to detect interactions of the beam with the sample, such as reflection, photoemission (in the case of fluorescence), emitted radiation, or electrons.In the case of confocal microscopy and two-photon microscopy, the beam is a laser beam; in the case of scanning electron microscopy, it is an electron beam. This beam is directed to the various positions on the sample (according to a grid) by means of the beam-guiding element. In the case of confocal microscopy and two-photon microscopy, a mirror or other reflective element is usually used (as shown in Fig. 2a); in the case of scanning electron microscopy, coils are used to direct the electron beam. The present disclosure deals with the calibration of the control of this beam-guiding element using the control unit. In the case of confocal microscopy and two-photon microscopy, the control unit can, for example, comprise a so-called galvo motor or a micro-electro-mechanical system (MEMS) that performs the movement of the beam-guiding element.In the case of scanning electron microscopy, the control unit may comprise a control circuit for controlling the coils.

[0052] For example, Fig. 2a shows the characteristic structure of a confocal microscope. On the left side, the laser emitter 228 is shown, which emits laser light onto the mirror 226, wherein the mirror is moved by the control unit 224 to scan the pattern 10. In some other scanning imaging devices, several mirrors are used instead of a single mirror, each controlled by a control unit. The calibration of the control of the beam-guiding element by the control unit is based on the evaluation of sensor data from the detector (which detects the interactions of the beam with the sample) of the scanning imaging device. A characteristic geometry of a representation of a pattern is determined, compared with a reference geometry, and based on the comparison, calibration parameters for calibrating the control unit are determined.

[0053] In the following, a confocal microscope is used as an example scanning imaging device. However, the principle is also transferable to other scanning imaging devices.

[0054] To determine the characteristic geometry of the pattern representation, the sensor data from the detector is evaluated. For example, image data containing the representation of the pattern can be generated from the sensor data. Using image processing, the image data is analyzed to determine the characteristic geometry based on the representation of the pattern contained in the image data. The characteristic geometry can, for example, correspond to an absolute or relative extent of one or more geometric elements of the pattern in the pattern representation, which can be determined using image processing.

[0055] Preferably, however, the characteristic geometry corresponds to or comprises a periodicity of elements of the pattern. For example, the pattern may be a periodic pattern, i.e., a pattern in which one or more elements (such as dots, lines, triangles, squares, etc.) are repeated at regular intervals along at least one lateral dimension of the pattern. As shown in Fig. 3, the pattern (there the calibration standard 310) may, for example, be a two-dimensional periodic pattern, such as a two-dimensional periodic pattern of dots. Thus, the characteristic geometry may be a periodicity of the representation of the two-dimensional periodic pattern in two dimensions. The periodicity corresponds to the distance between two adjacent elements of the repeatedly represented elements of the pattern, such as the distance between the centers of two horizontally or vertically adjacent points in the pattern shown in Fig. 3.Such a periodicity can be calculated, for example, by calculating an auto-phase correlation. Accordingly, the method, as shown in Fig. 1b, can comprise calculating 145 an auto-phase correlation. In calculating the auto-phase correlation, a 2D Fourier transform is applied to the image data, and the cross-power spectrum between the 2D Fourier transform version of the image data and the 2D Fourier transform version of the image data (i.e., between the same 2D Fourier transform version of the image data, thus becoming the auto-phase correlation) is calculated. The result is back-transformed by an inverse Fourier transform. The periodicity can now be determined by determining the distances between the peaks in the back-transformed version.

[0056] This characteristic geometry (e.g., the determined periodicity in one or two lateral dimensions) is then compared with the reference geometry to determine the comparison result. The reference geometry can also correspond to an absolute or relative extension of one or more geometric elements of the pattern, or, preferably, a reference periodicity in one or two lateral dimensions.

[0057] The reference geometry can originate from various sources. For example, the reference geometry can be a factory-defined reference geometry or a geometry determined by another scanning imaging device, which is stored in a memory 216 (i.e., in a memory device 216 of the one or more memory devices of the system) of the optical imaging system. This can be factory-defined for a plurality of optical imaging systems or, in the laboratory of the user of the optical imaging system, can be created by an optical imaging system for a fleet of similar optical imaging systems.

[0058] Alternatively, the reference geometry can be determined using the optical imaging system. Thus, as further shown in Fig. 1b, the method can comprise determining 120 the reference geometry. For example, the scanning imaging device can be manually calibrated and, in the calibrated state, used to determine the reference geometry. In other words, the method can comprise determining 120 the reference geometry based on sensor data of the scanning imaging device after a factory calibration of the scanning imaging device. For this purpose, as described above, the characteristic geometry can be determined and subsequently used as the reference geometry. For this purpose, the reference geometry can be stored in a memory 216 of the optical imaging system.As already explained above, the present concept can also be used in particular not only to ensure the isotropy of the image data that can be obtained from the sensor data of the scanning imaging device, but also to adapt a field of view of the scanning imaging device to a field of view of another imaging device (the further optical imaging sensor 230 in Fig. 2b).

[0059] Fig. 2b shows an optical imaging system comprising a first scanning imaging device (a confocal microscope 220) and a second optical imaging device (formed by the further optical imaging sensor 230). In order to create intuitive operation of an optical imaging system with two imaging devices, it may be desirable for the fields of view of the two imaging devices to correspond. This can be made possible by the proposed concept by using the sensor data of the second optical imaging device to define the reference geometry. In the present case, the sensor data of the optical imaging sensor 230 is used for this purpose, since the field of view of the optical imaging sensor cannot be corrected to the field of view of the scanning imaging device without a loss of image quality, but the reverse is possible.Also, ensuring isotropy is less of a problem in the sensor data / image data of the optical imaging sensor, as opposed to the sensor data of the scanning imaging device.

[0060] Thus, as further shown in Fig. 1b, the method may further comprise obtaining 110 further sensor data from the further optical imaging sensor 230 of the optical imaging system. The further sensor data comprise a further representation of the pattern recorded by the further optical imaging sensor. Based on this further sensor data, which may correspond to image data, a characteristic geometry (as previously explained) of the further representation of the pattern may now also be determined and used as a reference geometry and, optionally, stored. Accordingly, the method may comprise determining 120 the reference geometry based on the further representation of the pattern. A more detailed example of this is discussed, for example, in connection with Figs. 3 and 4.

[0061] The determination of the characteristic geometry or the reference geometry is based on the recording of the pattern by the respective imaging device. The pattern can be arranged at different locations in the optical imaging system. Two locations are shown as examples in Fig. 2b. So that the pattern can be "seen" by both imaging devices, the pattern 30 can be imaged, for example, on a sample carrier. The sample carrier can in turn be introduced into a sample holder (not shown) of the optical imaging system for calibration or can be arranged on a sample stage 250 of the optical imaging system. Alternatively, the sample stage 250 can comprise the pattern 30, i.e., the pattern can be printed or glued onto the sample stage.

[0062] Alternatively, for example, in the case where the reference information is not to be determined by means of the further optical imaging sensor 230, the pattern 20 can also be arranged within a housing 240, as further shown in Fig. 2b, but outside the field of view usable by a user. For this purpose, the beam-guiding element can be tilted further for calibration in order to enlarge the field of view of the scanning imaging device 220 during calibration. The method can then comprise controlling the control unit such that the pattern is detected by the detector of the scanning imaging device outside the field of view accessible to users of the optical imaging system.

[0063] Once the characteristic geometry, and optionally the reference geometry, have been determined, the subsequent comparison 150 of the characteristic geometry with the reference geometry can be performed to determine the comparison result. For example, a relationship can be formed between the characteristic geometry and the reference geometry, such as between the periodicities. This relationship can, for example, be determined separately for both lateral dimensions. Consequently, the comparison result can include a relationship between the characteristic geometry and the reference geometry (in two dimensions). This relationship can then, in turn, be used to determine the at least one calibration parameter.

[0064] The use of a relationship between the geometries is due to the nature of the movement of the beam-guiding element. A key aspect of the calibration is the scaling of the movement, i.e., how far the mirror moves for a given input value. For example, the beam-guiding element can be moved in two dimensions by the control unit. The at least one calibration parameter can now comprise at least a first scaling factor for scaling the movement of the beam-guiding element in a first (lateral) dimension and a second scaling factor for scaling the movement of the beam-guiding element in a second (lateral dimension orthogonal to the first dimension). These scaling factors now determine how far the mirror moves for a given input value. Since the reference geometry represents the target of the calibration, i.e.If the characteristic geometry corresponds to the reference geometry, then no further calibration is necessary. The relationship between the characteristic geometry and the reference geometry can be used to determine to what extent the scaling needs to be adjusted.

[0065] If the aim of the calibration is to align the field of view of the scanning imaging device with the field of view of the further optical imaging sensor, then, in addition to the scaling factor, a lateral offset between the fields of view must also be taken into account. This can be seen, for example, in Fig. 4, where the elements of the representations 410; 420 of the pattern not only have different scales, but are also laterally offset in two dimensions. This (two-dimensional) offset can be determined, on the one hand, by adapting (i.e., scaling) the representation of the pattern according to the determined scaling factors and determining the offset based on the adapted version. For example, the method, as further shown in Fig.1b, further comprise adapting 160 the representation based on the comparison result and determining 165 a difference value (such as a lateral offset) between the adapted representation and the further representation. Alternatively, the offset can be determined by applying the scaling parameters, obtaining new sensor data from the optical imaging device with a new representation of the pattern, and comparing the characteristic geometry of the new representation with the reference geometry. In other words, after applying the at least one calibration parameter, the method can comprise re-obtaining 130 the sensor data with the representation of the pattern and determining 165 the difference value between the newly obtained representation and the further representation. The at least one calibration parameter can now be further determined 180 based on the difference value.

[0066] The at least one calibration parameter (which includes the scaling factors and the offset in two lateral dimensions) is ideally determined such that, after calibration, the (newly determined) characteristic geometry corresponds to the reference geometry. In other words, the at least one calibration parameter can be determined such that, after applying the at least one calibration parameter and re-obtaining the sensor data and determining the characteristic geometry, the characteristic geometry corresponds to the reference geometry within a suitable tolerance range. If this is the case, and in particular, the offset is also taken into account, the field of view of the scanning imaging device should coincide with the field of view of the wide optical imaging sensor.In other words, the at least one calibration parameter can be determined such that, after applying the at least one calibration parameter, a field of view of the scanning imaging device corresponds to a field of view of another optical imaging sensor of the optical imaging system within a suitable tolerance range. The respective tolerance ranges depend on the application and the selected reference value.

[0067] In some cases, it may happen that the at least one calibration parameter cannot be determined in one run such that the result lies within the aforementioned tolerance range. This may be due, for example, to measurement inaccuracies and non-linearities in the control of the movement of the beam-guiding element. Therefore, the at least one calibration parameter can be adjusted iteratively until the desired precision is achieved. In other words, the comparison result can be repeatedly determined 150 (by re-obtaining the sensor data, determining the characteristic geometry, and comparing it with the reference geometry), and the at least one calibration parameter can be repeatedly adjusted 185 (based on the adjusted comparison result).However, even such an iterative method may not necessarily be successful, for example, if the fields of view differ too much or if galvo motors operate too imprecisely due to increasing age. In this case, as shown in Fig. 1b, the method may further comprise providing 170 a warning if the characteristic geometry does not correspond to the reference geometry within the tolerance range after repeated adjustment of the at least one calibration parameter.

[0068] However, if at least one calibration parameter is determined that ensures sufficiently accurate calibration, the at least one calibration parameter can be used in operation.

[0069] The calibration presented here is versatile. Particularly when the sample does not have to be manually inserted by a user of the optical imaging system, calibration can be performed automatically and regularly, for example, according to a predefined schedule or upon (or each) startup of the scanning imaging device. The system can trigger calibration upon startup or according to the schedule. If the sample is mounted on a sample carrier, it may be necessary for a user to insert the sample carrier or position it on the sample stage. Alternatively or in addition to regular calibration, calibration can also be performed as needed. Such a need arises, in particular, when a new optical imaging sensor 230 is mounted or its position is changed, or when the optical imaging system is transported or subjected to shock.The vibration can be detected by the system 210, for example, and the system can perform the calibration in response to the vibration.

[0070] In the proposed optical imaging system, in some imaging systems, an optical imaging sensor 230, also referred to as a camera, is used. Accordingly, the optical imaging sensor can be configured to generate the additional sensor data, which is imaging sensor data. For example, the one optical imaging sensor of the stereoscopic imaging device can comprise or correspond to an APS (Active Pixel Sensor) or a CCD (Charge-Coupled Device)-based image sensor. In APS-based image sensors, for example, the light at each pixel is captured using a photodetector and an active amplifier of the pixel. APS-based image sensors are often based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology.In CCD-based image sensors, incoming photons are converted into electron charges at a semiconductor-oxide interface, which are then moved by an image sensor circuit between capacitive bins (sinks) in the image sensors to perform imaging.

[0071] The system 210 may be configured to obtain (i.e., receive or read out) the sensor data of the detector 222 from the detector of the scanning imaging device 220 and / or to obtain (i.e., receive or read out) the further sensor data of the optical imaging sensor 230 from the optical imaging sensor 230, for example via the interface 212. The one or more interfaces 212 of the system 210 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be present in digital (bit) values ​​according to a specific code within a module, between modules, or between modules of different units. The one or more interfaces 212 may, for example, comprise interface circuits configured to receive and / or transmit information.

[0072] The one or more processors 214 of the system 210 may be implemented by one or more processing units, one or more processing devices, any means of processing, such as a processor, a computer, or a programmable hardware component operable with appropriately adapted software. In other words, the described function of the one or more processors 214 may also be implemented in software that is then executed on one or more programmable hardware components. Such hardware components may include a general-purpose processor (such as a central processing unit), a digital signal processor (DSP), a microcontroller, etc.

[0073] The one or more storage devices 216 of the system 210 may include at least one member from the group of computer-readable storage media, such as a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a network storage device.

[0074] More details and aspects of the method, system, corresponding computer program, optical imaging system, and scanning imaging device are mentioned in connection with the concept or examples described before or after (for example, in connection with Figures 3 to 5). The method, system, computer program, optical imaging system, and scanning imaging device may include one or more additional optional features corresponding to one or more aspects of the proposed concept or the described examples as described before or after.

[0075] Various aspects of the present disclosure relate to an apparatus for automatically calibrating the field of view of an optical imaging device or an optical imaging system, such as a microscope or microscope system, using image processing. In particular, some embodiments address the automatic calibration of a field of view of multiple imaging devices.

[0076] For this purpose, a shared calibration target (such as the pattern used in connection with Figs. 1a to 2b) is used for both modalities. The calibration target can be designed as a sample slide. For example, a calibration target can be used that represents a periodic (in x and y dimensions) pattern. This periodicity can then be determined, for example, using auto-phase correlation. The periodicity can, for example, correspond to the characteristic geometry discussed in connection with Figs. 1a to 2b or represent an aspect of the characteristic geometry. The advantage of using auto-phase correlation is the low technical effort. Segmentation can be omitted if necessary. A reduced noise sensitivity compared to conventional targets and algorithms is also worth highlighting.

[0077] The same target can be imaged both in the wide-field (by a camera-based imaging device, such as the additional optical imaging sensor 230 of Fig. 2b) and confocally. The deviation between the confocal image area and the wide-field image area can be determined. For example, the wide-field image can be used as a reference for the scaling to be achieved by the confocal scanner (with respect to the scaling in the x-dimension and the y-dimension), since the camera sensors (e.g., CCD sensors) do not allow modification.

[0078] This allows the confocal image to be limited to the identical image area (of the wide-field camera). At the same time, the isotropy of the image can be ensured, or alternatively, at least set to be identical to the wide-field sensor. Fig. 3 shows a flow chart of an example of a calibration procedure. In the example of Fig. 3, the calibration standard 310 (such as the pattern discussed in connection with Figs. 1a to 2b, which can be a periodic point grid) is first inserted. Then a wide-field image is taken, which is used as the reference image 320. The microscope modality is then switched to confocal, with the calibration standard 310 remaining inserted. Another image of the south is taken with the confocal scanner to obtain the confocal image 350 (also CLSM image, Confocal Laser Scanning Microscope).

[0079] For both images 320; 350, the period (periodicity) of the sample in the x-dimension and y-dimension is determined using autophase correlation 330; 360 and other morphological operations. The wide-field image is used below as the reference / target value 340, since the camera cannot offer this type of parameterization and is therefore fixed. The deviation of the periods in the confocal image compared to the wide-field image allows the relevant settings of the galvo control to be adjusted 370. For example, the x-offset, the y-offset, the x-scaling, and the y-scaling are calculated. Adjusting these values ​​leads to a modified mirror movement so that the calibration standard images, as closely as possible, identically in confocal and wide-field. As a result, the image area and the scaling are set identically to the reference image.

[0080] For verification, after setting the values, a confocal image is acquired again and processed using the same method. The deviation from the wide-field image must remain within a certain tolerance range. Optionally, a further 380 iteration can be performed based on the confocal image.

[0081] Alternatively, an older confocal image can be used to determine the reference 340, for example, if the proposed concept is used to compensate for aging phenomena or to readjust the image area when changing between different stands.

[0082] Fig. 4 shows a schematic representation of the effect of calibration. Fig. 4 shows, on the one hand, the reference image 410 of the calibration standard and, on the other hand, the confocal image 420. In an original overlay 430, before the calibration is performed, it can be seen that the confocal image has a larger scale in the x-dimension and a slightly smaller scale in the y-dimension. Furthermore, an offset is present. After correcting the image area, a precise overlay 440 of the representations of the calibration standard is achieved.

[0083] The proposed method can further be used to test the settings of the confocal scanner on a captured image using periodicity, such as isotropy, distortion, acquisition errors that may arise due to different scanning speeds (isotropy should be maintained at different scanning speeds), etc.

[0084] This calibration can be performed automatically or used to evaluate the current settings and assist in manual adjustment. The independent evaluation of the scaling and offset values ​​provides added value compared to manual calibration. Significant time savings can also be achieved in the production of the optical imaging system, while simultaneously increasing quality and reducing calibration costs. Calibration validation also becomes automated and quantifiable, compared to manual, purely visual evaluation.

[0085] Present embodiments of the application of the concept to a microscope system comprising a confocal microscope and a wide-field microscope are described herein. However, the present concept can also be applied to other optical imaging systems and scanning imaging devices, such as other optical imaging systems comprising a scanning imaging device and a camera-based imaging device.

[0086] More details and aspects of the device for automatic calibration of the field of view are mentioned in connection with the concept or examples described previously (e.g., Figs. 1a to 2b). The device for automatic calibration of the field of view may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or the described examples, as described previously or subsequently. Some embodiments relate to an optical imaging system or an optical imaging device, such as a microscope system or a microscope comprising a system as described in connection with one or more of Figs. 1a to 4. Alternatively, an optical imaging device, such as a microscope, may be part of, or connected to, a system as described in connection with one or more of Figs. 1a to 4. Fig.5 shows a schematic representation of a system 500 configured to carry out a method described herein. The system 500 comprises an optical imaging device 510, such as a microscope (e.g., a scanning microscope or a non-scanning microscope), and a computer system 520. The optical imaging device 510 is configured to capture images and is connected to the computer system 520. The computer system 520 is configured to carry out at least part of a method described herein. The computer system 520 may be configured to execute a machine learning algorithm. The computer system 520 and the optical imaging device 510 may be separate units, but may also be integrated together in a common housing.The computer system 520 could be part of a central processing system of the optical imaging device 510 and / or the computer system 520 could be part of a subcomponent of the optical imaging device 510, such as a sensor, an actuator, a camera, or an illumination unit, etc. of the optical imaging device 510.

[0087] Computer system 520 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system having one or more processors or one or more storage devices distributed at various locations, for example, at a local client and / or one or more remote server farms and / or data centers). Computer system 520 may include any circuitry or combination of circuitry. In one embodiment, computer system 520 may include one or more processors, which may be of any type.As used herein, processor may mean any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set microprocessor (CISC), a reduced instruction set microprocessor (RISC), a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field-programmable gate array (FPGA), e.g., of a microscope or a microscope component (e.g., camera), or any other type of processor or processing circuit. Other types of circuitry that may be included in computer system 520 may include a custom-built circuit, an application-specific integrated circuit (ASIC), or the like, such as one or more circuits (e.g., a communications circuit) for use with wireless devices such asMobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 520 may include one or more storage devices, which may include one or more storage elements suitable for the particular application, such as main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives handling removable media such as CDs, flash memory cards, DVDs, and the like. The computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 520.

[0088] Some or all of the method steps may be performed by (or using) a hardware device, such as a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the key method steps may be performed by such a device.

[0089] Depending on specific implementation requirements, embodiments of the invention may be implemented in hardware or software. The implementation may be performed using a non-volatile storage medium, such as a digital storage medium, such as a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM and EPROM, an EEPROM, or a FLASH memory, on which electronically readable control signals are stored that interact (or can interact) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer-readable. Some embodiments according to the invention comprise a data carrier with electronically readable control signals that can interact with a programmable computer system such that one of the methods described herein is performed.

[0090] In general, embodiments of the present invention can be implemented as a computer program product with program code, wherein the program code is effective for executing one of the methods when the computer program product is running on a computer. The program code can, for example, be stored on a machine-readable medium.

[0091] Further embodiments include the computer program for carrying out one of the methods described herein, which is stored on a machine-readable carrier.

[0092] In other words, one embodiment of the present invention is therefore a computer program having a program code for carrying out one of the methods described herein when the computer program runs on a computer.

[0093] A further embodiment of the present invention is therefore a storage medium (or a data carrier or a computer-readable medium) comprising a computer program stored thereon for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium, or the recorded medium is typically tangible and / or non-seamless. A further embodiment of the present invention is an apparatus as described herein, comprising a processor and the storage medium.

[0094] A further embodiment of the invention is therefore a data stream or signal sequence representing the computer program for carrying out one of the methods described herein. The data stream or signal sequence can, for example, be configured to be transmitted via a data communication connection, for example, via the Internet.

[0095] A further embodiment comprises processing means, for example a computer or a programmable logic device, configured or adapted to carry out one of the methods described herein. A further embodiment comprises a computer on which the computer program for carrying out one of the methods described herein is installed. A further embodiment according to the invention comprises a device or a system configured to transmit (for example electronically or optically) a computer program for carrying out one of the methods described herein to a recipient. The recipient may, for example, be a computer, a mobile device, a storage device, or the like. The device or system may, for example, comprise a file server for transmitting the computer program to the recipient.

[0096] In some embodiments, a programmable logic device (e.g., a field-programmable gate array, FPGA) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field-programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are preferably performed by any hardware device.

[0097] Reference symbol

[0098] 10; 20; 30 patterns

[0099] 110 Obtaining additional sensor data

[0100] 120 Determining a reference geometry

[0101] 130 Obtaining sensor data

[0102] 140 Determining a characteristic geometry

[0103] 150 Comparing the characteristic geometry with the reference geometry

[0104] 160 Adjusting a representation of the pattern

[0105] 165 Determining a difference value

[0106] 170 Providing a warning

[0107] 180 Determining at least one calibration parameter

[0108] 185 Adjusting at least one calibration parameter

[0109] 190 Operating at least one control unit

[0110] 200a; 200b Optical imaging system

[0111] 210 System

[0112] 212 One or more interfaces

[0113] 214 One or more processors

[0114] 216 One or more storage devices

[0115] 220 scanning imaging device

[0116] 222 Detector

[0117] 224 control unit

[0118] 226 Light-conducting element, mirror

[0119] 228 laser emitters

[0120] 230 additional optical imaging sensors

[0121] 240 housings

[0122] 250 sample table

[0123] 310 Calibration standard

[0124] 320 Reference Figure

[0125] 330 Auto-Phase Correl ati on

[0126] 340 Reference

[0127] 350 Confocal imaging

[0128] 360 Auto-Phase Correl ati on

[0129] 370 Adjustment of the Gal vo control

[0130] 380 Further iteration 410 Reference recording

[0131] 420 Confocal image

[0132] 430 Original overlay

[0133] 440 Overlay after calibration 500 System

[0134] 510 Optical imaging device

[0135] 520 Computer system

Claims

A method for an optical imaging system (200a; 200b; 500) having a scanning imaging device (220; 510), the method comprising: Obtaining (130) sensor data from a detector (222) of the scanning imaging device (220), the sensor data comprising a representation of a pattern (10; 20; 30) captured by the detector; determining (140) a characteristic geometry of the representation of the pattern; Comparing (150) the characteristic geometry with a reference geometry to determine a comparison result; Determining (180) at least one calibration parameter for calibrating at least one control unit (224) for moving a beam-guiding element (226) of the scanning imaging device based on the comparison result; and Operating (190) the at least one control unit based on the at least one calibration parameter. The method according to claim 1, wherein the method further comprises obtaining (110) further sensor data of a further optical imaging sensor (230) of the optical imaging system, wherein the further sensor data comprises a further representation of the pattern recorded by the further optical imaging sensor, and determining (120) the reference geometry based on the further representation of the pattern. The method according to claim 2, wherein the method further comprises adapting (160) the representation based on the comparison result, determining (165) a difference value between the adapted representation and the further Representation, and determining (180) the at least one calibration parameter further based on the difference value. The method according to one of claims 2 or 3, wherein the method, after applying the at least one calibration parameter, comprises re-obtaining (130) the sensor data with the representation of the pattern, determining (165) a difference value between the newly obtained representation and the further representation, and determining (180) the at least one calibration parameter further based on the difference value. The method according to one of claims 1 to 4, wherein the reference geometry is a factory-defined reference geometry or a geometry determined by means of another scanning imaging device, which is stored in a memory (216) of the optical imaging system.The method according to one of claims 1 to 5, wherein the at least one calibration parameter is determined such that, after applying the at least one calibration parameter, a field of view of the scanning imaging device corresponds to a field of view of another optical imaging sensor of the optical imaging system within a tolerance range. The method according to one of claims 1 to 6, wherein the at least one calibration parameter is determined such that, after applying the at least one calibration parameter and re-obtaining the sensor data and determining the characteristic geometry, the characteristic geometry corresponds to the reference geometry within a tolerance range. The method according to one of claims 1 to 7, wherein the comparison result is repeatedly determined (150) and the at least one calibration parameter is repeatedly adjusted (185).The method of claim 8, wherein the method further comprises providing (170) a warning if the characteristic geometry after repeated adjustment. of the at least one calibration parameter does not correspond to the reference geometry within a tolerance range. The method according to one of claims 1 to 9, wherein the pattern is a periodic pattern and the characteristic geometry comprises a periodicity of the representation of the periodic pattern. The method according to claim 10, wherein the periodicity is determined by calculating (145) an auto-phase correlation. The method according to one of claims 1 to 11, wherein the pattern is a two-dimensional periodic pattern and the characteristic geometry comprises a periodicity of the representation of the two-dimensional periodic pattern in two dimensions.The method according to one of claims 1 to 12, wherein the beam-guiding element is movable in two dimensions by the control unit, and the at least one calibration parameter comprises at least a first scaling factor for scaling the movement of the beam-guiding element in a first dimension and a second scaling factor for scaling the movement of the beam-guiding element in a second dimension. The method according to one of claims 1 to 13, wherein the characteristic geometry and the comparison result are determined upon startup of the scanning imaging device, after detection of a vibration of the optical imaging system, and / or according to a predetermined schedule.A system (210; 520) for an optical imaging system (200; 500) having a scanning imaging device (220; 510), the system comprising one or more processors (214) and one or more memory devices (216), the system being adapted to carry out the method according to any one of the preceding claims.

Images