Apparatus, optical imaging system, method and computer program

Abstract

Examples provide an apparatus (130) for an optical imaging system. The apparatus (130) comprises one or more processors (134) and one or more storage devices (136). The apparatus (130) is configured to obtain sample position data indicative of a position of the microscope relative to a sample. Further, the apparatus (130) is configured to obtain position data indicative of a position of a microscope of the optical imaging system relative to the user of the optical imaging system. The apparatus (130) is further configured to determine deviation angle data indicative of a deviation of a viewing path of the user on the sample and a viewing path of the microscope on the sample in an image plane of the optical imaging system. The deviation angle data is determined based on the position data and the sample position data. Further, the apparatus (130) is configured to determine output data for informing the user about the deviation of the viewing path of the user on the sample and the viewing path of the microscope on the sample. The output data is determined based on the deviation angle data. The apparatus (130) is further configured to transmit a display signal indicative of the output data.

Description

[0001] Apparatus, Optical Imaging

[0002] System, Method and Computer Program

[0003] Technical field

[0004] Examples relate to an apparatus, an optical imaging system, a method, and a computer program.

[0005] Background

[0006] During surgical operations, a surgeon needs to maintain eye-hand coordination. Surgical microscopes traditionally employ optical eyepieces, which restrict the relative position of the surgeon and the microscope. As a result, the microscope-provided view is always aligned with the surgeon, i.e., the eye-hand coordination is always preserved. However, in a full-digital microscope, the visualization is done via a monitor or a head-mounted display. The use of digital displays allow more flexibility in the relative positioning of the microscope and the surgeon, which can easily lead to loss of vision-hand coordination. For example, in cases such as endoscopic surgeries, where eye-hand coordination is typically lost, the surgeon needs special training to mitigate the increased risk of an accident.

[0007] Therefore, there may be a desire for an improved concept for avoiding losing eye-hand coordination, e.g., due to a misalignment of the microscope and the surgeon.

[0008] Summary

[0009] This desire is addressed by the subject-matter of the independent claims.

[0010] The concept proposed in the present disclosure is based on the insight, that a user of an optical imaging system can be informed about a deviation, e.g., a deviation angle and / or an alignment level, angle of his viewing path on a sample and a viewing path of the microscope on the sample. Informing the user about the deviation may allow to raise an awareness of the user to a misalignment of the viewing path of the microscope with the viewing path of the user. This may allow the user to observe, e.g., on a display device such like a monitor, a head-mounted display, an alignment of the microscope viewing path with his viewing path. Thus, the user can take certain measures, e.g., performed a realignment, to counteract a loss of eye-hand coordination. In this way, an eye-hand coordination can be persevered in an improved way.

[0011] Examples provide an apparatus for an optical imaging system. The apparatus comprises one or more processors and one or more storage devices. The apparatus is configured to obtain sample position data indicative of a position of the microscope relative to a sample. Further, the apparatus is configured to obtain position data indicative of a position of a microscope of the optical imaging system relative to the user of the optical imaging system. The apparatus is further configured to determine deviation angle data indicative of a deviation of a viewing path of the user on the sample and a viewing path of the microscope on the sample in an image plane of the optical imaging system. The deviation angle data is determined based on the position data and the sample position data. Further, the apparatus is configured to determine output data for informing the user about the deviation of the viewing path of the user on the sample and the viewing path of the microscope on the sample. The output data is determined based on the deviation angle data. The apparatus is further configured to transmit a display signal indicative of the output data. Obtaining the position data and the sample position data may allow to determine a deviation, e.g., a deviation angle and / or an alignment level, between the viewing path of the user and the viewing path of the microscope. Thus, a misalignment of the natural view of the user and the image captured by the microscope can be determined. The user can be informed about the deviation or misalignment of the viewing path of the user and the viewing path of the microscope to raise an awareness of the user of the deviation or misalignment. In this way, the user can take certain countermeasures, e.g., the user can realign the microscope, change his position and / or does not perform certain operation tasks, for example. In this way, a user experience can be improved.

[0012] In an example, the apparatus may be configured to obtain the position data by obtaining the position of a head of the user relative to the microscope and / or a position of eyes of the user. Determining the position of the head of the user and / or the position of the eyes of the user may allow to determine the position data in a facilitated way.

[0013] In an example, the apparatus may be further configured to determine the position of the user relative to the microscope by determining a distance between the user and the microscope and an angle between the user and the microscope with respect to a viewing path of the microscope based on the position data. Determining the distance and the angle may allow to determine the position of the user relative to the microscope with less effort. For example, no absolute position of the user and the microscope has to be determined.

[0014] In an example, the apparatus may be configured to determine viewing path data indicative of the viewing path of the user based on the position of the head of the user and / or the position of the eyes of the user. Further the apparatus may be configured to determine the output data based on the viewing path data. Determining the viewing path data may allow to determine a possible alignment of the microscope. For example the viewing path of the user cannot be aligned with the viewing path of the microscope due to a restricted possible movement of the microscope. Thus, the user can be informed about a non-possible alignment of his viewing path with the viewing path of the microscope, for example. In this way, the user can take certain countermeasures if necessary.

[0015] In an example, the apparatus may be configured to obtain working distance data indicative of a working distance of the microscope and to determine the sample position data based on the working distance data. Using working distance data the apparatus may determine a distance between the microscope and the sample in a facilitated way. Based on the distance the apparatus may determine a position of the microscope relative to the sample. In this way, a computational effort can be reduced and / or a sensor needed to detect the position of the microscope relative to the sample can be omitted.

[0016] In an example, the apparatus may be configured to obtain movement data indicative of a movement of the microscope and to generate visual representation data indicative of a visual representation of the viewing path of the microscope and the viewing path of the user. Obtain movement data may allow the apparatus to detect an adjustment of the viewing path of the microscope. Thus, the apparatus may generate the visual representation data to inform the user about a current adjustment of the viewing path of the microscope. In this way, the user can be informed about an adjustment of the alignment of the microscope during the adjustment. Thus, a user can pause certain operation tasks during the alignment of the microscope, for example.

[0017] In an example, the apparatus may be further configured to receive sensor data of a position sensor. The sensor data is indicative of a position of the user. Further, the apparatus may be configured to determine the position data based on the sensor data. In this way, the apparatus may obtain the position data in a facilitated way.

[0018] In an example, the position sensor may be a camera, a time-of-fhght sensor, a structured light projector and / or an optical imaging sensor of the microscope. Thus, an appropriate sensor for measuring the sensor data can be used or selected.

[0019] In an example, the apparatus may be configured to obtain threshold data indicative of a threshold for the deviation between the viewing path of the user and the viewing path of the microscope. Further, the apparatus may be configured to compare the threshold data with the deviation angle data and determine the output data based on the threshold data. For example, the apparatus may only determine the output data if the deviation angle data indicates a deviation angle which exceeds a threshold (angle) indicated by the threshold data. Thus, the user can only be informed about a misalignment, which is greater than a certain threshold (angle). In this way, a user experience can be improved.

[0020] In an example, if the threshold for the deviation is exceeded the output data may be indicative of information about the exceedance of the threshold. Thus, the user can be informed about an infringement of the threshold.

[0021] In an example, the apparatus may be configured to receive the threshold data from an input device. For example, the user can define the threshold using the input device. In this way, user experience may be increased.

[0022] Examples provide an optical imaging system comprising an apparatus as described above.

[0023] In an example, the optical imaging system may further comprise a microscope. The position data may be determined based on sensor data of the microscope. In this way, determination of the position data can be facilitated.

[0024] Examples provide a method for an optical imaging system. The method comprises obtaining position data indicative of a position of a microscope of the optical imaging system relative to a user of the optical imaging system. The method further comprises obtaining sample position data indicative of a distance between the microscope and the sample. Further, the method comprises determining, based on the position data and the sample position data, deviation angle data indicative of a deviation of a viewing path of the user on the sample and a viewing path of the microscope on the sample in an image plane of the optical imaging system. The method further comprises determining, based on the deviation angle data, output data for informing the user about the deviation of the viewing path of the user on the sample and the viewing path of the microscope on the sample and transmitting a display signal indicative of the output data.

[0025] Various examples of the present disclosure relate to a corresponding computer program with a program code for performing the above method when the computer program is executed on a processor.

[0026] Short description of the Figures

[0027] Some examples of apparatuses and / or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

[0028] Figs, la and lb show schematic diagrams of an example of an apparatus for an optical imaging system and of a corresponding optical imaging system comprising the apparatus;

[0029] Figs. 2a-2c show different examples of microscope-user alignment;

[0030] Fig. 3 shows a flow chart of an example of a method for an optical imaging system that uses reflectance imaging and fluorescence imaging; and

[0031] Fig. 4 shows a schematic diagram of a system comprising a microscope and a computer system.

[0032] Detailed Description

[0033] Various examples will now be described more fully with reference to the accompanying drawings in which some examples are illustrated. In the figures, the thicknesses of lines, layers and / or regions may be exaggerated for clarity. Figs, la and lb show schematic diagrams of an example of an apparatus 130 for an optical imaging system 100 and of a corresponding optical imaging system 100 comprising the apparatus 130. The apparatus 130 is tasked with controlling various aspects of a microscope 120, e.g., a stereo microscope, of the optical imaging system 100, which may be a surgical optical imaging system, and of the entire optical imaging system 100 and / or with processing various types of sensor data of the optical imaging system 100. Consequently, the apparatus 130 may be implemented as a computer system, which interfaces with the various components of the optical imaging system 100, e.g., the optical imaging sensor system.

[0034] The apparatus 130 comprises, as shown in Fig. la, one or more processors 134 and one or more storage devices 136. Optionally, the apparatus 130 further comprises one or more interfaces 132. The one or more processors 134 are coupled to the one or more storage devices 136 and to the optional one or more interfaces 132. In general, the functionality of the apparatus 130 may be provided by the one or more processors 134 (for determining the deviation angle data), in conjunction with the one or more interfaces 132 (for exchanging information, e.g., with the optical imaging sensor system 122a, 122b or a position sensor 124a) and / or with the one or more storage devices 136 (for storing and / or retrieving information).

[0035] The apparatus 130 is configured to obtain sample position data indicative of a position of the microscope 120 relative to the sample 110. For example, the sample position data may be obtained by receiving (first) sensor data from a position sensor, e.g., the position sensor 124b or the position sensor 124a. For example, the position sensor 124b may be an optical imaging sensor 122a, 122b of the microscope 120. The apparatus 130 may be communicatively coupled with the position sensor 124b. Thus, the apparatus 130 may receive (first) sensor data from the position sensor 124b. Optionally or alternatively, the apparatus 130 may determine the position data, e.g., based on the received (first) sensor data. For example, the received (first) sensor data may be raw data and the apparatus 130 may post process the (first) sensor data to determine the position data. Optionally or alternatively, the apparatus 130 may comprise a position sensor, e.g., the microscope 120 comprising the position sensor 124b. Thus, the apparatus 130 may control the position sensor 124b to measure the (first) sensor data. In this way, the apparatus can measure the sample position data using the position sensor 124b.

[0036] The sensor position data may allow the apparatus 130 to determine an orientation and / or position of the microscope 120 relative to the sample 110. However, a determination of a deviation of a viewing path 162 of the microscope 120 and a viewing path 152 of the user 150 can be facilitated by obtaining data indicative of a position of the user 150. Thus, the apparatus 130 is configured to obtain position data indicative of a position of the microscope 120 of the optical imaging system 100 relative to the user 150 of the optical imaging system 100.

[0037] For example, the position data may be obtained by receiving (second) sensor data from a position sensor e.g., the position sensor 124a. The apparatus 130 may be communicatively coupled with the position sensor 124a. Thus, the apparatus 130 may receive (second) sensor data from the position sensor 124a. Optionally or alternatively, the apparatus 130 may determine the position data, e.g., based on the received (second) sensor data. For example, the received (second) sensor data may be raw data and the apparatus 130 may post process the (second) sensor data to determine the position data. Optionally or alternatively, the apparatus 130 may comprise a position sensor, e.g., the position sensor 124a. Thus, the apparatus 130 may control the position sensor 124a to measure the (second) sensor data. In this way, the apparatus can measure the position data using the position sensor 124a. The first sensor data and the second sensor may be received from different position sensors. Alternatively, the first sensor data and the second sensor may be received from the same position sensor, e.g., the position sensor 124b. For example, the first sensor data and the second sensor data may be received simultaneously or separated.

[0038] Based on the position data and the sample position data the apparatus 130 can determine the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120. Thus, the apparatus 130 can determine a misalignment between the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120. Therefore, the apparatus 130 is further configured to determine deviation angle data indicative of a deviation of a viewing path 152 of the user 150 on the sample 110 and the viewing path 162 of the microscope 120 on the sample 110 in an image plane 170 of the optical imaging system 100. The deviation angle data is determined based on the position data and the sample position data.

[0039] Obtaining the position data may allow the apparatus 130 to determine the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120. Thus, the apparatus 130 can determine the deviation by determining a deviation of the viewing path 152 of the user 150 and the viewing path 162 of the microscope. However, not every deviation between the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120 may affect an experience of the user 150. Thus, the apparatus 130 may be further configured to determine output data to be provided to the user 150.

[0040] Therefore, the apparatus 130 is configured to determine output data for informing the user 150 about the deviation of the viewing path 152 of the user 150 on the sample 110 and the viewing path 162 of the microscope 120 on the sample 110. The output data is determined based on the deviation angle data. For example, the output data may comprise information about a deviation angle and / or an alignment level between the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120. The deviation angle may be a deviation angle 0 of the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120 in the image plane 170. The misalignment level can be determined based on the deviation angle 0. For example, the output data may be indicative of the deviation angle 0, information about an exceedances of a threshold angle and / or information about a neglectable deviation between the viewing path 162 of the microscope 120 and the viewing path 152 of the user 150 (i.e., a still acceptable alignment level). In this way, the apparatus 130 can generate data to inform the user 150 of the optical imaging system 100 about an alignment of the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120.

[0041] To inform the user 150 the apparatus 130 is further configured to transmit a display signal indicative of the output data. Thus, the user 150 can be informed about the deviation of the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120.

[0042] The apparatus 130 can help a user 150, e.g., a surgeon, to avoid loss of eye-hand coordination due to misalignment of the microscope 120 and the surgeon. If the microscope 120 and the user 150 are misaligned, i.e., the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120 are misaligned, the user 150 may lose his eye-hand coordination, because the image of the sample 110 displayed on a display device does not match with an orientation of the user 150 relative to the sample 110. Informing the user 150 may allow the user 150 to take certain measures. For example, the user 150 may only realign the microscope 120 if the image of the sample 110 displayed on the display device may likely cause a loss of eye-hand coordination. Thus, by transmitting the display signal the user 150 can be informed about the misalignment, e.g., about an exceedances of threshold angle (which could be indicative of a likely loss of eye-hand coordination). In this way, a permanent or periodical control of the alignment until an eye-hand coordination is achieved or is convenient enough can be avoided using the apparatus 130.

[0043] It is a finding of the inventors that the apparatus 130 can be used to inform the user 150 about a misalignment of the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120. For example, the apparatus 130 may measure the relative position and orientation of the microscope 120 automatically. Further, the apparatus 130 may measure the relative position of the user 150 to the microscope 120. The determined output data may indicate whether the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120 are aligned or not. For example, the output data may be indicative of an icon. The icon can be displayed on a display device. The icon can indicate the alignment level, for example, perfect, acceptable, and bad alignment (see also Figs. 2a-2c).

[0044] For example, using the apparatus 130 manual alignment by the user 150 can be avoided. Manual alignment by the user 150 suffers from several disadvantages. For example, it is non- intuitive, since during manual alignment, the hands of the user 150 or handheld tools are not shown in the image (of the sample 110) on the display device, and thus it is not obvious what would be the correct orientation. For example, it is prone to alignment errors due to lack of obvious alignment indications, i.e., while the user 150 is using his hands to align the microscope, the hands are not visible in the image displayed on the display device, and thus the surgeon can easily lose the up-down orientation. For example, the realignment may interrupt the use of the optical imaging system, e.g., a surgical procedure, unduly. The user 150 must free the hand from the tools to grab the microscope 120, then find the right tool again and resume the previous task and / or position. These disadvantages can be overcome using the apparatus 130 to inform the user 150 about a deviation of the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120.

[0045] The deviation, e.g., the alignment level, can be determined by the relative angle 0 between the viewing path 162 of the user 150 and the viewing path 162 of the microscope 120. The calculation concept is shown in lb. For example, the microscope 120 may comprise a position sensor 124b to detect the position of the head of the user 150 relative to the microscope 120. For example, the microscope 120 may measure the distance 148 between microscope 120 and user 150 and angle (|). Optionally, as described below the distance of the surgical cavity and thus of the sample 110 from the microscope 120 can be determined from the working distance of the optics, e.g., an objective, an illumination lens, of the microscope 120. These measurements may allow to determine the deviation angle 0. The deviation angle 0 may be indicative for the alignment level.

[0046] Optionally or alternatively, the apparatus 130 may determine the deviation angle based on the global position of the user 150 and the microscope 120 with respect to the sample 110, e.g., the sample 110 may be arranged at the origin of a Cartesian coordinate system. For example, the position data may be indicative of a global position of the user 150 relative to the sample 110. Thus, the position of the microscope 120 relative to the user 150 can be determined based on the global position of the user 150 and the sample position data. For example, the sample position data may be indicative of a global position of the microscope 120 relative to the sample 110. In a Cartesian coordinate system, a global position refers to a specific point's location using a set of coordinates that can uniquely identify that point within the chosen system. For example, in a three-dimensional Cartesian coordinate system, a global position is represented using three coordinates: X, Y, and Z. Thus, instead of using distance and an angle between the user 150 and microscope 120 the apparatus 130 may utilize the global position to determine the deviation angle data.

[0047] Optionally or alternatively, the apparatus 130 may receive the position data and the sensor data from only, i.e., at most, one sensor. For example, the one sensor may be a sensor separate to the optical imaging system 100. The one sensor may be communicatively coupled with the optical imaging system 100. For example, the one sensor may be a camera mounted to a ceiling of a room, in which the optical imaging system 100 is installed, a camera of a smartphone, a camera of a laptop.

[0048] The proposed concept may be built around two main components - the stereo microscope 120, which comprises the optical components and the apparatus 130, which may be used to control the optical imaging system 100, process sensor data of the microscope 120, e.g., the optical imaging sensor system 122a, 122b, and / or to generate the control data.

[0049] In general, a microscope, such as the microscope 120, is an optical instrument that is suitable for examining objects that are too small to be examined by the human eye (alone). For example, a microscope 120 may provide an optical magnification of a sample, such as a sample 110 shown in Fig. la. In modern microscopes, the optical magnification is often provided for a camera or an imaging sensor, such as the optical imaging sensor system 124b of the microscope 120.

[0050] There are a variety of different types of optical imaging systems. If the optical imaging system 100 is used in the medical or biological fields, the sample 110 may be a sample of organic tissue, e.g., arranged within a petri dish or present in a part of a body of a patient. In some examples of the present disclosure the optical imaging system 100 may be a surgical optical imaging system, e.g., an optical imaging system 100 that is to be used during a surgical procedure, such as an oncological surgical procedure or during tumor surgery. However, the proposed concept may also be applied to other types of microscopy, e.g., microscopy in a laboratory or microscopy for the purpose of material inspection.

[0051] Fig. lb shows a schematic diagram of an example of an optical imaging system 100, e.g., a surgical optical imaging system, comprising the microscope 120, e.g., a stereo microscope, and the apparatus 130. The apparatus 130 may be part of the microscope 120. In general, a (surgical) optical imaging system 100 is a system that comprises a (stereo) microscope 120 and additional components, which are operated together with the stereo microscope 120. In other words, a (surgical) optical imaging system 100 is a system that comprises the (stereo) microscope 120 and one or more additional components, such as the apparatus 130 (which may be a computer system being adapted to control the microscope 120 and, for example, to determine the deviation angle data), an illumination system (which is used to illuminate a sample 110 being imaged by the microscope 120), additional sensors, a display device, etc.

[0052] As can be seen in Fig. lb the optical imaging sensor system may comprise two separated optical imaging sensors 122a, 122b. Alternatively, the optical imaging sensor system may be a sensor array. For example, the optical imaging sensors 122a, 122b may define the microscope stereopsis axis. For example, the microscope stereopsis axis may define the viewing path 162 of the microscope 120.

[0053] In an example, the apparatus 130 may be configured to obtain the position data by obtaining a position of the head of the user 150 relative to the microscope and / or a position of eyes of the user 150. For example, the position of the head of the user 150 may be an orientation of the head of the user 150. Based on the orientation of the head of the user 150 a viewing direction of the user 150 can be determined. Alternatively, the position of the eyes of the user 150 can be used to determine a gaze direction of the user 150. This information can be used by the apparatus 130 to determine the deviation angle data in a facilitated and / or improved way. For example, the apparatus 130 may obtain the position of the head of the user 150 and / or the position of eyes of the user 150 by receiving sensor data, e.g., from the position sensor 124a, and / or by measuring (e.g., as described above with reference to the position data).

[0054] Two factors may determine an angle of view of a person: a head or facial orientation (i.e., face pose or face direction) and eye orientation (i.e., eye gaze direction or angle of view). The head orientation can be used to determine a global direction of the gaze. The eye orientation can be used to determine a local direction of the gaze, i.e., the angle of view. When a head of the person is level and he or she looks straight ahead, then the line of sight of the eye gaze and the facial orientation are straight ahead with 0° azimuth and 0° elevation and 0° tilt. This head position can also be described as a neutral head orientation position.

[0055] The position of the eyes of the user 150, e.g., an angle of view of the user 150, may be used by the apparatus 130 to determine the position data. For example, the angle of view of the user 150 may indicate the viewing angle of the user 150 or may be identical with the viewing path 152 of the user 150. This can be used to determine the position data. In this way, the position data can be generated by the apparatus 130 based on an actual angle of view of the user 150, for example.

[0056] Optionally or alternatively, the position of the head of the user 150, e.g., an orientation of the head, relative to the microscope 120 can be used by the apparatus 130 to determine the position data. For example, the orientation of the head may be indicative of the viewing angle of the user 150, which could be used to determine the position data. Obtaining the position data by obtaining the position of the head of the user 150 relative to the microscope 120 and / or the position of the eyes of the user 150 may allow to increase a reliability of the determination of the position data.

[0057] In an example, the apparatus 130 may be further configured to determine the position of the user 150 relative to the microscope 120 by determining a distance 148 between the user 150 and the microscope 120 and an angle c|) between the user 150 and the microscope 120 with respect to a viewing path 162 of the microscope 120. Determining the distance 148 and the angle (|) may allow to determine the deviation angle data with less computational effort. For example, the global position of the user 150 and the global position of the microscope 120 need not be known to determine the deviation angle data. As shown in Fig. la the deviation angle data can be determined based on the distance 148, the angle (|) and the position of the microscope 120 relative to the sample 110. For example, the position of the microscope 120 relative to the sample 110 can be determined based on a working distance of the microscope 120 as described below. Thus, information about a position of the user 150 relative to the sample 110 need not to be known. In this way, a determination of the deviation angle data can be facilitated.

[0058] In an example, the apparatus 130 may be configured to determine viewing path data indicative of the viewing path 152 of the user 150 based on the position of the head of the user 150 and / or the position of the eyes of the user 150. Further the apparatus 130 may be configured to determine the output data based on the viewing path data. Determining the viewing path data may allow to inform the user 150 about certain restrictions viewing usage of the optical imaging system 100. For example, an adjustment of the viewing path 162 of the microscope 120 may be restricted to certain areas in space. Thatis, the viewing path 162 of the microscope 120 cannot be set to an arbitrary value, for example. Thus, a misalignment between the viewing path 162 of the microscope 120 and the viewing path 152 of the user 150 may arise, which cannot be compensated by realignment of the microscope 120.

[0059] Therefore, the apparatus 130 may generate output data indicative of the restricted possibility to align the viewing path 162 of the microscope 120 and the viewing path 152 of the user 150. In this way, the user 150 can be informed about a non-compensable deviation. Thus, the user 150 can take certain countermeasures, e.g., change the position, to reduce the misalignment of the viewing path 162 of the microscope 120 and the viewing path 152 of the user 150.

[0060] In an example, the apparatus 130 may be configured to obtain working distance data indicative of a working distance of the microscope 120 and to determine the sample position data based on the working distance data. The working distance of a microscope refers to the distance between the front lens element of the objective lens and the sample being observed on the stage of the microscope. That is, the working distance is the space between the objective lens and the sample. For example, the apparatus 130 may retrieve a working distance of a currently used objective of the microscope 120 from a storage device, such as the one or more storage devices 136. For example, the working distance can be assumed as the actual distance 158 between the microscope 120 and the sample 110. For example, the sample position data may be obtained by obtaining the working distance data. Thus, the sample position data can be obtained in a facilitated way, especially without a need of a position sensor. That is, only the position data may be received and / or measured using a position sensor. In this way, data transfer and / or measurement efforts can be reduced.

[0061] In an example, the apparatus 130 may be configured to obtain movement data indicative of a movement of the microscope 120 and to generate a visual representation data indicative of a visual representation of the viewing path 162 of the microscope 120 and the viewing path 152 of the user 150. The visual representation data may be part of the output data. That is, the visual representation may be transmitted with the deviation angle data simultaneously. Alternatively, the visual representation data may be separated from the output data. That is, the visual representation data can be transmitted separately from the output data. The visual representation data can be used to inform the user 150 about a current misalignment. For example, the user 150 may perform a realignment of the microscope 120 by changing his position relative to the sample 110 (and thus the microscope 120). The visual representation data displayed on a display device may guide the user 150, such that the user 150 does not has to interact with the microscope 120, e.g., grab a handle of the microscope 120. In this way, the user 150 can improve the alignment of the viewing path 162 of the microscope 120 and the viewing path 152 of the user 150 without interacting with the microscope 120 by changing his position relative to the sample 110.

[0062] Optionally or alternatively, when a movement of the microscope 120 is detected, the visual representation data can be utilized to display the surgeon’s eye-hand coordination axis (i.e., the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120) to help the alignment. The movement of the microscope 120 can be detected in various ways, e.g., using an internal sensor, such like a gyroscope and / or an accelerometer, a release of arm break, a rapid change of the imaged objects within the FOV, and / or detection via an external sensor, e.g., indented for the detection of the surgeon such like the position sensor 124a. For example, if a movement of the microscope 120 is detected, the apparatus 130 may generate the visual representation data for informing the user 150 about the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120. The visual representation data, i.e., the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120 can be displayed on the display device. In this way, an alignment of the microscope 120 can be facilitated.

[0063] In an example, the apparatus 130 may be further configured to receive (second) sensor data of a position sensor 124a, 124b. The (second) sensor data is indicative of a position of the user 150. Further, the apparatus 130 may be configured to determine the position data based on the (second) sensor data. In this way, the apparatus 130 may obtain the position data in a facilitated way. For example, the position sensor 124a, e.g., a camera, may be external to the optical imaging system 100. For example, the position sensor 124a may be a camera integrated into the display device, e.g., a monitor and / or a head-mounted display. Optionally or alternatively, the position sensor 124b may be part of the optical imaging system 100, e.g., part of the microscope 120.

[0064] In an example, the position sensor 124a, 124b may be a camera, a time-of-flight sensor, a structured light projector and / or an optical imaging sensor 122a, 122b of the microscope 120. Thus, an appropriate sensor for measuring the (second) sensor data can be used.

[0065] In an example, the apparatus 130 may be configured to obtain threshold data indicative of a threshold for the deviation between the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120. Further, the apparatus 130 may be configured to compare the threshold data with the deviation angle data and determine the output data based on the threshold data. For example, the threshold data may be indicative of a threshold angle. That is, the threshold angle may be an angle of deviation between the viewing path 152 of the user 150 and the viewing path 162 of the microscope 120, for which a loss of the eye-hand coordination of the user 150 is likely. Thus, if the misalignment exceeds the threshold angle, i.e., angle 0 is larger than the threshold angle, the apparatus 130 may generate the output data for informing the user 150 about the exceedances the threshold angle. Thus, the user 150 can be informed about a misalignment if the misalignment may be likely to affect the user 150, especially the eye-hand coordination of the user 150. In an example, if the threshold for the deviation is exceeded, e.g., if the threshold angle is exceeded, the output data may be indicative of information about the exceedance of the threshold.

[0066] In an example, the apparatus 130 may be configured to receive the threshold data from an input device. For example, the user 150 can define the threshold using the input device. For example, the input device may be a keyboard, a touchpad and / or a mouse. In this way, the user 150 can adjust the threshold according to a personal need.

[0067] In an example, the optical imaging system 100 may further comprise a microscope 120. The position data may be determined based on (second) sensor data of the microscope 120. In this way, determination of the position data can be facilitated. For example, an additional (position) sensor (such as the position sensor 124a) to determine the position data can be omitted. For example, an optical imaging sensor 122a, 122b of the microscope 120 can be used to determine the position data. For example, the microscope 120 may be arranged such that the user 150 may be in the field of view of the microscope 120. In this way, the microscope 120 can acquire the image of the sample 110 and simultaneously measure the position data. Thus, costs of the setup can be reduced.

[0068] As shown in Fig. la the optional one or more interfaces 132 is coupled to the respective one or more processors 134 at the apparatus 130. In examples the one or more processors 134 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. Similar, the described functions of the one or more processors 134 may as well be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc. The one or more processors 134 is capable of controlling the one or more interfaces 132, so that any data transfer that occurs over the one or more interfaces 132 and / or any interaction in which the one or more interfaces 132 may be involved may be controlled by the one or more processors 134.

[0069] In an embodiment the apparatus 130 may comprise a memory, e.g., the one or more storage devices 136 and at least one or more processors 134 operably coupled to the memory and configured to perform the method described below.

[0070] In examples the one or more interfaces 132 may correspond to any means for obtaining, receiving, transmitting or providing analog or digital signals or information, e.g., any connector, contact, pin, register, input port, output port, conductor, lane, etc. which allows providing or obtaining a signal or information. The one or more interfaces 132 may be wireless or wireline and it may be configured to communicate, e.g., transmit or receive signals, information with further internal or external components.

[0071] The apparatus 130 may be a computer, processor, control unit, (field) programmable logic array ((F)PLA), (field) programmable gate array ((F)PGA), graphics processor unit (GPU), application-specific integrated circuit (ASICs), integrated circuits (IC) or system-on-a-chip (SoCs) system.

[0072] More details and aspects are mentioned in connection with the examples described below. The example shown in Fig. 1 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described below (e.g., Fig. 2 - 4).

[0073] Figs. 2a-2c show different examples of microscope-user alignment. In Fig. 2a the viewing path 252 of the user 250 is aligned (perfect alignment level) with the viewing path 262 of the microscope 220. In this case, the apparatus (as described above, e.g., with reference to Fig. 1) may generate no output data. Fig. 2b shows an acceptable alignment (acceptable alignment level) of the viewing path 252 of the user 250 and the viewing path 262 of the microscope 220. The alignment is not as good as in Fig. 2a but may be still good enough to provide eyehand coordination of the user 250. Thus, the apparatus 130 may not generate the output data.

[0074] Fig. 2C shows an insufficient alignment (bad alignment level) of the viewing path 252 of the user 250 and the viewing path 262 of the microscope 220. Thus, the user 250 may loss the eye-hand coordination. Therefore, the apparatus may generate the output data to inform the user 250 about the misalignment. In this way, the user 250 can take certain measures, e.g., realign the microscope 220 and / or pause certain (operation) tasks.

[0075] More details and aspects are mentioned in connection with the examples described above and / or below. The example shown in Fig. 2 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1) and / or below (e.g., Fig. 3 - 4). Fig. 3 shows a flow chart of an example of a method 300 for an optical imaging system. The method 300 comprises obtaining 310 position data indicative of a position of a microscope of the optical imaging system relative to a user of the optical imaging system. The method 300 further comprises obtaining 320 sample position data indicative of a distance between the microscope and the sample. Further, the method 300 comprises determining 330, based on the position data and the sample position data, deviation angle data indicative of a deviation of a viewing path of the user on the sample and a viewing path of the microscope on the sample in an image plane of the optical imaging system. The method 300 further comprises determining 340, based on the deviation angle data, output data for informing the user about the deviation of the viewing path of the user on the sample and the viewing path of the microscope on the sample and transmitting 350 a display signal indicative of the output data. The method 300 can be performed by an apparatus as described with reference to Fig. 1.

[0076] More details and aspects are mentioned in connection with the examples described above and / or below. The example shown in Fig. 3 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1 - 2) and / or below (e.g., Fig. 4).

[0077] Some embodiments relate to a microscope comprising an apparatus as described in connection with Fig. 1. Alternatively, a microscope may comprise of or can be communicatively connected to an apparatus as described in connection with Fig. 1. Fig. 4 shows a schematic illustration of a system 400, e.g., an optical imaging system, configured to perform a method described herein, e.g., with reference to Fig. 5. The system 400 comprises a microscope 410 and a computer system 420. The microscope may comprise the apparatus as described above, e.g., with reference to Fig. 1. The microscope 410 is configured to take images and is connected to the computer system 420. The computer system 420 is configured to execute at least a part of a method described herein. The computer system 420 may be configured to execute a machine learning algorithm. The computer system 420 and microscope 410 may be separate entities but can also be integrated together in one common housing. The computer system 420 may be part of a central processing system of the microscope 410 and / or the computer system 420 may be part of a subcomponent of the microscope 410, such as a sensor, an actor, a camera or an illumination unit, etc. of the microscope 410. The computer system 420 may be a local computer device (e.g., personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 420 may comprise any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g., camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 420 may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 420 may include one or more storage devices, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system 420.

[0078] More details and aspects are mentioned in connection with the examples described above. The example shown in Fig. 4 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1 - 3).

[0079] Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0080] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non- transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[0081] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0082] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0083] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0084] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0085] A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium. A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

[0086] A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

[0087] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0088] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0089] In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[0090] If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method and vice versa. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system.

[0091] The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Furthermore, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

[0092] The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the fur- ther example.

[0093] List of reference Signs

[0094] (|) angle between microscope and user

[0095] 0 angle between viewing path microscope and viewing path user

[0096] 100 optical imaging system

[0097] 110 sample

[0098] 120 microscope

[0099] 122a, 122b optical imaging sensor

[0100] 124a, 124b position sensor

[0101] 130 apparatus

[0102] 132 interface

[0103] 134 processor

[0104] 136 storage device

[0105] 148 distance microscope to user

[0106] 150 user

[0107] 152 viewing path of the user

[0108] 158 distance sample to microscope

[0109] 162 viewing path of the microscope

[0110] 170 image plane

[0111] 220 microscope

[0112] 250 user

[0113] 252 viewing path of the user

[0114] 262 viewing path of the microscope

[0115] 300 method for an optical imaging system

[0116] 310 obtaining position data

[0117] 320 obtaining sample data

[0118] 330 determining deviation angle data

[0119] 340 determining output data

[0120] 350 transmit a display signal

[0121] 400 system

[0122] 410 microscope

[0123] 420 computer system

Claims

Claims1. An apparatus (130) for an optical imaging system (100), comprising one or more processors (134) and one or more storage devices (136), wherein the apparatus (130) is configured to: obtain sample position data indicative of a position (158) of the microscope (120) relative to a sample (110); obtain position data indicative of a position of a microscope (120) of the optical imaging system (100) relative to a user (150) of the optical imaging system (100); determine, based on the position data and the sample position data, deviation angle data indicative of a deviation of a viewing path (152) of the user (150) on the sample (110) and a viewing path (162) of the microscope (120) on the sample (110) in an image plane (170) of the optical imaging system (100); determine, based on the deviation angle data, output data for informing the user (150) about the deviation of the viewing path (152) of the user (150) on the sample (110) and the viewing path (162) of the microscope (120) on the sample (110); and transmit a display signal indicative of the output data.

2. The apparatus (130) according to claim 1, wherein the apparatus (130) is configured to obtain the position data by obtaining at least one of a position of a head of the user (150) relative to the microscope (120) and a position of eyes of the user (150).

3. The apparatus (130) according to any one of the preceding claims, wherein the apparatus is further configured to determine, based on the position data, the position of the user relative to the microscope (120) by determining a distance (148) between the user and the microscope (120) and an angle (<p) between the user (150) and the microscope (120) with respect to the viewing path (162) of the microscope (120).

4. The apparatus (130) according to claim 2 or 3, wherein the apparatus (130) is configured todetermine, based on at least one of the position of the head of the user (150) and the position of the eyes of the user (150), viewing path data indicative of the viewing path (152) of the user (150); and determine, based on the viewing path data, the output data.

5. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is further configured to obtain working distance data indicative of a working distance of the microscope (120); and determine the sample position data based on the working distance data.

6. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is further configured to obtain movement data indicative of a movement of the microscope (120); and generate visual representation data indicative of a visual representation of the viewing path (162) of the microscope (120) and the viewing path (152) of the user (150); and transmit a second display signal indicative of the visual representation for displaying the viewing path (162) of the microscope (120) and the viewing path (152) of the user (150).

7. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is further configured to receive sensor data of a position sensor (124), the sensor data indicative of a position of the user (150); and determine, based on the sensor data, the position data.

8. The apparatus (130) according to claim 7, wherein the position sensor (124) is at least one sensor of a camera, a time-of-flight sensor, a structured light projector and an optical imaging sensor (122a; 122b) of the microscope (120).

9. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured toobtain threshold data indicative of a threshold for the deviation between the viewing path (152) of the user and the viewing path (162) of the microscope (120); compare the threshold data with the deviation angle data; and determine the output data based on the threshold data.

10. The apparatus (130) according to claim 9, wherein if the threshold for the deviation is exceeded, the output data is indicative of information about the exceedance of the threshold.

11. The apparatus (130) according to claim 9 or 10, wherein the apparatus (130) is configured to receive the threshold data from an input device.

12. An optical imaging system (100), comprising an apparatus according to any one of the preceding claims.

13. The optical imaging system (100) according to claim 12, further comprising a microscope (120), wherein the position data is determined based on sensor data of the microscope (120).

14. A method (300), comprising: obtaining (310) position data indicative of a position of a microscope of the optical imaging system relative to a user of the optical imaging system; obtaining (320) sample position data indicative of a distance between the microscope and the sample; determining (330), based on the position data and the sample position data, deviation angle data indicative of a deviation of a viewing path of the user on the sample and a viewing path of the microscope on the sample in an image plane of the optical imaging system; determining (340), based on the deviation angle data, output data for informing the user about the deviation of the viewing path of the user on the sample and the viewing path of the microscope on the sample; and transmitting (350) a display signal indicative of the output data.

15. A computer program with a program code for performing the method (300) according to claim 14 when the computer program is executed on a processor.

Images