Apparatus, method and computer program for controlling a microscope system

Abstract

The embodiments relate to an apparatus, method, computer program product, and corresponding microscope system for controlling a microscope system. The apparatus for controlling the microscope system includes an interface for communicating with a camera module. The camera module is adapted to provide camera image data of a head of a user of the microscope system. The apparatus includes a processing module configured to acquire the camera image data from the camera module via the interface. The processing module is configured to process the camera image data to determine information regarding an angular orientation of the user's head relative to a display of the microscope system. The processing module is configured to provide a control signal for a robotic adjustment system of the microscope system based on the information regarding the angular orientation of the user's head.

Description

[Technical Field]

[0001] The embodiments relate to an apparatus, a method, a computer program for controlling a microscope system, as well as a corresponding microscope system. [Background technology]

[0002] Surgical microscope systems are often designed to move the optical carrier (i.e., the surgical microscope) as easily and smoothly as possible, ideally providing a sense of weightlessness. In some cases, a robotic adjustment system, such as a robotic arm, is used to move the surgical microscope. Movement is often achieved using knobs and buttons, which are used to individually control the surgical microscope's position, surgical microscope angle, and other functions, such as zoom level and lighting. Due to the many adjustable factors for achieving a desired surgical microscope field of view, controlling the robotic adjustment system can take up a surgeon's time and attention. U.S. Pat. No. 5,345,087 discloses a light guide system for spatially positioning a surgical microscope, providing contactless instrument guidance. In this patent, a light source is attached to the surgeon's head, which is detected to determine the surgeon's head orientation and control a motorized adjustment mechanism accordingly. The head is then brought to a predetermined position during an initialization phase, and head movement relative to this predetermined position is tracked. This provides basic contactless control of the motorized adjustment mechanism without taking into account the mechanisms used by humans to adopt different viewpoints.

[0003] There may be a need for an improved microscope system that improves the control of the robotic adjustment system. Summary of the Invention [Means for solving the problem]

[0004] Embodiments of the present disclosure are based on the discovery that instead of determining the viewpoint a surgeon desires to achieve when performing head movements, other systems provide only purely mechanical movements equivalent to the surgeon's head movements. In embodiments, the angular orientation of the user's head is determined relative to a point, i.e., the display or relative to the surgical site, and this angular orientation is used to determine from which viewpoint the operator intends to view that point and how to adjust the robotic adjustment system to achieve this desired viewpoint change. Thus, in embodiments, head movements, gestures, and facial expressions can be used to control the microscope, thereby providing head tracking for intuitive robotic control. In one embodiment, the microscope alignment is adjusted by the robot following the user's head.

[0005] An embodiment of the present disclosure provides an apparatus for controlling a microscope system. The apparatus includes an interface for communicating with a camera module. The camera module is adapted to provide camera image data of a user's head to the microscope system. The apparatus includes a processing module configured to acquire the camera image data from the camera module via the interface. The processing module is configured to process the camera image data to determine information regarding an angular orientation of the user's head relative to a display of the microscope system. The processing module is configured to provide a control signal for a robotic adjustment system of the microscope system based on the information regarding the angular orientation of the user's head. Determining the angular orientation of the user's head relative to the display of the microscope system allows changes in the head orientation relative to the display to be translated into changes in the viewpoint of the microscope system in an easily understandable format, thereby improving control of the microscope system by the user / surgeon of the microscope system.

[0006] In at least some embodiments, the processing module is configured to determine a viewing angle of a user of the microscope system toward the display based on the camera image data. The processing module may be configured to provide a control signal based on the user's viewing angle, which can be used to adjust the robotic arm so that the microscope provides a field of view derived from the viewing angle.

[0007] For example, the processing module may be configured to determine the user's field of view angle based on the angular orientation of the user's head, which can provide a coarse-grained determination of the field of view angle.

[0008] According to some embodiments, the processing module is configured to perform eye tracking of one or both of the user's eyes. The processing module may be configured to determine the user's field of view based on the angular orientation of the user's head and the eye tracking, which may provide a finer granularity in determining the field of view.

[0009] The processing module may be configured to determine information regarding angular orientation using image analysis on the camera image data, thereby enabling angular orientation to be determined without requiring the user to wear additional tracking markers.

[0010] In some embodiments, the display of the microscope system is an auxiliary display located adjacent to the eyepiece of the microscope of the microscope system, allowing the surgeon to make in situ adjustments of the robotic arm without having to leave the surgical field.

[0011] Alternatively, the display of the microscope system may be an auxiliary display located on the base unit of the microscope system, which may allow for the use of a larger display that can be used in preparation for a surgical procedure or by additional surgical staff.

[0012] In some embodiments, the display of the microscope system may be an eyepiece display in the eyepiece of the microscope of the microscope system, which again may allow the surgeon to make in situ adjustments of the robotic arm without having to leave the surgical site.

[0013] In various embodiments, the processing module is configured to determine a user's gaze at a single point based on the camera image data. The processing module may be configured to determine control signals for a robotic coordination system such that a (central) viewpoint of a microscope of the microscope system remains fixed at the single point. The (central) single point (e.g., focus of gaze) may serve as a visual anchor for the coordination of the robotic arm.

[0014] The processing module may be configured to determine, based on the camera images, information regarding the position of the user's head and information regarding the angle of the user's head. The processing module may be configured to determine, based on the information regarding the position of the user's head and information regarding the angle of the user's head, information regarding the angular orientation of the user's head. Because the angular orientation is defined relative to the display, both the position and the angle may contribute to the determination of the angular orientation (e.g., absolute or relative to the display).

[0015] For example, the processing module may be configured to generate a control signal such that a pan of the microscope field of view of the microscope system is triggered by the robotic adjustment system when the position of the user's head changes without causing a corresponding change in the angle of the user's head, which can provide a natural translation between head position movement and robotic arm direction.

[0016] Additionally or alternatively, the processing module may be configured to generate a control signal such that an adjustment of the viewing angle of the microscope of the microscope system is triggered by the robotic adjustment system when the position of the user's head changes with a corresponding change in the user's head angle, thereby providing an easy-to-understand translation between changes in head angle and the direction of the robotic arm.

[0017] In some embodiments, the interface is adapted to receive a trigger signal from an input device of the microscope system. The processing module may be configured to provide the control signal in response to the trigger signal of the input device. Thus, the control signal can only be provided when explicitly triggered by a user, thereby avoiding undesired adjustments of the robotic arm.

[0018] For example, the control signal may be configured to adjust at least one of the spatial position of the microscope, the observation angle of the microscope, the zoom of the microscope, the working distance of the microscope, and the illumination of the microscope of the microscope system. For example, multiple elements of the above group may be adjusted simultaneously, thereby providing adjustment of various characteristics of the microscope.

[0019] An embodiment further provides another device for controlling a microscope system. The device includes an interface for communicating with a camera module. The camera module is adapted to provide camera image data of a user's head to the microscope system. The device includes a processing module configured to acquire the camera image data from the camera module via the interface. The processing module is configured to process the camera image data to determine information about the angular orientation of the user's head and to determine the user's gaze at a point. The processing module is configured to provide a control signal for a robotic adjustment system of the microscope system based on the information about the angular orientation of the user's head so that the (central) viewpoint of the microscope of the microscope system remains fixed at the point. By determining the angular orientation of the user's head, changes in the head orientation relative to the display can be organically translated into changes in the viewpoint of the microscope system, thereby improving interaction between the surgeon and the microscope system. The user's gaze at the point can provide a visual anchor for adjusting the robotic arm.

[0020] For example, the processing module may be configured to determine information regarding the angular orientation of the microscope user's head relative to the microscope system's display, which in turn may be used as a reference point for determining the angular orientation, providing a reference for the surgeon in adjusting the robotic arm.

[0021] Alternatively, the microscope system may be a surgical microscope system, and the processing module may be configured to determine information regarding the angular orientation of the microscope user's head relative to a surgical site viewed by a surgical microscope of the surgical microscope system. In this case, the surgeon can control the robotic arm by adjusting his or her own angular orientation toward the surgical site, which is then translated to provide an equivalent orientation of the microscope.

[0022] Embodiments of the present disclosure further provide a microscope system that includes one or both of the above devices.

[0023] An embodiment of the present disclosure further provides a method for controlling a microscope system. The method includes acquiring camera image data of a head of a user of the microscope system from a camera module of the microscope system. The method includes processing the camera image data to determine information regarding an angular orientation of the user's head relative to a display of the microscope system. The method includes providing a control signal for a robotic adjustment system of the microscope system based on the information regarding the angular orientation of the user's head.

[0024] An embodiment of the present disclosure further provides a method for controlling a microscope system. The method includes acquiring camera image data of a head of a user of the microscope system from a camera module of the microscope system. The method includes processing the camera image data to determine information about an angular orientation of the user's head and to determine the user's gaze at a point. The method includes providing a control signal for a robotic adjustment system of the microscope system based on the information about the angular orientation of the user's head such that a (center) viewpoint of a microscope of the microscope system remains fixed at the point.

[0025] An embodiment of the present disclosure further provides a computer program comprising program code for performing at least one of the above methods when the computer program is run on a processor.

[0026] Some embodiments of the apparatus and / or methods will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0027] [Figure 1a] FIG. 1 is a block diagram illustrating an embodiment of an apparatus for controlling a microscope system. [Figure 1b] 1 is a block diagram illustrating an embodiment of a microscope system including an apparatus for controlling the microscope system. [Figure 2] 1 is a flow chart illustrating an embodiment of a method for controlling a microscope system. [Figure 3] 10 is a flow chart illustrating another embodiment of a method for controlling a microscope system. [Figure 4] FIG. 1 is a schematic diagram illustrating a system including a microscope and a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0028] Various embodiments will now be described in more detail with reference to the accompanying drawings, in which several embodiments are shown, in which line thicknesses, layer thicknesses and / or area sizes may be exaggerated for clarity.

[0029] 1a and 1b show block diagrams of an embodiment of an apparatus 110 for controlling a microscope system 100. The apparatus includes an interface 112 in communication with a camera module 120. The camera module is adapted to provide camera image data of a head of a user of the microscope system. The apparatus includes a processing module 114 coupled to the interface 112. The processing module 114 is configured to obtain the camera image data from the camera module via the interface.

[0030] The camera image data is then processed to determine information regarding the angular orientation of the user's head relative to the display or relative to a point at which the user's gaze is fixed (on the display or at the surgical site). Accordingly, in some embodiments, the processing module 114 is configured to process the camera image data to determine information regarding the angular orientation of the user's head relative to the display 140 of the microscope system. In this case, the processing module is configured to provide a control signal for the robotic adjustment system 130 of the microscope system based on the information regarding the angular orientation of the user's head. Alternatively or additionally, the processing module 114 may be configured to process the camera image data to determine information regarding the angular orientation of the user's head and determine the user's gaze relative to a point. In this case, the processing module is configured to provide a control signal for the robotic adjustment system 130 of the microscope system based on the information regarding the angular orientation of the user's head so that the central viewpoint of the microscope 150 of the microscope system remains fixed at the point.

[0031] FIG. 1b further shows a microscope system 100 comprising a device 110, a camera module 120, a robotic adjustment system 130, an optional display 140, a further optional microscope 150, and a further optional input device 160.

[0032] The following description relates to the control device 110 and microscope system 100 of FIGS. 1a and 1b and the corresponding method of FIGS.

[0033] Embodiments of the present disclosure relate to devices, methods, and computer programs for controlling a microscope system 100. Generally, a microscope is an optical instrument suitable for inspecting objects too small for human visual inspection (solely). For example, a microscope can provide optical magnification of an object, such as an organic tissue sample. A surgical microscope is a microscope used during surgery, i.e., a microscope suitable for use (by a surgeon) during surgery. Such surgical microscope systems often include an arm or positioning means used to position the surgical microscope as desired by the surgeon, for example, near the surgical space, so that the surgical microscope can be used to provide a magnified view of a vascular bundle or tissue. Such an arm or positioning means is typically adapted to provide a wide variety of different positions and / or angles to provide the surgeon with sufficient space to perform the surgery while providing a magnified image of the surgical site. Furthermore, the arm or positioning means of a surgical microscope system is often designed to allow for easy positioning of the surgical microscope, for example, using weights and counterweights, pneumatic systems, or motors to assist the movement of the arm or positioning means of the surgical microscope system. As described above, many microscope systems, such as surgical microscope systems, have a robotic adjustment system, such as a robotic arm, for positioning the microscope of the microscope system relative to a sample being observed using the microscope system. In other words, the robotic adjustment system may be configured to adjust the arm of the microscope system. For example, the robotic adjustment system may be a robotic arm of a (surgical) microscope system. Such robotic adjustment systems often provide multiple degrees of freedom for adjusting the system, often allowing adjustment of the angular orientation of the microscope relative to the sample as well as the three-dimensional position of the microscope. Therefore, adjustments to the robotic arm are often not straightforward.In some systems, a user / operator, e.g., a surgeon, can manually reposition the microscope by grasping a handle attached to the microscope, assisted in this case by a robotic adjustment system operating as a servo motor, thereby facilitating movement of the often heavy machinery. Note that in embodiments, control of the robotic adjustment system is selected based on the angular orientation of the user's head derived by processing camera image data. In other words, the microscope system may be a surgical microscope system having a robotic adjustment system, such as a robotic arm, that is controllable based on the angular orientation of the user's head.

[0034] The microscope system includes a camera module 120 suitable for providing camera image data of a user's head. In other words, the camera module 120 may be configured to provide camera image data of a user's head for the microscope system. Generally, the camera module may include an active pixel sensor (APS) or charge-coupled device (CCD) imaging sensor module. For example, an APS imaging sensor module uses a light-receiving element and an active pixel amplifier to record light at each pixel. APS imaging sensor modules are often based on complementary metal-oxide semiconductor (CMOS) or scientific CMOS (S-CMOS) technology. In a CCD imaging sensor module, incident photons are converted into electron charges at a semiconductor-oxide interface, which are then transferred between capacitive bins within the imaging sensor module by the sensor imaging module's control circuitry to perform imaging. Alternatively or additionally, the camera module may be a depth-sensing camera module or may include a depth-sensing camera including a depth sensor suitable for providing depth-sensing camera module images. Thus, the camera image data may be depth-sensing camera image data or may include a two-dimensional component and a depth-sensing component. For example, the camera module may include a depth sensor, such as a time-of-flight depth sensor or a structured-light depth sensor. The camera image data may include two-dimensional camera image data of the user's head and / or three-dimensional camera image data of the user's head.

[0035] The processing module 114 is configured to process the camera image data to determine information regarding the angular orientation of the user's head, for example, relative to the display 140 or the surgical site. Generally, the angular orientation of the user's head may include the angle of the head relative to a reference point, such as the resting position of the head or the display 140 or the surgical site. Furthermore, the angular orientation may include the position of the head relative to a coordinate system, which may be relative to the microscope system, the display, or the surgical site. In other words, the processing module may be configured to determine information regarding the position of the user's head and information regarding the angle of the user's head (e.g., angle relative to a reference point) based on the camera images. The processing module may be configured to determine information regarding the angular orientation of the user's head based on the information regarding the position of the user's head and the information regarding the angle of the user's head. The information regarding the angular orientation may include a numerical value of the head angle (relative to the reference point) and / or the head position.

[0036] The processing of the camera image data may be performed using image analysis. In other words, the processing module may be configured to determine information related to the angular orientation using image analysis of the camera image data. For example, the image analysis may be used to identify one or more features of the head in the camera image data. For example, the processing module may be configured to determine the angular orientation of the user's head by determining a contour of the user's head, identifying the eyes and / or nose in the (two-dimensional or three-dimensional) camera image data, and deriving the angular orientation of the user's head based on the head contour and the position of the eyes and / or nose relative to the head contour. The image analysis may be based on the (natural) contour of the head. For example, the image analysis may not be based on tracking markers attached to the user's head. Conversely, the processing module may be configured to determine information related to the angular orientation of the user's head without identifying (artificially added) tracking markers.

[0037] In various embodiments, information regarding the angular orientation of the user's head relative to a display 140 of the microscope system is determined. For example, the display 140 may be an electronic output device that presents information in a visual form. For example, the display 140 may be suitable and / or configured to electronically display an image of a sample observed through a microscope, such as an image of a surgical site observed through a surgical microscope. The processing module 114 may be configured to provide the image of the sample to be observed to the display 114. In at least some embodiments, the image is based on separate camera image data from another camera (not shown) of the microscope 150. For example, the processing module 114 may be configured to obtain separate camera image data from the separate camera and generate the image for the display 114. For example, the display 140 may be a liquid crystal display (LCD)-based display or an organic light-emitting diode (OLED)-based display. In some embodiments, the display may be located on or within an eyepiece of the microscope 150 of the microscope system. For example, the display may be located above or integrated into the microscope eyepiece. In other words, the display of the microscope system may be auxiliary display 140a disposed adjacent to the eyepiece of the microscope of the microscope system. For example, displays 140a and 140b may be attached to microscope 150 of the microscope system. For example, the display of the microscope system may be eyepiece display 140b in the eyepiece of the microscope of the microscope system. Alternatively, the display of the microscope system may be auxiliary display 140c disposed in the base unit of the microscope system (e.g., a stand including device 110). For example, display 140b may be attached to the base unit of the microscope system via another arm 145.

[0038] As described above, in some embodiments, the angular orientation of the user's head is determined relative to the display 140 or relative to the surgical site viewed by the surgical microscope. For example, this may be achieved by determining the user's viewing angle toward the display or the surgical site. In other words, the processing module may be configured to determine the user's viewing angle of the microscope system toward the display (or toward the surgical site) based on the camera image data. For example, the processing module may be configured to derive the user's viewing angle toward the display or the surgical site based on the contour of the head, the position of the eyes and / or nose relative to the contour of the head, and the position of the user's head. For example, the processing module may be configured to determine the user's viewing angle based on the angular orientation of the user's head. For example, the angular orientation of the user's head may include the user's viewing angle toward the display or the surgical site, e.g., when an angular component of the angular orientation is determined relative to the display or the surgical site as a reference point. In some embodiments, this may be sufficient to determine the viewing angle with sufficient accuracy for performing adjustments of the robotic adjustment system. However, in some embodiments, the processing module may be configured to perform eye tracking to more accurately learn where the user (i.e., the surgeon) is looking. In other words, the processing module may be configured to perform eye tracking on one or both of the user's eyes. For example, the processing module may be configured to perform image processing on camera image data to perform eye tracking on one or both of the user's eyes. Alternatively, image processing may be performed on additional camera image data, for example, from a dedicated eye-tracking camera, in addition to the three-dimensional camera image data used to determine the angular orientation. The processing module 114 may be configured to determine the user's field of view based on the angular orientation of the user's head and the eye tracking. In other words, the processing module 114 may be configured to fine-tune the determined user's field of view based on the eye tracking.

[0039] The processing module may be configured to provide a control signal based on the user's viewing angle. For example, the processing module 114 may be configured to convert the user's viewing angle into a corresponding orientation of the microscope 150 and provide a control signal to cause an action on the corresponding orientation of the microscope 150. For example, the control signal may be provided to cause the robotic adjustment system to cause the microscope 150 to position the microscope 150 at an angle based on the user's viewing angle, e.g., an angle corresponding to the user's viewing angle.

[0040] When changing the positioning of the microscope, a single point (of the specimen or surgical site) can be used as an “anchor point” that remains fixed while the viewing angle or distance / zoom level is changed, e.g., so that a user can focus on a point of interest and view it from various angles. Thus, the user's gaze at the single point can be determined, and this single point can serve as the anchor point for adjusting the microscope (via the robotic adjustment module). In other words, the processing module can be configured to determine the user's gaze at a single point (e.g., a point on the screen, a point on the specimen / surgical site shown on the screen, or a point on the surgical site) based on camera image data. The processing module can be configured to process the camera image data to determine the user's gaze at the single point. For example, the processing module can be configured to determine the user's gaze at the single point using eye tracking, e.g., an angular orientation of the user's head associated with the eye tracking. The processing module can be configured to determine a control signal for the robotic adjustment system such that the central viewpoint of the view of the microscope 150 of the microscope system remains fixed at the single point. In other words, the processing module may be configured to determine control signals for the robotic coordination system such that the point remains at a similar position (eg, the same position) within the field of view of the microscope 150.

[0041] In general, different approaches can be adopted depending on whether the user simply changes their head position or whether they change their head angle. For example, if the user only changes their head position without changing the angle (or with an angle change less than a threshold, e.g., less than 5°), a control signal can be generated to pan (i.e., scroll horizontally and / or vertically) the microscope's field of view across the sample field / surgical field. In other words, the processing module can be configured to generate a control signal such that panning of the microscope's field of view of the microscope of the microscope system is triggered by the robotic adjustment system when the user's head position changes without causing a corresponding change in the user's head angle. If only the head angle changes, the field of view can be adjusted without panning. In other words, the processing module can be configured to generate a control signal such that adjustment of the microscope's field of view of the microscope of the microscope system is triggered by the robotic adjustment system when the user's head position changes with a corresponding change in the user's head angle. Again, the processing module may be configured to determine control signals for the robotic coordination system so that the central viewpoint of the microscope 150 of the microscope system remains fixed at a single point within the field of view, e.g., based on a single point at which the user is gazing or a central point of the field of view. In other words, the processing module may be configured to determine control signals for the robotic coordination system so that the single point at which the user is gazing or a central point of the field of view remains at a similar (e.g., the same) position within the field of view of the microscope 150. In this case, panning of the field of view may also be performed by changing the viewing angle, e.g., based on the single point. Furthermore, according to some embodiments, the processing module may be configured to determine information regarding the position, posture, and / or movement of the user's shoulders based on the camera image data, and to generate control signals using the information regarding the position, posture, and / or movement of the shoulders.

[0042] The processing module is configured to provide control signals for the robotic adjustment system 130 of the microscope system based on information regarding the angular orientation of the user's head. As described above, the processing module 114 can be configured to translate the user's viewing angle into a corresponding orientation of the microscope 150 and provide control signals to effect an action on the corresponding orientation of the microscope 150. Various characteristics of the robotic adjustment system can be adjusted to effect an action on the orientation of the microscope 150. For example, the control signal can be configured to adjust at least one of the spatial position, the observation angle of the microscope, the zoom of the microscope, the working distance of the microscope, and the illumination of the microscope of the microscope system. For example, the control signal can be configured to adjust the spatial position and thus the working distance of the microscope 150 by controlling the robotic adjustment system such that the spatial position of the microscope is changed, e.g., by controlling a robotic joint of a robotic arm, to obtain a desired spatial position. The control signal can be configured to adjust the observation angle of the microscope by controlling the robotic adjustment system, which changes the spatial position of the microscope and changes the angle of the microscope toward the sample or surgical site to obtain the observation angle. The control signal may further be configured to control the microscope to change the zoom and / or illumination of the microscope, and therefore the control signal may also be provided to the microscope 150 or an illumination unit of the microscope system 150.

[0043] In some cases, it may be desirable to limit the provision of control signals to situations in which the user / surgeon intends to reposition the microscope and to refrain from generating or providing control signals outside of these situations. To achieve this, the user / surgeon can provide a trigger via the input device to activate (or deactivate) the generation / provision of control signals. Thus, the interface may be adapted to receive a trigger signal from the microscope system's input device 160. For example, the input device may be one of a button, a touch-based interface (such as a touchscreen or capacitive switch), an optical input device (e.g., the input device is activated by interrupting the path between the light source and the receiver), an ultrasonic input device (e.g., the input device is activated by bringing a hand or object close to the ultrasonic input device), and a voice-activated input device. The processing module may be configured to provide the control signal in response to a trigger signal from the input device. In other words, the processing module may be configured to provide the control signal (only) when (and / or while) triggered by the input device.

[0044] The interface 112 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, the interface 112 may comprise an interface circuit configured to receive and / or transmit information. In an embodiment, the processing module 114 may be implemented using one or more processing units, one or more processing devices, any processing means, e.g., processors, computers, or programmable hardware components operable with correspondingly adapted software. In other words, the functionality described for the processing module 114 may be implemented in software, where the software is executed in one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc.

[0045] Further details and aspects of the microscope system or the device for controlling the microscope system are mentioned in relation to the proposed concept or one or more embodiments described above or below (e.g., FIGS. 2-4). The microscope system or the device for controlling the microscope system may also include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more embodiments described above or below.

[0046] 2 shows a flowchart of one embodiment of a (corresponding) method for controlling a microscope system. For example, the microscope system can be implemented similarly to the microscope system of FIG. 1b. The method includes acquiring 210 camera image data of a head of a user of the microscope system from a camera module of the microscope system. The method includes processing 220 the camera image data to determine information regarding an angular orientation of the user's head relative to a display 140 of the microscope system. The method includes providing 230 a control signal for a robotic adjustment system of the microscope system based on the information regarding the angular orientation of the user's head.

[0047] As indicated above, the features described in connection with the apparatus 110 and microscope system 100 of FIGS. 1a and / or 1b are equally applicable to the method of FIG.

[0048] Further details and aspects of the method are referred to in relation to the proposed concept or one or more of the embodiments described above or below (e.g., FIG. 1 or FIGS. 3-4). The method may also include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0049] 3 shows a flowchart of a further embodiment of a (corresponding) method for controlling a microscope system. For example, the microscope system can be implemented similarly to the microscope system of FIG. 1b. The method includes acquiring 310 camera image data of a head of a user of the microscope system from a camera module of the microscope system. The method includes processing 320 the camera image data to determine information about the angular orientation of the user's head and to determine the user's gaze at a point. The method further includes providing 330 a control signal for a robotic adjustment system of the microscope system based on the information about the angular orientation of the user's head, such that a central viewpoint of the microscope of the microscope system remains fixed at the point.

[0050] As noted above, the features described in connection with the apparatus 110 and microscope system 100 of FIGS. 1a and / or 1b can be applied to the method of FIG. 3 as well.

[0051] Further details and aspects of the method are mentioned in relation to the proposed concept or one or more of the examples described above or below (e.g., Figures 1, 2, or 4). The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the examples described above or below.

[0052] At least some embodiments involve the use of an imaging system mounted on a robotic arm. Using an imaging system (e.g., a surgical microscope) mounted on a robotic arm allows the imaging system to move in space (typically with six degrees of freedom), which allows for many adjustments of the imaging settings, e.g. 1. Spatial position x, y, z (3 degrees of freedom) 2. Observation angle θ, φ, ω (3 degrees of freedom) 3.Zoom, working distance, lighting / sensitivity (3 degrees of freedom) This becomes possible.

[0053] Such adjustments can be complicated when performed with traditional knob / button adjustments and can require time and user attention. Furthermore, complex adjustments, such as point locking (gazing at the observation center while changing the observation angle), can be required, which can require combined adjustment of parameters (e.g., positioning relative to the sphere while adapting the observation angle accordingly).

[0054] However, human vision is mechanically / geometrically similar to robotic imaging systems, and humans intuitively adjust the observation geometry to perceive the shape of an object. Embodiments provide a control system (e.g., device 110 of FIGS. 1a and / or 1b) that allows humans to control a robotic imaging system (e.g., microscope system 100 of FIGS. 1a and / or 1b) in an intuitive manner that resembles natural observation movements / gestures. In particular, embodiments focus on recognizing human posture and head movements (e.g., angular head orientation), which can be interpreted as a guide for positioning the robotic imaging system (e.g., by providing control signals). Such a system may be any combination of the following concepts: 1. Replicate the observation geometry of the human head, i.e., head position (x,y,z) and angles (θ,φ,ω) (i.e., angular orientation). Additionally, eye-tracking information can be included. 2. Interpretation of human head movements / gestures to make certain adjustments. For example, if the head is moving in space (x,y,z) without rotation, this can mean a pan of the field of view, while if the (x,y,z) movement is combined with a counterbalance rotation of the head (θ,φ,ω), this can be interpreted as a point-locked adjustment. Additionally, eye tracking and shoulder position can also be used for more robust interaction.

[0055] Further details and aspects of the concept are referenced in relation to the proposed concept or one or more of the embodiments described above or below (e.g., FIGS. 1-3), and the concept may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0056] Some embodiments relate to a microscope including a system such as that described in connection with one or more of FIGS. 1-3. Alternatively, the microscope may be part of or connected to a system such as that described in connection with one or more of FIGS. 1-3. FIG. 4 shows a schematic diagram of a system 400 configured to perform the methods described herein. The system 400 includes a microscope 410, which may correspond to the microscope 150 of FIG. 1b, and a computer system 420, which may correspond to the device 110 of FIGS. 1a and / or 1b. The microscope 410 is configured for imaging and is connected to the computer system 420. The computer system 420 is configured to perform at least a portion of the methods described herein. The computer system 420 may be configured to perform a machine learning algorithm. The computer system 420 and the microscope 410 may be separate or may be integrated into a common housing. The computer system 420 may be part of the central processing system of the microscope 410 and / or may be part of a subordinate component of the microscope 410, such as a sensor, actor, camera or lighting unit of the microscope 410.

[0057] The computer system 420 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuits that may be included in computer system 420 may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 420 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.Computer system 420 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touch screen, voice recognition device, or any other device that allows a user of the system to input information to and receive information from computer system 420.

[0058] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry. In some embodiments, any one or more of the critical steps may be performed by such an apparatus.

[0059] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation may be performed by a non-transitory storage medium, such as a digital storage medium, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Therefore, the digital storage medium may be computer-readable.

[0060] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

[0061] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code that is operative to perform any of the methods when the computer program product is run on a computer, and that may be stored, for example, on a machine-readable carrier.

[0062] Further embodiments comprise the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0063] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein when the computer program runs on a computer.

[0064] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a recording medium.

[0065] A further embodiment of the present invention is, therefore, a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0066] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0067] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0068] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0069] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

[0070] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0071] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent descriptions of corresponding methods, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent descriptions of a corresponding block or item or feature of a corresponding apparatus. [Explanation of symbols]

[0072] 100 Microscope System 110 Device for controlling microscope system 112 Interface 114 Processing Module 120 Camera Module 130 Robot Adjustment System 140 Display (Suffix a~c) 145,155 Arm 150 Microscope 160 Input Devices 210 Acquire camera image data 220 Processing camera image data 230 Provides control signals 310 Acquire camera image data 320 Processing camera image data 330 Provides control signals 400 System 410 Microscope 420 Computer Systems

Claims

1. A device (110; 420) for controlling a microscope system (100; 400), said device comprising: an interface (112) in communication with a camera module (120) adapted to provide camera image data of a head of a user of said microscope system; a processing module (114); Equipped with The processing module (114) acquiring camera image data from the camera module via the interface; processing the camera image data to determine information regarding the changing angular orientation of the user's head relative to a display (140) of the microscope system; providing a control signal for a robotic adjustment system (130) of the microscope system based on the information; It is structured as follows: the processing module is configured to determine a gaze direction of the user towards the display based on the camera image data and to provide the control signal based on the gaze direction of the user; the processing module is configured to generate the control signal such that the robotic adjustment system is triggered to pan a microscope field of view of the microscope system when only the position of the user's head changes without causing a corresponding change in the angle of the user's head. Device.

2. the processing module is configured to determine a gaze direction of the user based on an angular orientation of the user's head.

10. The apparatus of claim 1.

3. the processing module is configured to determine information regarding the angular orientation using image analysis on the camera image data.

3. The device according to claim 1 or 2.

4. The display of the microscope system is an auxiliary display disposed in a base unit of the microscope system.

4. An apparatus according to any one of claims 1 to 3.

5. the processing module is configured to determine the user's gaze at a point based on the camera image data, and the processing module is configured to determine a control signal for the robotic coordination system such that a central viewpoint of a microscope (150; 410) of the microscope system remains fixed at the point.

5. An apparatus according to any one of claims 1 to 4.

6. the processing module is configured to determine information regarding a position of the user's head and information regarding an angle of the user's head based on the camera image data, and to determine information regarding an angular orientation of the user's head based on the information regarding the position of the user's head and the information regarding the angle of the user's head.

6. An apparatus according to any one of claims 1 to 5.

7. The processing module is configured to generate the control signal such that the robotic adjustment system is triggered to adjust the field of view of a microscope of the microscope system when the position of the user's head changes with a corresponding change in the angle of the user's head.

7. The apparatus of claim 6.

8. A device (110; 420) for controlling a microscope system (100; 400), said device comprising: an interface (112) in communication with a camera module (120) adapted to provide camera image data of a head of a user of said microscope system; a processing module (114); Equipped with The processing module (114) acquiring camera image data from the camera module via the interface; processing the camera image data to determine information regarding the changing angular orientation of the user's head relative to a display (140) of the microscope system to determine the user's gaze at a point; providing a control signal for a robotic adjustment system (130) of said microscope system based on said information so that the viewpoint of a microscope (150; 410) of said microscope system remains fixed at said one point; It is structured as follows: the processing module is configured to determine a gaze direction of the user towards the display based on the camera image data and to provide the control signal based on the gaze direction of the user; the processing module is configured to generate the control signal such that the robotic adjustment system is triggered to pan a microscope field of view of the microscope system when only the position of the user's head changes without causing a corresponding change in the angle of the user's head. Device (110; 420).

9. the microscope system is a surgical microscope system, and the processing module is configured to determine information regarding an angular orientation of the user's head relative to a surgical site observed by a surgical microscope (150; 410) of the surgical microscope system; 9. The apparatus of claim 8.

10. 1. A method for controlling a microscope system, the method comprising: acquiring (210) camera image data of a head of a user of the microscope system from a camera module of the microscope system; processing (220) the camera image data to determine information regarding the changing angular orientation of the user's head relative to a display (140) of the microscope system; providing (230) a control signal for a robotic adjustment system of the microscope system based on the information; Including, The method includes determining a gaze direction of the user toward the display based on the camera image data, and providing the control signal based on the gaze direction of the user; generating the control signal such that the robotic adjustment system is triggered to pan a microscope field of view of the microscope system when only the user's head position changes without causing a corresponding change in the user's head angle; Including, method.

11. 1. A method for controlling a microscope system, the method comprising: acquiring (310) camera image data of a head of a user of the microscope system from a camera module of the microscope system; processing (320) the camera image data to determine information regarding the changing angular orientation of the user's head relative to a display (140) of the microscope system to determine the user's gaze at a point; providing (330) a control signal for a robotic adjustment system of the microscope system based on said information so that the viewpoint of a microscope (150) of said microscope system remains fixed at said single point; Including, The method includes determining a gaze direction of the user toward the display based on the camera image data, and providing the control signal based on the gaze direction of the user; generating the control signal such that the robotic adjustment system is triggered to pan a microscope field of view of the microscope system when only the user's head position changes without causing a corresponding change in the user's head angle; Including, method.

12. 12. A computer program comprising a program code for performing at least one of the methods according to claim 10 or 11, when the computer program is executed on a processor.

Images