System and corresponding method and computer program for a microscope system

Abstract

The embodiments relate to a system for a microscope system, a microscope system having such a system, a corresponding method, and a computer program. The system includes one or more processors and one or more storage devices. The system is configured to receive a trigger signal. The trigger signal indicates a request by a user of the microscope system to display a visual representation of a control function associated with an input device of the microscope system. The system is configured to generate a visual overlay having the visual representation based on the trigger signal. The system is configured to provide a display signal having the visual overlay to a display device of the microscope system.

Description

[Technical Field]

[0001] The embodiments relate to a system for a microscope system, a microscope system comprising such a system, and corresponding methods and computer programs. [Background technology]

[0002] Today's microscope systems, particularly surgical microscope systems, offer a wide variety of features to assist the user (i.e., surgeon) while operating the microscope. At the same time, the user may not want to take their eyes off the eyepiece. This can complicate the operation of the microscope system because the input devices used to control various functions may be hidden from the user. Additionally, due to the large number of different functions available to the user, the user may be uncertain as to which function is handled by which input modality of the input device. Summary of the Invention [Problem to be solved by the invention]

[0003] It may be desired to provide an improved concept for a microscope system, which allows the user of the microscope system to more easily understand its function. [Means for solving the problem]

[0004] This need is addressed by the subject matter of the independent claims.

[0005] Embodiments of the present disclosure provide a microscope system, a corresponding system for a microscope system, a method, and a computer program. The embodiments of the present disclosure are based on the discovery that a large number of functions are freely available to a microscope user during operation of a microscope, particularly a surgical microscope. For example, in a surgical microscope used for ophthalmic surgery, each input device may include different input modalities (e.g., buttons, foot-activated buttons, control sticks, etc.), where different functions are assigned to the input modalities based on the operating mode of the surgical microscope. Furthermore, each user (e.g., a surgeon) can freely assign their own preferred set of functions to the input devices, such that there is no fixed relationship between input modalities and microscope (control) functions. Embodiments of the present disclosure provide a system for a microscope system, a corresponding microscope system, a method, and a computer program configured to generate a visual overlay of control functions available to a user via a given input device. The visual overlay is provided on a display device of the microscope system, such as an eyepiece display or an auxiliary display, and the microscope user can inspect the visual overlay to confirm that control functions have been assigned to the various input modalities. To invoke the visual overlay, the user can trigger the system using a trigger signal, which may be generated using the very input device on which the control function is displayed, or may be generated using a different system, such as voice control or a touch-based interface.

[0006] Various embodiments of the present disclosure relate to a system for a microscope system. The system has one or more processors and one or more storage devices. The system is configured to receive a trigger signal. The trigger signal indicates a request from a user of the microscope system to display visual representations of control functions associated with input devices of the microscope system. The system is configured to generate a visual overlay having the visual representation based on the trigger signal. The system is configured to provide a display signal to a display device of the microscope system. The display signal has the visual overlay. The visual overlay having the visual representations of the control functions associated with the input devices of the microscope system can be used by a user of the microscope system, such as a surgeon, to stay informed about the control functions assigned to various input modalities.

[0007] In some embodiments, the system is configured to receive a trigger signal from one of one or more input devices of the microscope system and generate a visual overlay with a visual representation of a control function associated with the input device from which the trigger signal is received. In other words, the triggered visual overlay may relate to the same input device used to generate the trigger. For example, the system may be configured to receive the trigger signal from a button on the input device. This button may remain constant regardless of the mode of operation in which the microscope system is being used.

[0008] Besides the input device itself, another modality may be used to trigger the display of the overlay. For example, the system may be configured to process audio captured via a microphone on the microscope system and generate a trigger signal based on one or more keywords spoken by the user in the captured audio. In this implementation, the user / surgeon may not be required to move a hand or foot from another input modality they are using. Alternatively or additionally, the system may be configured to obtain the trigger signal via a touchscreen on the microscope system. For example, a visual aid may be triggered by an assistant via the touchscreen.

[0009] Microscope systems, particularly surgical microscope systems, feature a wide range of different input devices, and thus the visual representation may represent control functions associated with the microscope system's foot pedals, with a handle, with the microscope system's mouth switches, or with the microscope system's eye tracking system.

[0010] As noted above, microscope systems, particularly surgical microscope systems, may be operated in different operational modes, where control functions are assigned to input modalities of an input device that change between the operational modes. Thus, an input device may have (or be configured to act on) two or more sets of functions associated with two or more modes of the input device, one of which is active on the input device. The system may be configured to generate a visual representation based on the set of functions associated with the mode active on the input device. In other words, the visual overlay may be adaptive based on the currently used operational mode.

[0011] In some examples, the visual representation of a control function includes a visual representation of the input device with which the control function is associated, which can allow for a more understandable display of controls, especially those associated with input devices having complex shapes, such as a steering wheel with input modalities on different sides.

[0012] As mentioned above, the control functions available through microscope systems, particularly surgical microscope systems, are constantly expanding. However, only a limited number of different input modalities of an input device are simultaneously available. To support more functions, an "unlimited selection of functions" is available through an additional trigger signal. In other words, the system may be configured to acquire another trigger signal, which indicates a user's desire to switch the control function associated with the input device. The system may be configured to switch the control function associated with the input device based on the another trigger signal. The system may also be configured to adapt the visual representation after switching the control function associated with the input device. By switching the control function, a greater number of functions are available through the input device, while a visual overlay can be used to indicate the input modality of the input device associated with the newly selected (i.e., switched) function.

[0013] The controls available for selection and the effect of the selection may be highlighted via the overlay. In other words, the system may be configured to generate a visual overlay having a visual representation of the controls currently associated with the input device and a visual representation of the controls available for association with the input device after a switch.

[0014] As previously noted, an input device may have at least one of multiple input modalities, e.g., one or more buttons, one or more pads, one or more control sticks, one or more rotary controls, etc. Thus, the visual representation may represent multiple control functions associated with the multiple input modalities of the input device.

[0015] Various embodiments of the present disclosure further provide a microscope system having a display device, an input device, and a display device. For example, the input device may be one of a foot pedal, a handle, a mouse switch, or an eye tracking system of the microscope system. As above, the input device may have multiple input modalities, and the multiple input modalities may include at least one of one or more buttons, one or more switches, one or more rotary controls, and one or more control sticks.

[0016] Various embodiments of the present disclosure further provide a corresponding method for a microscope system. The method includes obtaining a trigger signal, the trigger signal indicating a request from a user of the microscope system to display a visual representation of a control function associated with an input device of the microscope system. The method includes generating a visual overlay having the visual representation based on the trigger signal. The method includes providing a display signal having the visual overlay to a display device of the microscope system.

[0017] Various embodiments of the present disclosure further provide corresponding computer programs having program codes for performing the methods when the computer programs are run on a processor.

[0018] Some embodiments of the apparatus and / or methods will now be described, by way of example only, and with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0019] [Figure 1a] FIG. 1 is a block diagram of one embodiment of a system for a microscope system. [Figure 1b] 1 is a schematic diagram of a surgical microscope system used in ophthalmology. [Figure 2] 1 is a flowchart of one practical example of a method for a microscope system. [Figure 3a] FIG. 1 is a schematic diagram of the functions assigned to the foot pedals of a surgical microscope system. [Figure 3b] FIG. 10 is another schematic diagram of the functions assigned to the foot pedals of the surgical microscope system. [Figure 4a] 1 is a schematic diagram of an input device and input modalities of the input device for a microscope system. [Figure 4b] FIG. 10 is another schematic diagram of an input device and input modalities of the input device for a microscope system. [Figure 4c] FIG. 10 is yet another schematic diagram of an input device and input modalities of the input device for a microscope system. [Figure 5a] FIG. 1 is a schematic diagram of a surgical guidance overlay displayed on a display device of a microscope system. [Figure 5b] FIG. 10 is another schematic diagram of a surgical guidance overlay displayed on a display device of a microscope system. [Figure 6] FIG. 1 illustrates one embodiment of an on-screen menu. [Figure 7] FIG. 1 illustrates one embodiment of a system having a microscope and a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0020] Various embodiments will now be described more fully hereinafter with reference to the accompanying figures, in which several embodiments are shown, in which line thicknesses, layers and / or regions may be exaggerated for clarity.

[0021] 1a shows a block diagram of one embodiment of a system 110 for a microscope system 100. The system 110 includes one or more processors 114 and one or more storage devices 116. The system optionally further includes an interface 112. The one or more processors 114 are connected to the optional interface 112 and to the one or more storage devices 116. In general, the functionality of the system 110 is provided by the one or more processors 114, e.g., in conjunction with the optional interface 112 and / or one or more storage devices 116.

[0022] The system is configured to receive a trigger signal (e.g., via interface 112). The trigger signal indicates a request by a user of the microscope system to display a visual representation of a control function associated with the input devices 120, 125 of the microscope system. The system is configured to generate a visual overlay having the visual representation based on the trigger signal. The system is configured to provide a display signal to a display device 130 of the microscope system (e.g., via interface 112). The display signal has the visual overlay.

[0023] Various embodiments of the present disclosure further provide a microscope system 100, e.g., a surgical microscope system 100, including the system 110. FIG. 1b shows a schematic diagram of a surgical microscope system used in ophthalmology, including the system 100. The microscope system 100 further includes one or more input devices 120, 125, e.g., a foot pedal 120 and / or a handle 125, a display device 130, and a microscope 160. The microscope system 110 may include one or more additional optional features, such as a base unit 105 (including the system 110), an arm 170 to which the microscope 160 is attached, or a microphone 140. For example, the microscope 160 may include an eyepiece 130. The microscope system shown in FIG. 1b is a surgical microscope system that may be used by a surgeon at a surgical site. Notably, the microscope system shown in FIG. 1b is a surgical microscope system used in ophthalmology (eye surgery), but the same concept may also be used in other types of surgical microscope systems, such as surgical microscope systems used in neurosurgery. For example, the microscope system may include or be used in combination with one or more additional systems, such as an optical coherence tomography (OCT) system (not shown).

[0024] Embodiments of the present disclosure relate to systems, methods, and computer programs suitable for a microscope system, such as the microscope system 100 described in connection with FIG. 1b. As mentioned above, a distinction is made between the microscope 160 and the microscope system 100, where the microscope system includes the microscope 160 and various components used with the microscope 160, such as an illumination system, an auxiliary display, and the like. In a microscope system, the actual microscope is often referred to as an "optical carrier" because it includes the optical components of the microscope system. In general, a microscope is an optical instrument suitable for inspecting objects too small to be inspected by the human eye alone. For example, a microscope can provide optical magnification of an object. In modern microscopes, optical magnification is often provided to a camera or imaging sensor. The microscope 160 may also include one or more optical magnification components used to magnify the field of view on a sample.

[0025] There are various types of microscopes. When a microscope system is used in the medical or biological fields, the object viewed through the microscope may be a sample of organic tissue, such as a sample placed in a Petri dish, or a sample present in a patient's body. For example, microscope system 100 may be a laboratory microscope system, such as a microscope that can be used to examine an organic tissue sample in a Petri dish. Alternatively, microscope 160 may be part of a surgical microscope system 100, such as a microscope used during a surgical procedure. Such a system is shown, for example, in FIG. 1b. While several embodiments are described in connection with a microscope system, they are applicable in a more general manner to any optical device. For example, microscope system 100 may be a system for performing material testing or integrity testing of materials, such as metals or composites.

[0026] The system is configured to obtain a trigger signal indicative of a user of the microscope system's desire to display a visual representation of a control function associated with the microscope system's input device 120, 125. In general, the trigger signal may originate from two types of sources: either the input device for which the visual representation is to be generated or another input device or modality.

[0027] In the first case, the trigger signal may also indicate an input device for which a visual representation should be generated. For example, the microscope system may have one or more input devices (e.g., multiple input devices). Depending on the input device that generates the trigger signal (i.e., the input device from which the trigger signal is obtained), one of the one or more input devices may be selected, and the visual representation may be generated for the selected input device. In other words, the system may be configured to obtain a trigger signal from one of the one or more input devices 120, 125 of the microscope system and generate a visual overlay with a visual representation of a control function associated with the input device from which the trigger signal was received (i.e., the input device that generated the trigger signal). For example, if the trigger signal is received from a foot pedal of the microscope system, the visual representation may be associated with the foot pedal. For example, the system may be configured to obtain the trigger signal from or via a button on the input device. In other words, the trigger signal may be generated in response to activation of a button on the input device.

[0028] In the second case, the trigger signal is received from, originates from, and is generated by an input modality different from the input modality in which the visual representation is generated. For example, the trigger signal may be generated by the microscope system itself by processing an input signal generated by a sensor of the microscope system. For example, the microscope system may include or be connected to a microphone 140. The system may be configured to process audio captured via the microphone 140 and generate a trigger signal based on one or more keywords spoken by a user in the captured audio. For example, the system may be configured to detect one or more keywords (or key phrases) in the captured audio, such as "show me the function of the foot pedal," "show me an overview of the main input device," or "how to focus / zoom the microscope." The microscope system may also have other types of sensors, such as a camera for performing user eye tracking or a depth sensor for detecting user gestures. The system may be configured to process sensor data from the camera or depth sensor and generate a trigger signal based on the sensor data. In some embodiments, the trigger signal may be generated via the microscope system's touchscreen 150. In other words, the system may be configured to obtain the trigger signal via the microscope system's touchscreen 150. For example, the touchscreen may be actuated by the user / surgeon themselves or by an assistant to generate the trigger signal.

[0029] In general, the proposed concept can be used for any type of input device. In a (surgical) microscope system, the two main input devices used are the foot pedal 120 and the handle 125 of the microscope system. However, the same concept can also be applied to other input devices, such as a mouse switch or an eye-tracking system of the microscope system. In other words, the input device can be one of the foot pedal 120, the handle 125, the mouse switch, or the eye-tracking system of the microscope system. Thus, the visual representation can represent the control functions associated with the foot pedal 120, the handle 125, the mouse switch, or the eye-tracking system of the microscope system. In either case, the input device or input devices can be an input device / one or more input devices with one or more input modalities that can be reconfigured with respect to the control functions associated with one or more input modalities. At the same time, the control functions of the input device may not be displayed on the input device itself. In particular, the input device does not have to be a touchscreen, and the control functions of the graphical user interface elements can always be displayed on the screen, as in the case of a touchscreen. For example, the input device can be separate from the display device. For example, the input device may be a physical input device separate from a touchscreen.

[0030] This application distinguishes between an "input device" and an "input modality." Generally, an "input device" may refer to the device itself, such as a foot pedal or a handle, while an "input modality" may refer to the means provided by the input device for triggering a control function. For example, an input device may include multiple input modalities, where the multiple input modalities may include one or more buttons, one or more switches, one or more rotary controls, and / or one or more control sticks. For example, an input device "foot pedal" 120 may have multiple input modalities, such as a depressible button, a pad that can be tilted in one or another direction, or a control stick that can be tilted in one of multiple directions. For example, FIG. 4a shows a foot pedal (i.e., input device) with six buttons 401, 404, 405, 408, 413, and 414, two pads that can be pressed left (402, 406) and right (403, 407), respectively, and a control stick that can be tilted left 409, up 410, right 411, and down 412. The buttons provide a total of 14 input modalities (6 via the buttons, 4 via the pads, and 4 via the control stick) depending on the directions the pads can be pressed and the directions the control stick can be tilted. If the input device is an eye-tracking system, different directions in which the user gazes may correspond to input modalities. If the input device is a mouse switch, the input modalities may be an "actuation modality" (e.g., a switch activated via the chin) and a four-way joystick activated via the mouth.

[0031] As will be apparent, the term "foot pedal" is used not to refer to a single button operable by the foot, but to an entire input device having multiple input modalities. Similarly, the handles shown in FIG. 4b each have two input modalities: clockwise rotation and counterclockwise rotation. The handle in FIG. 4c has ten input modalities, including six buttons 435-440 and a control stick that can be oriented in four directions (431-434). As will be apparent, multiple control functions can be associated with multiple input modalities of the input device. Thus, the visual representation can represent multiple control functions associated with multiple input modalities of the input device.

[0032] As is evident from the above examples, multiple input modalities can be supported by input devices that may have complex shapes, such as the control handle shown in FIG. 4c. To aid in comprehension of the visual overlay, the visual representation of the input device may be included in the visual representation, e.g., a three-dimensional visual representation. In other words, the visual representation of a control function may include a visual representation, e.g., a three-dimensional representation, of the input device with which the control function is associated. In some examples, the display device 130 may be a display device for displaying three-dimensional images. For example, the display device may be configured to provide three-dimensional images using different polarized light while the user / surgeon is wearing polarized glasses.

[0033] The trigger signal indicates a request by a user of the microscope system to display a visual representation of a control function associated with the input device 120, 125 of the microscope system. In other words, the trigger signal is triggered by the user to display a visual representation of the control function associated with the input device on the display. Thus, the visual representation is displayed in response to the trigger signal. In some embodiments, the visual representation may be shown as long as (and only if) the trigger signal is active. Alternatively, the visual representation may be displayed for a predetermined time in response to the trigger signal and hidden after the predetermined time. For example, the system may be configured to hide and / or fade out the visual representation after the predetermined time.

[0034] The system is configured to generate a visual overlay having a visual representation based on the trigger signal. Generally, any type of visual representation of a control function may be part of the visual overlay. For example, the visual representation may have a pictogram representation of the control function, or the visual representation may have a textual representation of the control function. In other words, for each control function accessible through one of the input modalities, the visual representation may have a textual or pictogram representation, such as those shown in FIGS. 3a and 3b. For example, the visual overlay may have a visual representation of the control function associated with the input modality that mimics the layout of the input device, e.g., located near the visual representation of the input device itself, as shown in FIGS. 3a-4c. For example, the visual representation may represent the control function associated with the foot pedal 120 of the microscope system, the handle 125, the mouse switch of the microscope system, or the eye tracking system of the microscope system, and / or specifically associated with the input modality provided by the respective input device.

[0035] In general, the control function may relate to any function of the microscope system that can be triggered or adapted, such as a zoom function, a focus function, movement of a robotic arm of the microscope system, an image processing function of the microscope system, a display function of the microscope system, an illumination function of the microscope system (e.g., fluorescent stimulation or visible light illumination), etc. In some cases, the control function may relate to another device connected to the microscope system, such as optical coherence tomography and intraocular lens guidance in the case of a surgical microscope system used in ophthalmic surgery. In other words, the control function may relate to a function provided by the microscope system and / or a function provided by a device connected to or integrated into the microscope system, such as an OCT system or lens guidance.

[0036] As described above, a microscope system, particularly a surgical microscope system, may have one or more operating modes. For example, a surgical microscope system used in ophthalmic surgery may have a vitreoretinal mode and an OCT mode. In different operating modes, different control functions can be associated with the input device / input modality of the input device. For example, FIGS. 3a and 3b show examples of function assignments in the vitreoretinal (VR) and OCT operating modes. In the vitreoretinal mode, the following functions are assigned to different input modalities: all lights on / off, magnification -, magnification +, OCT mode on / off, focus -, focus +, VR sync focus +, VR sync focus -, X -, X +, Y +, Y -, and VR mode on / off. In the OCT mode, the following functions are assigned to different modalities of the input device: OCT Live Mode / Freeze, OCT Optimization, OCT Auto Positioning, OCT Mode On / Off, OCT Scan, OCT Next Procedure, OCT Z-, OCT Z+, OCT Left, OCT Right, OCT Up, OCT Down, OCT Save b, and OCT Joystick State Change. More formally, the input device may have or consist of two or more sets of functions associated with two or more modes of the input device. At the same time, two or more modes of the input device may be associated with two or more operating modes of the microscope system. One of the two or more modes (i.e., one mode) may be active in the input device (at a given time). The system may be configured to generate a visual representation based on the set of functions associated with the active mode in the input device. In other words, the visual representation may be generated depending on which mode is active in the input device / microscope system.

[0037] In some cases, the number of different available functions exceeds the number of input devices or input modalities provided by one or more input devices. A visual overlay can be used to enable selection of such functions that may not be assigned to any input modality of any input device. For example, the visual overlay can be used to display information about available functions in addition to a visual representation of the control function associated with the input device. This can enable the microscope system to provide “unlimited functionality” available to a user of the microscope system via the visual overlay. To access the unlimited functionality, the user can request the system to display the available functions via a separate trigger signal. In other words, the system can be configured to obtain a separate trigger signal. For example, the separate trigger signal can be obtained similarly to the trigger signal defined above and can originate from one of the multiple input devices, originate from a touchscreen, or be generated by the system based on sensor input. The separate trigger signal can indicate a user of the microscope system's desire to switch the control function associated with the input device. Thus, the system can be configured to switch the control function associated with the input device based on the separate trigger signal. In other words, the system may be triggered to switch the control function associated with the input device in response to receiving another trigger signal. For example, the function associated with each input modality of the input device may be changed by switching the control function associated with the input device based on another trigger signal. Accordingly, the system may be configured to adapt the visual representation (to indicate the control function associated with the input device after switching) after switching the control function associated with the input device.

[0038] In many cases, the amount of available functions is so large that the user / surgeon must toggle through numerous functions to reach the requested control function. Suitably, the available control functions can be displayed in a visual overlay so that the user / surgeon is made aware of the available functions and, aided by the visual overlay, can select the requested control function. For example, in FIG. 6, seven different control functions 610-670 are shown as on-screen items for selection by the user. Accordingly, the system can be configured to generate a visual overlay that includes a visual representation of the control function currently associated with the input device (i.e., a visual representation of the control function associated with the input device) and a visual representation of the control function available for association with the input device after a switch. In other words, the available control functions, i.e., the control functions available for association with the input device after a switch, can be displayed as part of the visual overlay. For example, the visual representation of the control function available for association with the switched input device may be shown alongside or in place of the visual representation of the control function currently associated with the input device, e.g., in response to or until another trigger signal is received.

[0039] In some embodiments, smart guidance features can be provided. For example, the system can be configured to select an available control function for association with a subsequent input device based on the progress of a procedure being performed with the assistance of the surgical microscope system. Additionally, the system can be configured to generate a visual overlay to guide the surgeon in selecting a control function associated with a subsequent step in the procedure they are performing.

[0040] The system is configured to provide a display signal having a visual overlay to a display device 130 of the microscope system (e.g., via the interface 112). The display device may be configured to display the visual overlay based on the display signal, e.g., insert the visual overlay over a view on the sample based on the display signal. For example, the display signal may include a video stream having the visual overlay or control instructions, e.g., causing the respective display devices to display the visual overlay. For example, the display device may be one of an eyepiece display of the microscope and an auxiliary display of the surgical microscope system. In modern (surgical) microscope systems, the view on the sample is often provided via a display, e.g., an eyepiece display, an auxiliary display, or a headset display, using, for example, a video stream generated based on image sensor data of an optical imaging sensor of the respective microscope. In this case, the visual overlay may simply be overlaid on the video stream. For example, the system may be configured to generate the display signal by acquiring image sensor data of the optical imaging sensor of the microscope, generating a video stream based on the image sensor data, and overlaying the visual overlay on the video stream.

[0041] Alternatively, the visual overlay may be overlaid on an optical view of the sample. For example, the eyepiece of a microscope may be configured to provide an optical view on the sample, and the display device may be configured to insert the overlay into the optical view on the sample, for example, using a one-way mirror or a semi-transparent display disposed in the optical path of the microscope. For example, the microscope may be an optical microscope having at least one optical path. A one-way mirror may be disposed in the optical path, and the visual overlay may be projected onto the one-way mirror and thus overlaid on the view on the sample. In this case, the display device may be a projection device configured to project the visual overlay toward the mirror, for example, so that the visual overlay is reflected toward the eyepiece of the microscope. Alternatively, a display may be used to provide the overlay in the optical path of the microscope. For example, the display device may have at least one display disposed in the optical path. For example, the display may be one of a projection-based display and a screen-based display, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED)-based display. For example, the displays may be located in the eyepieces of an optical stereo microscope, e.g., one display in each eyepiece. For example, two displays can be used to turn the eyepieces of an optical microscope into an augmented reality eyepiece, i.e., an augmented reality eyepiece. Alternatively, other technologies can be used to implement the augmented reality eyepiece / eyepiece.

[0042] The interface 112 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, the interface 112 may comprise an interface circuit configured to receive and / or transmit information. In various embodiments, the one or more processors 114 may be implemented using one or more processing units, one or more processing devices, any processing means, such as a processor, computer, or programmable hardware component operable with appropriately adapted software. In other words, the described functions of the one or more processors 114 may be implemented in software, which in this case executes on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a digital signal processor (DSP), a microcontroller, or the like. In at least some embodiments, one or more storage devices 116 may comprise at least one element of the group such as a computer-readable storage medium, e.g., a magnetic or optical storage medium, e.g., a hard disk drive, flash memory, floppy disk, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or network storage.

[0043] Further details and aspects of the present system and microscope system are described in connection with the proposed concept or one or more of the embodiments described above or below (e.g., FIGS. 2-7). The present system or microscope system may have one or more additional, optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0044] 2 shows a flowchart of one practical example of a corresponding method for a microscope system. The method includes step 210 of receiving a trigger signal. The trigger signal indicates a request from a user of the microscope system to display a visual representation of a control function associated with an input device of the microscope system. The method includes step 220 of generating a visual overlay having the visual representation based on the trigger signal. The method includes step 230 of providing a display signal having the visual overlay to a display device of the microscope system.

[0045] The features described in relation to the system 110 and microscope system 100 of FIGS. 1a and / or 1b are equally applicable to the method of FIG.

[0046] Further details and aspects of the method are described in relation to the proposed concept or one or more of the embodiments described above or below (e.g., FIGS. 1a-1b, 3-7). The method may have one or more additional, optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0047] Various aspects of the present disclosure relate to surgical control guidance for microscope interaction.

[0048] Foot pedals (also known as foot switches) and handle controls play a vital role in surgery. While there are minor variations between models and manufacturers, the basic operation of microscope foot pedals and microscopes is similar. The primary microscope controls are focus, zoom, and centering. Additional functions include the ability to turn the microscope light on and off as well as adjust the brightness.

[0049] However, during various types of surgery, the surgeon's hands and feet can become busy as more than one foot pedal is used; for example, in basic cataract surgery in ophthalmology, one foot is placed on the microscope pedal, the other on the phacoemulsification pedal, one hand interacts with the phacoemulsification handpiece, and the other hand holds the phacoemulsification chopper instrument.

[0050] In ophthalmology, a typical procedure may require multiple foot pedals, allowing the surgeon to control a microscope foot pedal, a phacoemulsification foot pedal to control cataract irrigation, aspiration, and ultrasound power delivery, a vitrectomy foot pedal, and a laser photocoagulation foot pedal. In microsurgery, the microscope handles can be programmed with up to 10 functions each, providing a total of 20 functions for both the right and left handles.

[0051] This presents several challenges: First, inexperienced surgeons can have difficulty remembering what functions are programmed into the foot pedals, sometimes opting to stick hard-copy printouts or hand-drawn illustrations of the controls on the microscope for easy reference during surgery.

[0052] With the introduction of advanced surgical guidance technology to improve patient outcomes and surgeon efficiency, new software applications are integrated into microscopes (e.g., intraocular lens guidance, optical coherence tomography visualization, enhanced fluorescence imaging). These surgical guidance applications are activated and controlled by the same foot pedals and handles available on the microscope. As a result, experienced surgeons face a new set of challenges as they program additional functions and become reacquainted with these new controls. A pain point for these surgeons can be remembering the associated function of each foot pedal or handle button for each surgical "mode" (typically, vitreoretinal, OCT function, etc.), as the programmed functions may differ between modes.

[0053] 3a and 3b show a schematic diagram of the functions assigned to the foot pedals of a surgical microscope system. Figures 3a and 3b illustrate how additional complexity can be introduced with new functionality for advanced surgical guidance applications, allowing experienced surgeons to memorize specific functions programmed into the foot pedals and handles when switching between surgical modes (e.g., vitreoretinal function, OCT function). The foot pedal of Figures 3a and 3b has multiple input modalities including a lower left button 301a / b, a lower right button 304a / b, a lower pad having left side 302a / b and right side 303a / b, a middle left button 305a / b, a middle right button 306a / b, a middle pad having left side 307a / b and right side 308a / b, an upper control stick having a stick movable to left position 309a / b, upper position 310a / b, right position 311a / b and lower position 312a / b, and an upper left button 313a / b and an upper right button 314a / b. For example, FIG. 3 a shows an exemplary assignment of functions for the vitreoretinal mode, where the (control) functions All Lights On / Off are assigned to the lower left button 301 a, Magnification − are assigned to the left side of the lower pad 302 a, Magnification + are assigned to the right side of the lower pad 303 a, OCT Mode On / Off are assigned to the lower right button 304 a, Focus − are assigned to the middle left button 305 a, Focus + are assigned to the middle right button 306 a, and VR Sync Focus + are assigned to the left side of the middle pad 307 a. VR sync focus - is assigned to the right side 308a of the center pad, X- is assigned to the upper control stick moving to the left position 309a, X+ is assigned to the upper control stick moving to the right position 311a, Y+ is assigned to the upper control stick moving to the up position 310a, Y- is assigned to the upper control stick moving to the down position 312a, nothing is assigned to the upper left button 313a, and VR mode on / off is assigned to the upper right button 314a.

[0054] FIG. 3b shows an exemplary assignment of functions for OCT mode, where the (control) functions OCT Live Mode / Freeze are assigned to the lower left button 301b, OCT Optimize are assigned to the left side of the lower pad 302b, OCT Auto Position are assigned to the right side of the lower pad 303b, OCT Mode On / Off are assigned to the lower right button 304b, OCT Scan are assigned to the middle left button 305b, OCT Next Procedure are assigned to the middle right button 306b, OCT Z- are assigned to the left side of the middle pad 307b, and OCT Z+ is assigned to the right side of the center pad 308b, OCT left is assigned to the upper control stick moving to the left position 309b, OCT right is assigned to the upper control stick moving to the right position 311b, OCT up is assigned to the upper control stick moving to the up position 310b, OCT down is assigned to the upper control stick moving to the down position 312b, OCT save is assigned to the upper left button 313b, and OCT joystick state change is assigned to the upper right button 314b.

[0055] Limited interactions are available on the foot pedal and handle, and these limited interactions can limit the number of functions that can be programmed into the control. For example, ophthalmology may add 12 to 14 functions on the foot pedal, plus four functions for various modalities on the handle, and microsurgery may assign up to 10 functions on the handle. Before surgery, surgeons can prioritize programming by selecting the functions they need, while the remaining unselected functions remain inaccessible.

[0056] This can lead to adoption barriers for new technologies that require programming additional functionality onto a foot pedal or handle. With limited available interactions, surgeons often program dedicated buttons to switch between "modes" (typically vitreoretinal, OCT functions, etc.), which introduces the complexity discussed above.

[0057] 4a-4c show schematic diagrams of input devices and input modalities for a microscope system. Figures 4a-4b show the limited interactions available on the foot pedal and handle in ophthalmology: 12-14 on the foot pedal (lower left button 401, lower right button 404, lower pad with left side 402 and right side 403, center left button 405, center right button 406, center pad with left side 407 and right side 408, upper control stick with a stick that can be moved to left position 409, upper position 410, right position 411, and lower position 412, upper left button 413, upper right button 414), plus four on the handle (clockwise rotation on the left side 421, counterclockwise rotation on the left side 422, clockwise rotation on the right side 423, and counterclockwise rotation on the right side 424). FIG. 4c shows the limited interactions available on the handle in microsurgery, i.e., up to 10 interactions on the handle (upper control stick that can be moved to the up 431, left 432, right 433 and down 434 positions, upper left button 435, upper right button 436, lower left button 437, lower right button 438, right button 439 and back button 440).

[0058] Various aspects of the present disclosure provide a surgical guidance overlay of foot pedal or handle functions on a screen (of a display device) projected onto either a microscope stand monitor, a head-up display, or an external operating room monitor. This overlay can display functions preprogrammed into the handle or foot switch as a reminder to the surgeon during surgery. This overlay can be activated by triggering a dedicated button on the handle or foot switch, through a user interface on the microscope touchscreen, or through activation of either a handle or foot switch function. In other words, a trigger signal may be generated in response to triggering a dedicated button on the handle or foot switch, through a user interface on the microscope touchscreen, or through activation of either a handle or foot switch function. This overlay can be automatically faded in and out without interrupting the surgical workflow.

[0059] Below is an overview of the proposed concept. In the proposed concept, a surgical guidance overlay for the surgeon's controls, i.e., a visual overlay, i.e., a visual representation of the control functions (via footswitches or handles), is projected onto the microscope stand monitor, heads-up 3D display, external operating room monitor, or digital viewer. This overlay provides a quick guide for the surgeon to refer to during surgery. The overlay can be displayed on a heads-up 3D display or any digital viewer to allow the surgeon to have an uninterrupted view of the surgical workflow.

[0060] In some cases, the input device and microscope system may support multiple operating modes, or modes for short. In this case, the overlay may display the associated functions of the foot pedal buttons and handle interactions according to the surgical mode being used during surgery. The displayed functions may differ depending on the mode.

[0061] In some embodiments, smart guidance features can be provided, for example, integrated to prompt and guide the surgeon to select the next function in a surgical step to improve clinical workflow efficiency.

[0062] The overlay can also be programmed to display functionality and guidance from third-party accessories, such as pedal controls for phacoemulsification or vitreoretinal instruments, thereby providing the surgeon with a surgical cockpit experience, with all useful data and controls displayed for the surgeon to see at a glance.

[0063] For display formats used for overlays, the overlay may be displayed as a transparent fade in layer upon activation, which may fade out automatically without requiring additional steps.

[0064] The surgical guidance overlay may be activated by triggering a dedicated pre-programmed interaction on the handle or foot switch, or via a touch interface on the microscope monitor, or may be activated automatically when a standard function is triggered on the handle or foot switch. The surgical guidance overlay may also be activated by other means, such as voice recognition, head movement, eye tracking, or a touch-enabled interface on the foot pedal or handle.

[0065] Additionally, the overlay can provide the surgeon with an unlimited selection of functions by displaying a list of options on the screen that can be rotated for selection. In other words, the surgical guidance overlay can provide an infinite range of functions that can be displayed on the screen for selection and activation by the surgeon. This allows the surgeon to go beyond what is pre-programmed on the foot pedal (up to 14 functions) or on the handle (up to 4 ophthalmic, up to 10 microsurgery).

[0066] These additional features, or “unlimited features,” can be displayed as rotating or pop-up menus on the screen, providing a range of selectable functions for activation via a foot pedal, handle, or touch interface connected to the microscope. The displayed additional features can be a preset range recommended based on the surgical mode being operated, or the surgeon can preselect their preferred features to be displayed on the screen for activation or deactivation. This is particularly beneficial considering that more and more features will be integrated into microscope systems to guide surgical workflows, such as optical coherence tomography and intraocular lens guidance. As future surgical trends move toward increasingly digital workflows, new software features, such as tool tracking, feature recognition, and voice recognition, can be integrated as part of the microscope functionality, and these new software features can be accessed via the same input device. These numerous new features can benefit from the proposed concept of being displayed, selected, and activated during surgery. A surgical guidance overlay can provide an enabling platform, allowing the surgeon to operate beyond limited interactions on the handle and foot switch.

[0067] 5a and 5b show schematic diagrams of surgical guidance overlays displayed on a microscope system's display device. Figure 5a shows a foot pedal 120 or handle function 125 displayed as an overlay on the microscope display 130 of the microscope system 500. This overlay can be activated by triggering a dedicated button on the handle or foot switch, through a user interface on the microscope touchscreen, or through activation of either the handle or foot switch function. Figure 5b shows a foot pedal or handle function displayed as an overlay on the 3D head-up surgical display 130 as part of a surgical cockpit experience in another implementation of the microscope system 500. Again, this overlay can be activated by triggering a dedicated button on the handle or foot switch, through a user interface on the microscope touchscreen, or through activation of either the handle or foot switch function.

[0068] As shown in Figure 6, rather than being limited by the number of physical interactions available on the foot pedals and handles, the on-screen options menu provides the surgeon with a virtually unlimited number of functions that can be viewed and selected during surgery. Figure 6 shows an example of an on-screen menu with menu items 610 (activate tool tracking), 620 (activate digital filters), 630 (activate voice commands), 640 (OCT playback), 650 OCT scan, 660 (photo capture), and 670 (video record).

[0069] Further details and aspects of the surgical guidance overlay are described in relation to the proposed concept or one or more of the embodiments described above or below (e.g., FIGS. 1a-2). The surgical guidance overlay may have one or more additional, optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0070] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0071] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0072] Some embodiments relate to a microscope including a system such as that described in connection with one or more of FIGS. 1-6. Alternatively, the microscope may be part of or connected to a system such as that described in connection with one or more of FIGS. 1-6. FIG. 7 shows a schematic diagram of a system 700 configured to perform the methods described herein. The system 700 includes a microscope 710 and a computer system 720. The microscope 710 is configured to capture images and is connected to the computer system 720. The computer system 720 is configured to perform at least a portion of the methods described herein. The computer system 720 may be configured to execute a machine learning algorithm. The computer system 720 and the microscope 710 may be separate entities or may be integrated within a common housing. The computer system 720 may be part of a central processing system of the microscope 710 and / or part of a subsidiary component of the microscope 710, such as a sensor, actor, camera, or lighting unit of the microscope 710.

[0073] Computer system 720 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed across various locations, such as local clients and / or one or more remote server farms and / or data centers). Computer system 720 may include any circuit or combination of circuits. In one embodiment, computer system 720 may include one or more processors, which may be of any type. As used herein, processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor in a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuits that may be included in computer system 720 may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 720 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.Computer system 720 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touch screen, voice recognition device, or any other device that allows a user of the system to input information into computer system 720 and receive information from computer system 702.

[0074] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry. In some embodiments, any one or more of the critical steps may be performed by such an apparatus.

[0075] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation may be performed by a non-transitory storage medium, such as a digital storage medium, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Therefore, the digital storage medium may be computer-readable.

[0076] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

[0077] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code that is operative to perform any of the methods when the computer program product is run on a computer, and that may be stored, for example, on a machine-readable carrier.

[0078] Further embodiments comprise the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0079] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein when the computer program runs on a computer.

[0080] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a recording medium.

[0081] A further embodiment of the present invention is, therefore, a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0082] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0083] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0084] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0085] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus. [Explanation of symbols]

[0086] 100 Microscope System 105 Base Unit 110 System 112 Interface 114 one or more processors 116 one or more storage devices 120 Foot Pedal 125 handle 130 Display 140 microphones 150 touchscreen 160 Microscope 170 Arm 210 Acquiring a trigger signal 220 Generating Visual Overlays 230 Display signal supply 301a / b Lower left button 302a / b Left side of lower pad 303a / b Right side of lower pad 304a / b Lower right button 305a / b center left button 306a / b center right button 307a / b Left side of center pad 308a / b Right side of center pad 309a / b Control stick left position 310a / b Control stick up position 311a / b Control stick right position 312a / b Control stick down position 313a / b Top left button 314a / b Top right button 401 Lower left button 402 Left side of lower pad 403 Right side of lower pad 404 Lower right button 405 Center Left Button 406 Center Right Button 407 Left side of center pad 408 Right side of center pad 409 Control stick left position 410 Control stick up position 411 Control stick right position 412 Control stick down position 413 Top left button 414 Top right button 421 Clockwise rotation of left handle 422 Left handle counterclockwise rotation 423 Clockwise rotation of right handle 424 Counterclockwise rotation of right handle 431 Control stick up position 432 Control stick left position 433 Right position of upper control stick 434 Upper control stick down position 435 Upper left button 436 Upper right button 437 Lower left button 438 Lower Right Button 439 Right button 440 Back Button 500 Microscope System 610~670 Menu items 700 System 710 Microscope 720 Computer Systems

Claims

1. A system (110, 720) for a microscope system (100, 500, 700), the system (110, 720) having one or more processors (114) and one or more storage devices (116), the system comprising: obtaining a trigger signal indicative of a request by a user of the microscope system to display a visual representation of a control function associated with an input device (120, 125) of the microscope system; generating a visual overlay having the visual representation based on the trigger signal; providing a display signal having the visual overlay to a display device (130) of the microscope system; It is structured as follows: the visual representation represents a control function associated with only one of a foot pedal (120) of the microscope system, a handle (125) of the microscope system, a mouse switch of the microscope system, and an eye tracking system of the microscope system; the visual representation of the control function comprises a visual representation of the input device with which the control function is associated. System (110, 720).

2. the system is configured to receive the trigger signal from one of one or more input devices (120, 125) of the microscope system and generate the visual overlay having a visual representation of the control function associated with the input device from which the trigger signal is received. The system of claim 1 .

3. the system is configured to obtain the trigger signal from a button on the input device.

3. The system according to claim 1 or 2.

4. the system is configured to process audio captured via a microphone (140) of the microscope system and generate the trigger signal based on one or more keywords spoken by a user in the captured audio; or the system is configured to receive the trigger signal via a touchscreen (150) of the microscope system. The system of claim 1 .

5. the input device having two or more sets of functions associated with two or more modes of the input device, with one of the two or more modes active on the input device; the system is configured to generate the visual representation based on a feature set associated with a mode active on the input device. A system according to any one of claims 1 to 4.

6. the system is configured to receive another trigger signal, the another trigger signal indicating a user's request to switch a control function associated with the input device, the system is configured to switch the control function associated with the input device based on the another trigger signal, and the system is configured to adapt the visual representation after switching the control function associated with the input device. A system according to any one of claims 1 to 5.

7. the system is configured to generate the visual overlay such that the visual overlay has a visual representation of a control function currently associated with the input device and a visual representation of a control function available for association with the input device after a switch. The system of claim 6.

8. the visual representation represents a plurality of control functions associated with a plurality of input modalities of the input device; A system according to any one of claims 1 to 7.

9. A microscope system (100) having a system (110) described in any one of claims 1 to 8, an input device (120, 125), and a display device (130).

10. the input device is one of a foot pedal (120), a handle (125), a mouse switch, or an eye tracking system of the microscope system; The microscope system according to claim 9.

11. the input device has a plurality of input modalities, the plurality of input modalities including at least one of one or more buttons, one or more switches, one or more rotary controls, and one or more control sticks; 11. The microscope system according to claim 9 or 10.

12. 1. A method for a microscope system, the method comprising: obtaining (210) a trigger signal indicative of a request by a user of the microscope system to display a visual representation of a control function associated with an input device of the microscope system; generating (220) a visual overlay having the visual representation based on the trigger signal; Providing (230) a display signal having the visual overlay to a display device of the microscope system; and the visual representation represents a control function associated with only one of a foot pedal (120) of the microscope system, a handle (125) of the microscope system, a mouse switch of the microscope system, and an eye tracking system of the microscope system; the visual representation of the control function comprises a visual representation of the input device with which the control function is associated. How to do it.

13. 13. A computer program having a program code for performing the method according to claim 12, when the computer program is run on a processor.

Images