Surgical microscope system, corresponding system, method and computer program for a surgical microscope system - Patents.com

Abstract

Examples relate to a surgical microscope system, corresponding systems, methods and computer programs for the surgical microscope system. The system includes one or more processors and one or more storage devices. The system is configured to track progress of a surgical procedure. The system is configured to select two or more functions of a plurality of functions of the surgical microscope system based on the progress of the surgical procedure. The system is configured to assign the two or more functions to two or more input modalities of the surgical microscope system. The system is configured to generate a visual overlay with a visual representation of the two or more functions. The two or more functions are illustrated in relation to the visual representation of the two or more input modalities. The system is configured to provide a display signal including the visual overlay to a display device of the surgical microscope system.

Description

[Technical field]

[0001] Examples relate to a surgical microscope system, a corresponding system, a method and a computer program for a surgical microscope system. [Background technology]

[0002] Surgical microscope systems are generally used in operating rooms. In many cases, assistants join the surgeon in the operating room and assist the lead surgeon to perform the surgical procedure. This assistance ranges from assisting with the surgery, handing over medical instruments, to adjusting the settings of medical equipment. A typical flow may include the lead surgeon telling the chief assistant the direct action required, for example to increase the light intensity or hand over a lancet, scissors, etc., and then the assistant performs this action. The assistant needs to have sufficient experience and knowledge in the surgical procedure and the patient's medical condition so that the communication time between the surgeon and the assistant can be minimized.

[0003] For example, in cases where an operating room does not have the space to accommodate extra medical staff, other than the surgeon, the time spent on the patient will be reduced if the surgeon has to adjust the equipment himself. To help the surgeon in such situations, the surgeon is allowed to have up to 10 system setting presets. During the surgical procedure, either the surgeon or the assistant can trigger the preset settings to issue commands via a graphical user interface (GUI) or via a foot switch as and when necessary.

[0004] Although such preset settings are a valuable aid to the surgeon, they require the surgeon or assistant to trigger them via the GUI or footswitch, which may distract the surgeon or assistant from the primary activity at that moment. Furthermore, generally, only a limited number of preset settings (e.g., 10) may be supported for each user. This may limit the utility that can be provided to the surgeon. In order for the surgeon / assistant to trigger the appropriate preset setting, the surgeon / assistant must become familiar with the identifiers of each setting, which may require practice and experience with the device. If an incorrect setting is triggered, additional time may be required to correct to the appropriate setting. Furthermore, in cases where the surgeon desires to have a changed setting, even if only slightly, an additional command may be required.

[0005] Improved concepts for surgical microscope systems that provide additional assistance to the surgeon during a surgical procedure would be desirable. Summary of the Invention [Means for solving the problem]

[0006] The above mentioned problem is solved by the subject matter of the independent claims.

[0007] Various embodiments of the present disclosure are based on the finding that many tasks performed by a surgeon during a surgical procedure lead to changes in settings or triggering of functions in the surgical microscope system being used, and that many surgical procedures follow a strict sequence of steps. Generally, this leads to a sequence of functions and / or settings of the surgical microscope system that are commonly selected / triggered as the surgical procedure progresses, which entails the same sequence of functions and / or settings used (by a particular surgeon) in many surgical procedures of the same type. The proposed concept is based on tracking the progress of the surgical procedure, selecting functions of the surgical microscope system that are particularly relevant to the current progress of the surgical procedure, and facilitating the surgeon's access to those functions.

[0008] One aspect of the present disclosure relates to a system for a surgical microscope system. The system includes one or more processors and one or more storage devices. The system is configured to track progress of a surgical procedure. The system is configured to select two or more functions of a plurality of functions of the surgical microscope system based on the progress of the surgical procedure. The system is configured to assign the two or more functions to two or more input modalities of the surgical microscope system. The system is configured to generate a visual overlay with visual representations of the two or more functions. The two or more functions are shown in relation to the visual representations of the two or more input modalities. The system is configured to provide a display signal including the visual overlay to a display device of the surgical microscope system. By tracking the progress of the surgical procedure, it can be determined which functions are likely to be required at each step of the progress. By assigning the two or more functions to the two or more input modalities and providing the corresponding visual representations, the surgeon recognizes the identified functions that the surgeon can easily access, thereby reducing the surgeon's cognitive load and reducing interruptions caused by the surgeon operating the surgical microscope system during the surgical procedure.

[0009] Generally, the two or more input modalities may be two or more input modalities of a tactile input device that is different from the display device of the surgical microscope system. Display-based input devices are often inaccessible to the surgeon due to the need to maintain a sterile operating environment. For example, the two or more input modalities may be two or more input modalities of a foot pedal of the surgical microscope system, or two or more input modalities of one or more handles of the surgical microscope system. The tactile input device may be located on the optics carrier or on the foot pedal of the surgical microscope system, providing easy access to the functions without the need to look at the input modality. In particular, the two or more input modalities may be implemented by a four-way switch on the foot pedal of the surgical microscope system. The four-way switch may thus facilitate access to the two or more identified functions, leaving the remaining buttons / switches of the foot pedal for statically assigned functions.

[0010] In some cases, instead of (or in addition to) a tactile input device, voice recognition may be used to trigger a desired function. Thus, the two or more input modalities may be two or more keywords of a voice recognition-based control mechanism of a surgical microscope system. A voice recognition-based control mechanism assisted by each keyword used shown on the screen can provide easy access to functions without the need to divert attention from the surgical site.

[0011] In the proposed concept, the selected two or more features are based on the progress of the surgical procedure. Thus, the system may be configured to update the selection of the two or more features based on the progress of the surgical procedure. This can ensure that the selected two or more features are relevant to the current progress of the surgical procedure.

[0012] In some examples, the system is configured to determine a ranking of the features with respect to their relevance in a current step of the progress of the surgical procedure and select two or more features based on the ranking, which may be useful in scenarios that reconcile different types of criteria (e.g., the progress of the surgical procedure and the personal preferences of the surgeon).

[0013] For example, two or more functions may be selected based on a deterministic allocation between the progress of the surgical procedure and functions of the plurality of functions. In other words, two or more functions can be deterministically allocated (i.e., allocated in a predefined manner) for each step of the surgical procedure. This may facilitate implementation of the feature.

[0014] Alternatively, the two or more features may be selected using a machine learning model trained to rank the features based on the progress of the surgical procedure, thereby allowing the two or more features to be selected based on the surgeon's personal preferences. Thus, the machine learning model trained to rank the features may be trained based on the personal preferences of the surgeon using the surgical microscope system, thereby allowing the two or more features to be selected based on the surgeon's personal preferences. This may increase the acceptance and usefulness of the selection of the two or more features, thereby helping the surgeon based on their practice.

[0015] There are various approaches to tracking the progress of a surgical procedure. For example, the system may be configured to track the progress of a surgical procedure based on a sequence of commands issued at the surgical microscope system. This is particularly applicable to surgical procedures that follow a strict surgical plan that mandates the use of a predefined sequence of commands at the surgical microscope system. For example, the system may be configured to navigate a state machine that represents the progress of the surgical procedure based on the sequence of commands issued at the surgical microscope system. Thus, the state machine can be used to model and follow the progress of the surgical procedure.

[0016] Alternatively or additionally, the system may be configured to track the progress of the surgical procedure using a machine learning model that is trained to track the progress of the surgical procedure based on image data of the optical imaging sensor of the surgical microscope system. In other words, object recognition or similar techniques may be used to determine the progress of the surgical procedure, for example, by recognizing the tool being used and / or by recognizing the action being performed.

[0017] For example, a surgical procedure may include a sequence of steps, each step including one or more tasks depicted in the image data. A machine learning model may be trained to detect the tasks depicted in the image data. The machine learning model may be trained to output information regarding the tasks depicted in the image data. The system may be configured to track the progress of the surgical procedure based on the information regarding the tasks output by the machine learning model. For example, each task may be differentiated by the tools being used and / or the actions being performed.

[0018] To facilitate interpretation of the output of the machine learning model, each step of the sequence may be assigned an identifier, and the machine learning model may be trained to output said identifiers. Thus, the machine learning model may be trained to output identifiers of steps depicted in the image data. The system may be configured to track the progress of the surgical procedure based on the identifiers provided by the machine learning model.

[0019] For example, the surgical procedure may be an eye surgical procedure. Eye surgical procedures, such as cataract surgery, often require strict adherence to a surgical plan.

[0020] Aspects of the present disclosure relate to a surgical microscope system that includes the system introduced above.

[0021] One aspect of the present disclosure relates to a corresponding method for a surgical microscope system. The method includes tracking progress of a surgical procedure. The method includes selecting two or more functions of a plurality of functions of the surgical microscope system based on the progress of the surgical procedure. The method includes assigning the two or more functions to two or more input modalities of the surgical microscope system. The method includes generating a visual overlay with a visual representation of the two or more functions. The two or more functions are indicated in relation to the visual representation of the two or more input modalities. The method includes providing a display signal including the visual overlay to a display device of the surgical microscope system.

[0022] One aspect of the present disclosure relates to a corresponding computer program having a program code for performing the above-mentioned method when the computer program is run on a processor.

[0023] Some examples of apparatus and / or methods are now described, by way of example only, and with reference to the accompanying figures. [Brief description of the drawings]

[0024] [Figure 1a] FIG. 1 is a block diagram of an example system for a surgical microscope system. [Figure 1b] FIG. 1 is a schematic diagram of an example of a surgical microscope system. [Diagram 2] 1 is a flowchart of an example method for a surgical microscope system. [Figure 3a] FIG. 1 is a schematic diagram of an example of a foot pedal for a surgical microscope system. [Figure 3b] 1A-1C are schematic diagrams of examples of various input modalities of foot pedals for a surgical microscope system. [Figure 4] 1 is a schematic diagram of the progression of an example eye surgical procedure. [Figure 5a] FIG. 1 is a schematic diagram of an example of the proposed concept. [Figure 5b] FIG. 1 is a schematic diagram of an example of the proposed concept. [Figure 5c] FIG. 1 is a schematic diagram of an example of the proposed concept. [Figure 6] 1 is a schematic diagram of an example of a system including a microscope and a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Various examples will now be described in more detail with reference to the accompanying drawings in which some examples are shown, in which line thicknesses, layer thicknesses and / or area sizes may be exaggerated for clarity.

[0026] 1a shows a block diagram of an example of a system 110 for a surgical microscope system 100 (shown in FIG. 1b). The system 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the system further includes an interface 112. The one or more processors 114 are coupled to the optional interface 112 and the one or more storage devices 116. In general, functionality of the system 110 is provided by the one or more processors 114, e.g., in conjunction with the optional interface 112 and / or the one or more storage devices 116.

[0027] The system is configured to track progress of the surgical procedure. The system is configured to select two or more functions of a plurality of functions of the surgical microscope system based on the progress of the surgical procedure. The system is configured to assign the two or more functions to two or more input modalities of the surgical microscope system. The system is configured to generate a visual overlay with a visual representation of the two or more functions. The two or more functions are shown in relation to the visual representation of the two or more input modalities. The system is configured to provide a display signal to a display device 130a; 130b; 130b (shown in FIG. 1b) of the surgical microscope system. The display signal includes the visual overlay.

[0028] Various embodiments of the present disclosure further provide a surgical microscope system 100 including the system 110. FIG. 1b shows a schematic diagram of a surgical microscope system 100 for use in ophthalmology, including the system 110. Thus, the surgical procedure may be an eye surgical procedure. The surgical microscope system 100 further includes one or more input devices 120; 125, such as foot pedals 120 and / or handles 125, a display device 130, and a microscope 140. The surgical microscope system 100 may include one or more additional optional features, such as a base unit 105 (which includes the system 110), a microphone 150 (for a voice recognition based control mechanism), and an arm 160 to which the microscope 140 is attached. The surgical microscope system further includes one or more display devices 130a-130c. For example, the microscope 140 may include an eyepiece 130a. Furthermore, the surgical microscope system may include one or more auxiliary displays 130b; 130c, which may be located on the base unit 105 and / or on the joint between the arm 160 and the microscope 140. The microscope system shown in FIG. 1b is a surgical microscope system that may be used at a surgical site by a surgeon. In particular, the surgical microscope system shown in FIG. 1b is a surgical microscope system for use in ophthalmology (eye surgery), but the same concept may also be used in other types of surgical microscope systems, such as surgical microscope systems for use in neurosurgery. For example, the surgical microscope system may include or be used in combination with one or more additional systems, such as an optical coherence tomography (OCT) system (not shown).

[0029] The embodiments of the present disclosure relate to a system, method and computer program suitable for a microscope system such as the surgical microscope system 100 introduced in relation to FIG. 1b. As introduced above, a distinction is made between the microscope 140 and the surgical microscope system 100, with the surgical microscope system including the microscope 140 and various components used in relation to the microscope 140, such as a lighting system, an auxiliary display, etc. In a microscope system, the actual microscope is often also referred to as the "optics carrier" since it includes the optical components of the surgical microscope system. Generally speaking, a microscope is an optical instrument suitable for inspecting objects that are too small for a human to inspect visually (alone). For example, the microscope may provide optical magnification of an object. Thus, the microscope may include one or more optical magnification components used to magnify the view of a sample. In modern microscopes, the optical magnification is often provided for a camera or imaging sensor of the microscope.

[0030] There are various types of microscopes. In the example described in relation to Fig. 1a and / or 1b, the microscope 140 is part of a surgical microscope system 100, e.g., a microscope used during a surgical procedure. Although the examples are described in relation to a surgical microscope system, they may be applied in a more general manner to any optical device. For example, the microscope system may be a system for performing material testing or integrity testing of materials, e.g., metals or composite materials.

[0031] The starting point of the proposed concept is the tracking of a surgical procedure by a surgical microscope system, in particular by system 110. There are several ways to achieve this. In a simple example, the progress of the surgical procedure may be tracked manually by the surgeon, e.g., via either a touch screen or voice control, by progressing from step / stage of the surgical plan to check the progress in the surgical plan. However, in many examples, the progress of the surgical procedure can be tracked without manual intervention. In other words, the progress of the surgical procedure can be tracked without the need for manual / human input.

[0032] This can be accomplished by analyzing signals available to the system 110 / surgical microscope system 100. In general, two types of signals may be analyzed: signals related to commands being issued in the surgical microscope system (or devices coupled to the surgical microscope system) and sensor signals related to one or more sensors of the surgical microscope system. In general, the two types of signals may be analyzed in conjunction, since some aspects of progress can be tracked by identifying commands and some aspects of progress can be tracked by identifying objects or movements in the visual sensor data, as will become apparent with respect to FIG.

[0033] For example, the system may be configured to track the progress of a surgical procedure based on a sequence of commands issued at the surgical microscope system. For example, as outlined in detail in relation to FIG. 4, each step (or several steps) of a sequence of steps of a surgical procedure may be associated with one or more commands issued at the surgical microscope system or at an auxiliary device coupled to the surgical microscope system. For example, in a first step of a surgical procedure, the surgeon may issue various commands to adjust the surgical microscope system to the patient, e.g., with respect to focus, magnification, lighting, video recording, etc. In subsequent steps, different commands may be issued. For example, as shown in FIG. 4, during the "viscoat application" step of a cataract surgical procedure, two pump commands may be issued at a pump coupled to the surgical microscope system. In the "phacoemulsification" stage, a command may be issued at an ultrasound tool coupled to the surgical microscope system. Thus, the system may be configured to obtain one or more signals from the surgical microscope system and / or one or more devices coupled to the surgical microscope system, detect one or more commands being issued based on the one or more signals, and track the progress of a surgical procedure based on the sequence of the one or more commands. For example, the sequence of commands may include two or more commands, each of which is one of a command for controlling an aspect of the surgical microscope system (magnification, focus, illumination, imaging mode, etc.) and a command for controlling an aspect of a device coupled to the surgical microscope system (OCT device, pump, ultrasound tool, etc.).

[0034] Generally, a sequence of commands is used since some commands may be issued at various steps of the sequence of steps. Depending on the position of the command within the sequence of commands, the appropriate step of the sequence of steps may be identified. For example, the system may be configured to navigate a state machine representing the progress of a surgical procedure based on the sequence of commands issued in the surgical microscope system. For example, the state machine may include multiple states and multiple transitions between the states. A transition between the states may occur when a command is issued indicating that the surgical procedure has moved from one step to another subsequent step.

[0035] Additionally or alternatively, machine learning can be used to track the progress of the surgical procedure. In other words, the system may be configured to track the progress of the surgical procedure using a machine learning model that is trained to track the progress of the surgical procedure based on image data of the optical imaging sensor 140 of the surgical microscope system. Thus, the embodiments may be based on the use of machine learning models or algorithms. Instead of relying on models and inferences, machine learning may refer to algorithms and statistical models that a computer system may use to perform a particular task without using explicit instructions. For example, machine learning may use data transformations that are inferred from analysis of past data and / or training data instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or using a machine learning algorithm. For the machine learning model to analyze image content, the machine learning model may be trained with training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize image content, such that image content not included in the training data can be recognized using the machine learning model. The same principles may be used for other types of sensor data as well: by training a machine learning model with training sensor data and a desired output, the machine learning model "learns" the transformation between the sensor data and the output, which can be used to provide an output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata and / or image data) may be pre-processed to obtain feature vectors that are used as inputs to the machine learning model.

[0036] In the proposed concept, machine learning may be used to identify tasks indicative of steps of a sequence of steps of a surgical procedure. In other words, the machine learning model may be trained to recognize tasks indicative of steps in image data of an optical imaging sensor of a microscope. For example, each step of a sequence of steps of a surgical procedure may include one or more tasks depicted (represented) in the image data. Thus, the machine learning model may be trained to detect tasks depicted in the image data and output information about the tasks depicted in the image data. Thus, the system may be configured to track the progress of the surgical procedure based on the information about the tasks depicted in the image data output by the machine learning model.

[0037] Such training may be performed using different techniques: The machine learning model may be trained using training input data. The example given for the general introduction to machine learning uses a training method called "supervised learning". In supervised learning, the machine learning model is trained using multiple training samples, where each sample may include multiple input data values ​​and multiple desired output values, i.e., each training sample is associated with a desired output value. By specifying both the training samples and the desired output values, the machine learning model "learns" during training which output value to provide based on input samples similar to the provided sample.

[0038] In the proposed concept, supervised learning can be used to train a machine learning model that is trained to track the progress of a surgical procedure. For example, still images and / or videos of a surgical procedure recording can be used as input data values ​​for training, and desired output values ​​can also be applied, such as indicators of the task being performed, positions of objects indicative of the task being performed, positions of movements (or actions) indicative of the task being performed, etc. For example, in the case of still images, an identifier and position of each object indicative of the task being performed may be provided. In the case of videos, an identifier and position of each object indicative of the task being performed may be provided along with a timestamp (identifying a time point or time interval), and an identifier and position of each movement (or action) indicative of the task being performed along with a timestamp (identifying a time point or time interval) may be provided as the desired output value. For example, the machine learning model may be trained to perform image segmentation, feature extraction, and object detection or classification on the segmented images. For example, multiple machine learning models may be used, possibly trained together, such as a dense trajectory CNN / ST-CNN (Spatial-Temporal Convolutional Neural Network) for feature extraction and an RNN (Recurrent Neural Network) for classification. The machine learning models may be trained to output an output corresponding to a desired output when fed still images and / or videos of a recording of a surgical procedure. For example, the machine learning models may be trained to output an output vector containing an object or motion (or action) identifier indicative of the task being performed, or a binary value for each object or motion (or action) the machine learning models are trained to detect.

[0039] Besides supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on supervised learning algorithms (e.g. classification algorithms, regression algorithms or similarity learning algorithms). Classification algorithms may be used when the output is restricted to a limited set of values ​​(categorical variables), i.e. the input is classified into one of a limited set of values. Regression algorithms may be used when the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples with a similarity function that measures how similar or related two objects are. Besides supervised or semi-supervised learning, unsupervised learning may be used to train machine learning models. In unsupervised learning, input data (only) may be provided, and unsupervised learning algorithms may be used to find structure in the input data (e.g. by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data containing multiple input values ​​into multiple subsets (clusters) such that input values ​​in the same cluster are similar according to one or more (predefined) similarity criteria, but dissimilar to input values ​​contained in another cluster.

[0040] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as "software agents") are trained to take actions in the environment. Based on the actions taken, rewards are calculated. Reinforcement learning is based on training one or more software agents to select actions that result in an increasing cumulative reward (as manifested by an increase in reward) resulting in the software agent becoming better at a given task. Reinforcement learning may be used instead of or in addition to supervised learning to train machine learning models that are trained to track the progress of a surgical procedure by using still images / videos of the recording of the surgical procedure as input to the machine learning model and defining a reward function that represents the deviation between the desired output value and the output generated by the machine learning model.

[0041] In the above examples, the output of the machine learning model is described in terms of the task being performed. That is, the output of the machine learning model can represent the task being performed (e.g., via objects and / or movements indicative of the task). In some examples, the training of the machine learning model may proceed further with the output of the machine learning model representing the steps shown in the image data (the steps being based on the objects / movements detected in the image data). In other words, the machine learning model may be trained to output identifiers of the steps shown in the image data, and the system is configured to track the progress of the surgical procedure based on the identifiers provided by the machine learning model. For example, one or more additional layers may be added to the machine learning model and trained to convert the output of the task being performed into a corresponding step in the sequence of steps.

[0042] The progress of the surgical procedure is then used to select two or more functions of the surgical microscope system. In particular, the two or more functions may be selected based on their suitability in the current step of the surgical procedure. For example, a prediction may be made as to which functions the surgeon / user of the surgical microscope system is likely to trigger, and those functions may be selected. For example, those two or more functions may be functions that the surgeon / user is more likely to trigger in the current step of the surgical procedure than other functions. Thus, those functions are made more easily accessible to the surgeon / user of the surgical microscope system.

[0043] These two or more functions are selected from the plurality of functions of the surgical microscope system. Generally, the plurality of functions of the surgical microscope system may relate to functions of the surgical microscope system and / or settings of the surgical microscope system that are triggerable by the surgeon / user. In other words, any function that is triggerable by the surgeon / user or any setting that is changeable by the user may be part of the plurality of functions. In this regard, a "function" may be considered as anything that can be controlled / triggered via the surgical microscope system. For example, the plurality of functions may include one or more functions related to controlling an aspect of the surgical microscope system, such as a function related to controlling magnification, a function related to controlling focus, a function related to controlling illumination, a function related to controlling working distance, or a function related to controlling an imaging mode (e.g., visual image or OCT image).

[0044] In some examples, the plurality of functions may include functions of one or more devices coupled to the surgical microscope system, such as a pump, an OCT device, an ultrasound device, etc. These devices may be controlled via and / or used in conjunction with the surgical microscope system, and therefore may be considered functions of the surgical microscope system. Thus, the plurality of functions may include functions related to control of auxiliary devices, such as an OCT device, a pump, or an ultrasound device.

[0045] Generally, the functions may be limited to those considered relevant to the (overall) surgical procedure, which may include basic functions of a surgical microscope system and may exclude functions of auxiliary devices or components that are not typically used during a particular surgical procedure.

[0046] In some examples, a predefined (i.e. deterministic) assignment between the progress of the surgical procedure and the selected function may be used. In other words, the two or more functions are selected based on a deterministic assignment between the progress of the surgical procedure and a function of the plurality of functions. For each step of the sequence of steps of the surgical procedure, two or more functions to be assigned to two or more input modalities may be defined.

[0047] Alternatively, various more subtle approaches may be used, for example to determine a contextual ranking of the features based on their relevance to the current step of the surgical procedure progress. Thus, the system may be configured to determine a ranking of the features with respect to their relevance in the current step of the surgical procedure progress and to select two or more features based on this ranking. For example, two or more very highly ranked features may be selected from this ranking. This ranking may represent the relevance of each feature in the current step of the surgical procedure progress. Some basic features, for example, those related to focusing and illumination, may be considered to be always relevant, while some features, for example those related to auxiliary devices coupled to the surgical microscope system, may be considered to be relevant only in some steps of the sequence of steps. This may result in the proposed ranking.

[0048] For example, machine learning can be used to determine the ranking. In other words, the two or more features may be selected using a machine learning model that is trained to rank the features based on the progress of the surgical procedure. When machine learning is used, a non-deterministic selection may be made, which may be based in part on the personal preferences of the surgeon or on the practice of the hospital that implements the surgical microscope system. For example, the machine learning model that is trained to rank the features may be trained based on the personal preferences of the surgeon using the surgical microscope system, and thus the two or more features are selected based on the personal preferences of the surgeon. Additionally or alternatively, the machine learning model that is trained to rank the features may be trained based on the practice used in the hospital that implements the surgical microscope system, and thus the two or more features are selected based on the practice used in the hospital. In the former case, the selection can be tailored specifically to the surgeon, which may require training a machine learning model separately for each surgeon (or training different machine learning models for each surgeon). In the latter case, a general practice within the hospital may be applied and followed by the surgeons in the hospital. Again, supervised learning may be used to train a machine learning model using information regarding the progress of the surgical procedure as input data values ​​and values ​​representing the desired ranking of the features or the suitability of those features at the current step of the progress of the surgical procedure as desired output values. The desired output values ​​may be collected during use of the surgical microscope system and adaptation of the training of the machine learning model may occur as new desired output values ​​are collected.

[0049] As the surgical procedure progresses, the selection of the two or more functions may be kept up to date. Thus, the system may be configured to update the selection of the two or more functions based on the progress of the surgical procedure. As a result, the system may also be configured to update the assignments and visual overlays based on the updated selections.

[0050] The two or more selected functions are then assigned to two or more input modalities of the surgical microscope system. In this context, a distinction is made between "input device" and "input modality". In general, an "input device" may relate to the device itself, such as a foot pedal or a handle, and an "input modality" may relate to the means provided by the input device to trigger a function. For example, an input device may include multiple input modalities, including at least one of one or more buttons, one or more switches, one or more rotary controls, and one or more control sticks. For example, the input device "foot pedal" 120 may include multiple input modalities, such as a depressible button, a pad that can be tilted in one direction or another, or a control stick that can be tilted in one of multiple directions. For example, FIG. 3b shows a foot pedal (i.e., input device) with six buttons / switches 301, 302, 307, 308, 309, 310, two two-way switches 303 / 304 and 305 / 306, and four-way switches 311-314. For example, if the input device is an eye-tracking system, different directions of the user's gaze can correspond to input modalities. If the input device is a mouse switch, the input modalities may be an "activation modality" (e.g., a switch activated via the chin) and a four-way joystick activated via the mouth. If the input device is a voice recognition-based control mechanism 150, the input modalities may be different keywords used to trigger functions.

[0051] In various examples, the two or more input modalities may be two or more input modalities of a tactile input device 120; 125, such as a foot pedal 120, a handle 125, or a mouse switch (not shown). For example, the two or more input modalities may be two or more input modalities of a foot pedal 120, one or more handles 125, or a mouse switch of a surgical microscope system. As shown in Figures 3b, 5a, and 5b, the two or more input modalities may be implemented, for example, by a four-way switch of a foot pedal of a surgical microscope system. In this context, the term "tactile input device" indicates that a switch, button, pedal, stick, etc. is tactilely activated. For example, the term "tactile input device" may exclude similar input devices, such as a touch screen. For example, the tactile input device may be different from the display device of the surgical microscope system, which is used to display the display signal.

[0052] Alternatively, a non-tactile input device may be used. For example, voice recognition may be used to trigger two or more functions. Thus, the two or more input modalities may be two or more keywords of a voice recognition based control mechanism 150 of the surgical microscope system. For example, the voice recognition based control mechanism 150 may be implemented by the system 110. In other words, the system may be configured to detect two or more keywords in an audio signal recorded via a microphone of the surgical microscope system.

[0053] A visual overlay is generated based on the selected functions and the input modalities to which they are assigned. Generally, any kind of visual representation of the two or more functions may be part of the visual overlay. For example, the visual representation may include a pictogram representation of the two or more functions, or the visual representation may include a textual representation of the two or more functions. In other words, for each of the two or more functions accessible via the input modality of this input modality, the visual representation may include a textual or pictogram representation, for example, as shown in FIG. 5b. Additionally, the visual overlay includes a representation of the input modality to which the two or more functions are assigned. For example, a pictogram or rendering representing the input modality and / or the input device including the input modality may be included in the visual overlay (as further shown in FIG. 5b). The representations of the two or more functions of may be included such that a textual or pictogram representation of each function is located adjacent to each input modality that is being used to trigger the function. For example, as shown in Figure 5b, if the two or more functions are four functions assigned to the four directions of a four-way switch, the input modalities may be represented by a pictogram 534 representing the four-way switch and the assigned functions may be represented by textual representations 532 (denoted by "XXXXXX" in Figure 5b) positioned adjacent to the different directions of the four-way switch. For example, if the two or more input modalities include a keyword or key phrase for a speech recognition based control mechanism, the speech recognition based control mechanism may be represented by a pictogram (such as pictogram 560 shown in Figure 5a) with the keyword or key phrase shown alongside the pictogram.

[0054] In some examples, instead of or in addition to the display signal with visual overlay, two or more functions and two or more associated input modalities (e.g., keywords) may be provided (e.g., announced) via a voice-based user interface. The surgeon / user can trigger the functions by triggering each input modality with a tactile input device or by speaking each keyword / key phrase.

[0055] In some examples, in addition to the two or more features, the proposed system can be used to suggest one or more surgical steps based on the progress of the surgical procedure. Similar to the selection of the two or more features described above, one or more surgical steps that are suitable for the current step of the progress of the surgical procedure may be selected using deterministic assignment or using a machine learning model. For example, the one or more surgical steps may include one or more suggestions for one or more surgical tasks to be performed by the surgeon and / or a position for performing the one or more surgical tasks (as shown in FIG. 5c).

[0056] The system is configured to provide a display signal to a display device 130 of the surgical microscope system (e.g., via the interface 112), the display signal including the visual overlay. The display device may be configured to show the visual overlay based on the display signal, e.g., to insert the visual overlay on a view of the sample based on the display signal. For example, the display signal may include a video stream including the visual overlay or control instructions, e.g., such that the visual overlay is shown by each display device. For example, the display device may be one of the eyepiece display 130a of the microscope and the auxiliary display 130b; 130c of the surgical microscope system. In modern surgical microscope systems, the view of the sample is often provided via a display, such as an eyepiece display, an auxiliary display, or a headset display, using a video stream generated based on image sensor data of an optical imaging sensor of each microscope, for example. In this case, the visual overlay may simply be overlaid on the video stream. For example, the system may be configured to generate the display signal by acquiring image sensor data of an optical imaging sensor of the microscope, generating a video stream based on the image sensor data, and overlaying the visual overlay on the video stream.

[0057] Alternatively, a visual overlay may be overlaid on the optical view of the sample. For example, an ocular eyepiece of a microscope may be configured to provide an optical view of the sample, and a display device may be configured to insert the overlay into the optical view of the sample, for example, using a one-way mirror or a semi-transparent display disposed in the optical path of the microscope. For example, the microscope may be an optical microscope having at least one optical path. A one-way mirror may be disposed in the optical path, and the visual overlay may be projected onto the one-way mirror and thus overlaid on the view of the sample. In this case, the display device may be, for example, a projection device configured to project the visual overlay towards the mirror such that the visual overlay is reflected towards the eyepiece of the microscope. Alternatively, a display may be used to provide the overlay in the optical path of the microscope. For example, the display device may include at least one display disposed in the optical path. For example, the display may be one of a projection-based display and a screen-based display, for example a liquid crystal display (LCD) or an organic light-emitting diode (OLED)-based display. For example, displays may be located inside the eyepieces of an optical stereo microscope, e.g., one display in each eyepiece. For example, two displays may be used to direct the eyepieces of an optical microscope into an augmented reality eyepiece, i.e., an augmented reality eyepiece. Alternatively, other technologies may be used to implement the augmented reality eyepiece / augmented reality eyepiece.

[0058] For example, the optical imaging sensor of the microscope may include or be an Active Pixel Sensor (APS)-based imaging sensor or a Charge-Coupled-Device (CCD)-based imaging sensor. For example, in an APS-based imaging sensor, light is recorded at each pixel using a pixel photodetector and an active amplifier. APS-based imaging sensors are often based on Complementary Metal-Oxide-Semiconductor (CMOS) or Scientific CMOS (S-CMOS) technology. In a CCD-based imaging sensor, incident photons are converted to electronic charges at the semiconductor-oxide interface, which are then moved between capacitive bins in the imaging sensor module by the sensor imaging module's control circuitry to perform imaging. Image data can be obtained by receiving image data from the optical imaging sensor (e.g., via the interface 112 and / or the system 110), by reading image data from the imaging sensor's memory (e.g., via the interface 112), or by reading image data from the storage device 116 of the system 110, for example, after the image data is written to the storage device 116 by the optical imaging sensor or another system or processor.

[0059] The interface 112 may correspond to one or more inputs and / or outputs for receiving and / or sending information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, the interface 112 may include an interface circuit configured to receive and / or send information. In an embodiment, the one or more processors 114 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor operable with appropriately adapted software, a computer, or a programmable hardware component. In other words, the described functions of the one or more processors 114 may be implemented in software, which is then executed in one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. In at least some embodiments, the one or more storage devices 116 may include at least one element of a group of computer-readable recording media, such as magnetic or optical recording media, e.g., a hard disk drive, flash memory, floppy disk, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or network storage.

[0060] Furthermore, some techniques may be applied to parts of the machine learning algorithm. For example, feature representation learning may be used. In other words, the machine learning model may be trained at least in part with feature representation learning, and / or the machine learning algorithm may include a feature representation learning component. A feature representation learning algorithm, which may be referred to as a representation learning algorithm, may not only preserve information in its input, but may also transform the information to make it useful, often as a pre-processing step before performing classification or prediction. Feature representation learning may be based on, for example, principal component analysis or cluster analysis.

[0061] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide identification of input values ​​that raise suspicion by differing significantly from the majority of the input or training data. In other words, a machine learning model may be trained at least in part with anomaly detection and / or a machine learning algorithm may include an anomaly detection component.

[0062] In some examples, the machine learning algorithm may use a decision tree as a predictive model. In other words, the machine learning model may be based on a decision tree. In a decision tree, an observation about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to this item may be represented by a leaf of the decision tree. The decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0063] Association rules are another technique that may be used in machine learning algorithms. In other words, a machine learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. A machine learning algorithm may identify and / or utilize one or more association rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply the knowledge.

[0064] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a set of data structures and / or rules that represent learned knowledge (e.g., based on training performed by a machine learning algorithm). In embodiments, the use of machine learning algorithm may refer to the use of an underlying machine learning model (or underlying machine learning models). The use of machine learning model may refer to the machine learning model and / or the set of data structures / rules that are the machine learning model being trained by a machine learning algorithm.

[0065] For example, the machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or the brain. An ANN contains a number of interconnected nodes and a number of junctions, so-called edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may convey information from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g. the sum of its inputs). The inputs of a node may be used in a function based on the "weights" of the edges or nodes that provide the inputs. The weights of the nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input. For example, the machine learning model being trained to track the progress of a surgical procedure may be a Convolutional Neural Network (CNN), e.g., a dense trajectory CNN, a spatial-temporal CNN, or a Recurrent Neural Network (RNN). CNNs are particularly suited to the analysis of image data, and spatial-temporal CNNs offer improved support for the analysis of sequences of image data. RNNs are particularly suited to the analysis of sequences, e.g., sequences of tasks.

[0066] Alternatively, the machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that may be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing input with multiple training input values ​​that belong to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine learning model may be based on a genetic algorithm, which is a heuristic method that mimics search algorithms and the process of natural selection.

[0067] Further details and aspects of the system and surgical microscope system are referred to in relation to the proposed concept or one or more examples described above or below (e.g., FIGS. 2-6). The system and / or surgical microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0068] 2 shows a flow chart of an example of a corresponding method for a surgical microscope system. The method includes tracking 210 progress of a surgical procedure. The method includes selecting 220 two or more functions of a plurality of functions of the surgical microscope system based on the progress of the surgical procedure. The method further includes assigning 230 the two or more functions to two or more input modalities of the surgical microscope system. The method includes generating 240 a visual overlay with a visual representation of the two or more functions, the two or more functions being shown in relation to the visual representation of the two or more input modalities. The method includes providing 250 a display signal including the visual overlay to a display device of the surgical microscope system.

[0069] The features described in relation to the system 110 and the surgical microscope system 100 of Figures 1a and / or 1b may be applied to the method of Figure 2 as well. For example, the method may be a computer-implemented method. The method may be performed by a surgical microscope system, for example by a computer system of a surgical microscope system such as the system 110 of Figures 1a and / or 1b, or by the computer system 620 introduced in relation to Figure 6.

[0070] Further details and aspects of the method are mentioned in relation to the proposed concept or one or more examples described above or below (e.g., Figures 1a-1b, 3a-6). The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0071] Various examples of the present disclosure relate to an AI (artificial intelligence-based and machine learning-based) surgical assistant, which includes command recommendations and voice control, with the use of artificial intelligence and voice control being optional.

[0072] The proposed surgical assistant is based on tracking the progress of each surgical procedure. The surgical assistant can provide a real-time surgical activity segmentation (to segment the surgical procedure into multiple steps). From here, the surgical assistant can suggest to the surgeon the commands most likely required by the surgical process at the current time step, for example ranked by probability. This recommendation can be prompted via a GUI or via machine speech (i.e. a text-to-speech system).

[0073] Additionally, the suggestions provided may be generated according to the surgical activities that occurred immediately before that moment. This can be obtained by a neural network that has been trained on various medical surgical videos. This feature can reduce the requirements for the knowledge level and experience of the chief assistant.

[0074] For example, the medical video clips and system data output logs can be used to train one or more neural networks (e.g., using dense trajectory CNN / ST-CNN (Spatial-Temporal Convolutional Neural Network) for feature extraction and RNN (Recurrent Neural Network) for classification, or using temporal convolutional network or reinforcement learning) to segment the surgical activities (i.e., determine the steps of the surgical procedure). Another family of neural networks may be trained to map the segmented surgical activities (i.e., steps of the surgical procedure) to multiple system commands (i.e., two or more functions) that can be issued at specific multiple time steps. The generated recommended commands may be ranked and suggested to the user either via a GUI (e.g., via a visual overlay) or audio.

[0075] In some examples, the triggering of commands may be performed by speech. Either the surgeon or the assistant can speak the command, be it one of the suggestions provided by the AI ​​surgical assistant or another command, without selecting a setting of the configuration on the GUI. This feature may therefore avoid possible adverse consequences of the surgeon getting distracted and then having to refocus on the task at hand. More importantly, it may save more time for the actual surgery. The speech recognition can be performed using either cloud-based voice recognition or one or more pre-trained models fine-tuned based on the medical context, which can map the user command (uttered) to a corresponding system command (triggering a function).

[0076] In a typical workflow of the proposed concept, the system can prompt the surgeon for possible reconfigurations of the system based on the current clinical activity, and the surgeon can then issue commands via GUI or voice to trigger specific adjustments.

[0077] In summary, the proposed concept can reduce or eliminate the need to consult the GUI for changing system settings, which can reduce the demands on assistant knowledge and experience and most importantly, save time during the actual surgery.

[0078] Further details and aspects of the surgical assistant are mentioned in connection with the proposed concept or one or more examples described above or below (e.g., Figures 1a-2, 4-6). The surgical assistant may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0079] FIG. 3a shows a schematic diagram of an example foot pedal for a surgical microscope system. The foot pedal includes multiple input modalities, such as a switch (i.e., a button), a two-way switch, and a four-way switch. The input modalities are triggered by the surgeon's foot.

[0080] 3b shows a schematic diagram of an example of various input modalities of a foot pedal for a surgical microscope system. In FIG. 3a, the foot pedal includes a lower left switch 301, a lower right switch 302, a lower two-way switch (left side 303 and right side 304), an upper two-way switch (left side 305 and right side 306), a center left switch 307 and a center right switch 308, an upper left switch 309 and an upper right switch 310, and a four-way switch with left direction 311, up direction 312, right direction 313 and down direction 314.

[0081] In the example of an ophthalmic surgical microscope system, a foot switch (foot pedal (the terms "foot switch" and "foot pedal" are used interchangeably in this disclosure)) can have multiple modes, namely, a forward mode, a forward OCT mode, a VR mode, and a VR OCT mode.

[0082] For example, in forward mode, the lower left switch 301 can be set to turn all lights on / off, the lower right switch 302 can be set to turn OCT mode on / off, the lower two-way switches 303;304 can be set to minus and plus magnification (left and right, respectively), the upper two-way switches 305;306 can be set to minus and plus focus (left and right, respectively), the center left switch 307 and the upper left switch 309 can be set to minus and plus main light, respectively, the center right switch 308 and the upper right switch 310 can be set to plus and minus red reflex, respectively, and the four-way switches 311-314 can be set to move X minus and X plus (left and right) and move Y minus and Y plus (down and up). In other modes, this setting may be changed. For example, in two OCT modes, the switches may be set to control the OCT instead of the microscope.

[0083] These many functions assigned to the foot pedals may be considered not user-friendly, especially for new users. In fact, surgeons may have difficulty remembering the assignments. Therefore, the user / surgeon may print out the foot switch assignments and attach the printed paper to the surgical microscope system.

[0084] Figure 4 shows a schematic diagram of the progress of an example of an eye surgical procedure. The example shown in Figure 4 shows a cataract surgery, which includes steps 410 to 490. The different steps can be distinguished based on commands issued by the user / surgeon at the surgical microscope system or auxiliary devices, objects detected in the image data of the surgical procedure and / or movements detected in the image data.

[0085] First, a port incision 410 is made. The port incision may be detected based on one or more of: turning on the light (command), focusing (series of commands), adjusting magnification (series of commands), starting video recording (command), and a puncture blade (detected object). The port incision 410 is followed by a second incision 420. The second incision 420 may be detected based on detection of a corneal incision blade (detected object). The second incision 420 is followed by a viscoat application 430. The viscoat application 430 may be detected based on one or more of an injection of viscoat (pump issued command) and an injection of an anesthetic (pump issued command). The viscoat application 430 is followed by a continuous annular capsulorhexis 440, which may be detected based on one or more of a capsulorhexis (detected object) and an injection of an anesthetic (pump issued command). The continuous annular capsulorhexis 440 is followed by flap formation 450, which may be detected based on one or more capsulorhexis (detected objects) and flap peeling (detected movement). The flap formation 450 is followed by phacoemulsification 460, which may be detected using one or more of a phacoemulsifier (detected objects) and a breaker core (ultrasound command and detected movement). The phacoemulsification 460 is followed by aspiration 470, which may be detected using one or more of a phacoemulsifier (detected objects) and aspiration (aspiration command and detected movement). The aspiration 470 is followed by irrigation / aspiration 480, which may be detected based on one or more of an irrigation handpiece (detected objects), an aspiration handpiece (detected objects), an irrigation process (detected movement), and red reflex illumination (illumination command). The irrigation / aspiration 480 is followed by intraocular lens insertion, which may be detected based on one or more of a lens injector (detected objects) and an adjustment (detected movement). The actions, objects and movements may be used to train the neural network 400 .

[0086] 5a, 5b and 5c show a schematic diagram of an example of the proposed concept. In 5a, the overall flow of this example is shown. It starts with the current stage / step 510 of the surgical procedure (port incision stage / step in 5a). A mechanism such as a trained neural network 520 may then be used to predict possible steps (i.e. possible functions of the surgical microscope system and / or possible surgical steps). The proposed functions may be shown overlaid on top of a visual image of the surgical site 530. In 5b, the view 530 is shown in more detail, where the proposed function 532 (whose textual description is indicated by "XXXXXX") is shown next to a representation 534 of the input modality (in this case, a four-way foot switch). The proposed surgical step 542 is shown in view 540, which is shown in more detail in 5c. 5c shows the proposed surgical step 542 (whose textual description is indicated by "XXX") along with the proposed incision location 544. The user selects the desired function via an input device / input modality 550;560, where input device 550 is a foot pedal / foot switch and input device 560 is a voice recognition based input modality. In FIG. 5a, a four-way switch 552-558 is used for suggested functions 1-4, as shown in view 530. In response to a selection by the user / surgeon, the desired function is triggered 570.

[0087] Further details and aspects of the surgical microscope system, the system or method for the surgical microscope system, and the corresponding surgical assistant are mentioned in relation to the proposed concept or one or more examples described above or below (e.g., Figs. 1a-2, 6). The surgical microscope system, the system or method for the surgical microscope system, and the corresponding surgical assistant may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0088] Some embodiments relate to a microscope including a system as described in relation to one or more of Figures 1 to 5c. Alternatively, the microscope may be part of a system as described in relation to one or more of Figures 1 to 5c or may be connected to a system as described in relation to one or more of Figures 1 to 5c. Figure 6 shows a schematic diagram of a system 600 configured to perform the methods described herein. The system 600 includes a microscope 610 and a computer system 620. The microscope 610 is configured to capture images and is connected to the computer system 620. The computer system 620 is configured to perform at least a part of the methods described herein. The computer system 620 may be configured to execute a machine learning algorithm. The computer system 620 and the microscope 610 may be separate entities, but may be integrated in one common housing. The computer system 620 may be part of a central processing system of the microscope 610 and / or the computer system 620 may be part of a subordinate component of the microscope 610, such as a sensor, actor, camera or lighting unit of the microscope 610.

[0089] The computer system 620 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 620 may include any circuit or combination of circuits. In one embodiment, the computer system 620 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, for example, of a microscope or a microscope component (e.g., a camera). Other types of circuits that may be included in computer system 620 may be custom circuits, application specific integrated circuits (ASICs), such as one or more circuits (such as communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 620 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives handling removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), and the like.Computer system 620 may also include a display device, one or more speakers and a keyboard and / or controller which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that enables a user of the system to input information to and receive information from computer system 620.

[0090] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the crucial steps may be performed by such an apparatus.

[0091] Depending on certain implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be performed by a non-transitory recording medium, such as a digital recording medium, for example a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM and EPROM, an EEPROM or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Thus, the digital recording medium can be computer readable.

[0092] Some embodiments of the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform any of the methods described herein.

[0093] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which is operable to perform any of the methods when the computer program product is run on a computer, the program code may for example be stored on a machine readable carrier.

[0094] Another embodiment comprises the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0095] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0096] Therefore, another embodiment of the present invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, including a processor and a recording medium.

[0097] A further embodiment of the invention is therefore also a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0098] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0099] Another embodiment comprises a computer having the computer program installed thereon for performing any of the methods described herein.

[0100] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server to transfer the computer program to the receiver.

[0101] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

[0102] As used in this specification, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0103] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus. [Explanation of symbols]

[0104] 100 Surgical Microscope System 110 System 112 One or more interfaces 114 One or more processors 116 One or more storage devices 120 Foot Pedal 125 Handle 130a, 130b, 130c Display device 140 Microscope 150 microphones, voice recognition based control mechanism 210 Tracking the progress of surgical procedures 220 Selection of two or more features 230 Assigning two or more features to two or more input modalities 240 Creating visual overlays 250 Display signal provision 301~314 Foot pedal switches 400 trained neural networks 410~490 Steps in cataract surgery 510 Current Progress in Surgical Procedures 520 trained neural networks 530 Visual overlay showing proposed features 532 proposed features 534 Representation of Input Modality 540 Visual overlay showing proposed surgical steps 542 Proposed Surgical Steps 544 Proposed incision location 550 Foot Pedal 552-558 Foot pedal input modality assigned to proposed function 560 Voice Control 570 Triggering the desired function 600 System 610 Microscope 620 Computer Systems

Claims

1. A system (110; 620) for a surgical microscope system (100; 600), said system including one or more processors (114) and one or more storage devices (116), the system is configured to track progress of a surgical procedure; the system is configured to select two or more functions of a plurality of functions of the surgical microscope system based on the progress of the surgical procedure; the system is configured to assign the two or more functions to two or more input modalities of the surgical microscope system; the system is configured to generate a visual overlay with visual representations of the two or more functions, the two or more functions being shown in relation to the visual representations of the two or more input modalities; the system is configured to provide a display signal including the visual overlay to a display device (130a; 130b; 130c) of the surgical microscope system. system.

2. the two or more input modalities are two or more input modalities of a tactile input device (120; 125) different from the display device of the surgical microscope system; The system of claim 1 .

3. the two or more input modalities are two or more input modalities of a foot pedal (120) of the surgical microscope system, or two or more input modalities of one or more handles (125) of the surgical microscope system; The system of claim 1 .

4. the two or more input modalities are implemented by a four-way switch on the foot pedal of the surgical microscope system; The system of claim 3.

5. the two or more input modalities are two or more keywords of a voice recognition-based control mechanism (150) of the surgical microscope system; The system of claim 1 .

6. the system is configured to update the selection of the two or more functions based on the progress of the surgical procedure. The system of claim 1 .

7. the system is configured to determine a ranking of features with respect to their relevance at a current step in the progress of the surgical procedure and to select the two or more features based on the ranking. The system of claim 1 .

8. the two or more functions are selected based on a deterministic assignment between the progress of the surgical procedure and a function of the plurality of functions. The system of claim 1 .

9. the two or more features are selected using a machine learning model that is trained to rank the features based on the progress of the surgical procedure. The system of claim 1 .

10. the machine learning model trained to rank the plurality of features is trained based on personal preferences of a surgeon using the surgical microscope system, whereby the two or more features are selected based on the personal preferences of the surgeon. The system of claim 9.

11. the system is configured to track the progress of the surgical procedure based on a sequence of commands issued at the surgical microscope system. The system of claim 1 .

12. the system is configured to track the progress of the surgical procedure using a machine learning model that is trained to track the progress of the surgical procedure based on image data from an optical imaging sensor (140) of the surgical microscope system. The system of claim 1 .

13. the surgical procedure includes a sequence of steps, each step of the sequence of steps including one or more tasks depicted in the image data; the machine learning model is trained to detect the task shown in the image data, and the machine learning model is trained to output information about the task shown in the image data; the system is configured to track the progress of the surgical procedure based on the information about the task output by the machine learning model. The system of claim 12.

14. 1. A method for a surgical microscope system, the method comprising: Tracking the progress of the surgical procedure (210); selecting (220) two or more functions of a plurality of functions of the surgical microscope system based on the progress of the surgical procedure; assigning (230) the two or more functions to two or more input modalities of the surgical microscope system; generating (240) a visual overlay with visual representations of the two or more functions, the two or more functions being shown in relation to visual representations of the two or more input modalities; providing (250) a display signal including the visual overlay to a display device of the surgical microscope system; A method comprising:

15. A computer program comprising: The computer program has program code for performing the method of claim 14 when the computer program is run on a processor. Computer program.

Images