Ophthalmic microscope systems, corresponding systems, methods, and computer programs

Abstract

Examples relate to an ophthalmic microscope system, corresponding systems, methods and computer programs for the ophthalmic microscope system. The system includes one or more processors and one or more storage devices. The system is configured to acquire intraoperative sensor data of an eye from at least one imaging device of the ophthalmic microscope system. The system is configured to process the intraoperative sensor data with a machine learning model. The machine learning model is trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data. The system is configured to generate a display signal for a display device of the ophthalmic microscope system based on the information regarding the one or more anatomical features of the eye. The display signal includes a visual guidance overlay for guiding a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye.

Description

Technical Field

[0001] The examples relate to an ophthalmic microscope system, corresponding systems, methods and computer programs for an ophthalmic microscope system, and more particularly, but not exclusively, to concepts for generating a visual guidance overlay for guiding a user of an ophthalmic microscope system.

Background Art

[0002] Visualization of tissue structures is a major focus in surgical microscopy. However, visualization of such tissue structures through the eyepiece of each surgical microscope system can have inherent challenges. For example, in cataract surgery performed using an ophthalmic microscope system, an ophthalmologist often relies on a so-called red reflex that provides an ideal contrast for visualizing the capsule, the lens, and the anterior chamber structures. The red reflex can provide the necessary contrast between the eye's lens and the posterior capsule, thereby providing information about the depth at which the surgeon is working inside the eye. However, in a cataract with progressive turbidity, the transmission of the red reflex light may be blocked by the turbidity of the cataract lens, thereby limiting the intensity of the observed red reflex.

[0003] The posterior surface of the lens capsule poses a barrier between the front and the back during cataract surgery. An accidental tear of the posterior capsule during cataract surgery complicates the removal of the lens, hinders the insertion of the implanted lens, and increases the incidence of postoperative problems. However, the surgeon may have difficulty measuring the depth and strength of the membranous collagen structure, especially when the red reflex illumination is insufficient.

[0004] Retinal surgery in the posterior part of the eye often involves detachment procedures, such as the removal of the epiretinal membrane (ERM) or internal limiting membrane (ILM) to treat various vitreoretinal diseases, including macular holes, macular constriction, epiretinal membranes, diabetic macular edema, and retinal detachment. Since both retinal membranes are semi-transparent and only a few microns thick, surgeons often use toxic dyes such as trypan blue or indocyanine green (ICG) to stain and visualize these membranes.

[0005] Removal of the vitreous humor (fluid) is another crucial workflow step in preventing recurrence of retinal detachment in patients. Surgeons sometimes use steroids to stain the clear vitreous humor white to ensure complete removal of the vitreous pocket in the eye. However, these steroids can also be toxic to the patient, and surgeons often try to limit or completely avoid their use in surgery whenever possible.

[0006] Observations from the posterior region can be further complicated by insufficient or weak illumination from the internal lighting system and translucent tissue features, making it difficult to distinguish ocular features.

[0007] Image recognition, recognition intelligence, and machine learning have been used in the medical field for image analysis, particularly for analyzing images showing the structure of the eye, where this structure has been identified. However, such systems are only used for static data, for example, to annotate still images of the structure of the eye.

[0008] Some intraocular lens (IOL) guidance systems have some image recognition capabilities, limited to identifying pupil size and scleral vessel tortuousness and thickness, to enable correct positioning of the IOL during cataract surgery. However, such systems are limited to preoperative planning and toric IOL positioning and generally do not provide the ability to identify and track features of the eye of interest during surgery. [Overview of the project] [Problems that the invention aims to solve]

[0009] Improvements to the concept of ophthalmic microscope systems may be desired. [Means for solving the problem]

[0010] This request is addressed by the subject matter of the independent claim.

[0011] Various embodiments of this disclosure are based on the finding that a surgeon can be guided during a surgical procedure by employing machine learning-based analysis of intraoperative sensor data from at least one imaging sensor of an ophthalmic microscope system. Anatomical features are identified and tracked in the intraoperative sensor data using machine learning, for example, to classify and locate anatomical features and / or to detect abnormalities related to anatomical features. Visual overlays are generated to be used to highlight or annotate anatomical features of interest within a visual representation of the intraoperative sensor data in order to guide the surgeon during surgery. For example, certain anatomical structures may be highlighted, for example, to highlight the rim of the posterior capsule as described above, or to annotate abnormalities such as a hole in the retina or misalignment of a corneal graft. In practice, the proposed concept provides a method for identifying ocular features of interest, such as the posterior capsule, retinal membrane, or macular hole, using digital augmentation to highlight and track one or more features within a surgical display.

[0012] Various examples of this disclosure relate to a system for an ophthalmic microscope system. The system includes one or more processors and one or more storage devices. The system is configured to acquire intraoperative sensor data of the eye from at least one imaging device of an ophthalmic microscope system. The system is configured to process the intraoperative sensor data using a machine learning model. The machine learning model is trained to output information about one or more anatomical features of the eye based on the intraoperative sensor data. The system is configured to generate a display signal for a display device of an ophthalmic microscope system based on the information about one or more anatomical features of the eye. The display signal includes a visual guidance overlay for guiding the user of the ophthalmic microscope system with respect to one or more anatomical features of the eye. For example, the machine learning model may be used to track anatomical features of the eye (i.e., eye features) in real time, while the visual guidance overlay may be used to annotate or highlight at least some anatomical features to assist the surgeon during surgery.

[0013] Generally, surgeons can be guided by annotating one or more anatomical features of the eye, for example by providing text annotations, by highlighting anatomical features, by highlighting / tracing the edges of one or more anatomical features, or by highlighting abnormalities with respect to one or more anatomical features. Therefore, a visual guidance overlay may include annotations of one or more anatomical features of the eye that are suitable for guiding users of ophthalmic microscope systems during surgical procedures.

[0014] During ophthalmic surgery, the condition of the eye continuously changes due to the surgeon's actions. Therefore, intraoperative sensor data may be updated and continuously processed, and the visual guidance overlay may be updated accordingly. In other words, the system may be configured to acquire intraoperative sensor data as a continuously updated stream of intraoperative sensor data. The system may be configured to update the visual guidance overlay based on the continuously updated stream of intraoperative sensor data.

[0015] A visual guidance overlay is used to guide the user of an ophthalmic microscope system with respect to one or more anatomical features of the eye. Therefore, the visual guidance overlay may be superimposed on, or rather, on, the visual representation of the intraoperative sensor data. In other words, the system may be configured to superimpose the visual guidance overlay on the visual representation of the intraoperative sensor data in the display signal.

[0016] As described above, one or more anatomical features may be annotated within the visual guidance overlay. The system may be configured to generate a visual guidance overlay comprising one or more visual indicators from a plurality of visual indicators. For example, the plurality of visual indicators may include one or more of the following: text annotations for at least a subset of one or more anatomical features; overlays for highlighting one or more surfaces of one or more anatomical features; overlays for highlighting one or more edges of one or more anatomical features; one or more directional indicators; and one or more indicators related to one or more anomalies relating to one or more anatomical features. For example, text annotations may be used during surgical procedures to label one or more anatomical features, to describe anomalies relating to one or more anatomical features, or to describe subsequent tasks. For example, overlays for highlighting the surface or edges of anatomical features can help the surgeon distinguish anatomical features in the visual representation of intraoperative sensor data. Directional operators can guide the surgeon toward the location of further operations to be performed. Similarly, indicators related to anomalies may be used to warn the surgeon of anomalies and to guide the surgeon toward the location of further operations to be performed.

[0017] However, different anatomical features may be of interest to the surgeon at different surgical procedures or different stages of a surgical procedure. Therefore, only a subset of anatomical features may be considered for the visual guidance overlay. In other words, the system may be configured to generate a visual guidance overlay based on a selection of a subset of multiple visual indicators. For example, the selection may be based on input from a user of an ophthalmic microscope system. In other words, the user, for example, a surgeon, may select an anatomical feature or a category of anatomical features for the visual guidance overlay. Additionally or alternatively, the system may be configured to determine this selection based on the progress of an ophthalmic surgical procedure performed with the help of an ophthalmic microscope system. In other words, an automated system may be used to track the progress of the surgical procedure and adjust one or more anatomical features considered based on the progress of the surgical procedure, which can reduce overhead for the surgeon so that the surgeon can concentrate on the surgical procedure.

[0018] The primary tool used during ophthalmic procedures is optical coherence tomography (OCT), which is used to obtain depth profiles of the layers of the eye in one or more scan lines. For example, intraoperative sensor data may include intraoperative OCT sensor data. A machine learning model may be trained to output information about one or more layers of the eye based on the intraoperative OCT sensor data. The system may be configured to generate a visual guidance overlay with visual indicators that highlight or annotate at least a subset of one or more layers of the eye. For example, since the layers of the eye can be difficult to distinguish visually through intraoperative imaging sensor data, OCT is used to guide the surgeon in the depth direction during surgery. Annotating / highlighting features shown in the intraoperative OCT sensor data can facilitate this navigation and allow the surgeon to notice abnormalities.

[0019] In some examples, machine learning models are trained to output information regarding the classification of one or more anatomical features within intraoperative sensor data. The system may be configured to generate a visual guidance overlay with visual indicators related to the classification of one or more anatomical features. For example, this may be used to assist surgeons in distinguishing anatomical features in a visual representation of intraoperative sensor data.

[0020] As described above, the proposed concept can be used to detect and highlight anomalies. Therefore, a machine learning model may be trained to output information about one or more anomalies relating to one or more anatomical features of the eye. The system may be configured to generate a visual guidance overlay with visual indicators for one or more anomalies. For example, a warning message may be displayed when an anomaly is detected. In other words, the system may be configured to include an alarm for one or more anomalies in the display signal, or to output an alarm via the output device of an ophthalmic microscope system. Additionally or alternatively, the location of the anomaly may be highlighted in the visual representation of the intraoperative sensor data, or the anomaly may be added to a list of tasks performed by the surgeon.

[0021] In some examples, the system may include multiple imaging devices, such as the imaging sensor of an ophthalmic microscope system and the OCT system described above. In many cases, intraoperative sensor data from one of the multiple imaging devices may be more suitable for detecting a given anatomical feature. For example, intraoperative OCT sensor data is particularly suitable for detecting and distinguishing layers of the eye. For example, if an anomaly is detected in one of the layers of the eye in the intraoperative OCT sensor data, this anomaly may be highlighted not only in the visual representation of the intraoperative OCT sensor data, but also at the corresponding location in the visual representation of the intraoperative sensor data from the microscope's imaging sensor (or only at the corresponding location in the visual representation of the intraoperative sensor data from the microscope's imaging sensor). More generally, the intraoperative sensor data may include first intraoperative sensor data from a first imaging device and second intraoperative sensor data from a second imaging device. The system may be configured to generate a display signal comprising a first visual representation of first intraoperative sensor data and a second visual representation of second intraoperative sensor data. Based on the first intraoperative sensor data, the system may be configured to superimpose visual indicators of anomalies detected by a machine learning model onto the corresponding positions of the second visual representation of the second intraoperative sensor data in the display signal. For example, corresponding visual indicators (e.g., having the same shape, color, and / or line style) may be superimposed on both visual representations so that the surgeon can recognize the correspondence between these visual indicators.

[0022] Another embodiment that may be used to guide the surgeon and facilitate surgery is the tracking of surgical instruments relative to one or more anatomical features. For example, the distance between a surgical instrument and an anatomical feature can be tracked, allowing the surgeon to operate close to the edges of the anatomical feature without causing undesirable incisions. Thus, the system may be configured to detect the presence of one or more surgical instruments in intraoperative sensor data. The system may be configured to determine the distance between the detected one or more surgical instruments and one or more anatomical features. The system may be configured to generate a visual guidance overlay with a visual indicator representing the distance between the detected one or more surgical instruments and one or more anatomical features.

[0023] As outlined above, the proposed concept allows for the processing of different types of intraoperative sensor data. For example, the intraoperative sensor data may include one or more of the following: intraoperative optical coherence tomography (OCT) sensor data from an intraoperative optical coherence tomography (OCT) device in an ophthalmic microscope system; intraoperative imaging sensor data from the imaging sensor of the microscope in an ophthalmic microscope system; and intraoperative endoscopic sensor data from the endoscope in an ophthalmic microscope system. Different types of intraoperative sensor data are particularly suitable for detecting different types of anatomical features.

[0024] The generated display signal may be output via a display device of an ophthalmic microscope system. For example, the system may be configured to provide the display signal to a display device of an ophthalmic microscope system. For example, the display device may be one of the following: a head-up display, a head-mounted display, a display mounted on the microscope of an ophthalmic microscope system, and an eyepiece display of the microscope of an ophthalmic microscope system.

[0025] Various aspects of the present disclosure relate to corresponding ophthalmic microscope systems that include at least one imaging device, a display device, and the system presented above. For example, the at least one imaging device may include at least one of an intraoperative optical coherence tomography device, an imaging sensor of a microscope, and an endoscope. For example, the intraoperative sensor data of different imaging devices may be particularly suitable for detecting different types of anatomical features.

[0026] Various aspects of the present disclosure relate to corresponding methods for an ophthalmic microscope system. The method includes obtaining intraoperative sensor data of an eye from at least one imaging device of the ophthalmic microscope system. The method includes processing the intraoperative sensor data using a machine learning model. The machine learning model is trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data. The method includes generating a display signal based on the information regarding one or more anatomical features of the eye. The display signal includes a visual guidance overlay for guiding a user of the ophthalmic microscope system regarding one or more anatomical features of the eye.

[0027] Various aspects of the present disclosure relate to corresponding computer programs having program code for implementing the above-described method when the computer program is executed on a processor.

[0028] Hereinafter, some examples of the device and / or method will be described by way of example only, with reference to the accompanying drawings.

Brief Description of the Drawings

[0029] [Figure 1a] FIG. shows a block diagram of an example of a system for an ophthalmic microscope system. [Figure 1b] FIG. shows a schematic diagram of an example of a system for an ophthalmic microscope system in the context of the components of the ophthalmic microscope system. [Figure 1c]This figure shows a schematic diagram of an example of an ophthalmic microscope system. [Figure 2a] This figure shows a schematic diagram of an example of anatomical feature annotations superimposed on a visual representation of camera sensor data. [Figure 2b] This figure shows a schematic diagram of an example of anatomical feature annotations superimposed on a visual representation of camera sensor data. [Figure 3a] This figure shows a schematic diagram of an example of anatomical feature text annotations superimposed on a visual representation of OCT sensor data. [Figure 3b] This figure shows a schematic diagram of an example of graphical annotations of anatomical features superimposed on a visual representation of OCT sensor data. [Figure 3c] This figure shows schematic diagrams illustrating examples of visual representations of camera sensor data and OCT sensor data, shown side by side. [Figure 4a] This figure shows a schematic diagram of examples of multi-layered text and graphical annotations for the eye. [Figure 4b] This figure shows a schematic diagram of an example of graphical annotations of multiple layers of the eye, superimposed on a visual representation of OCT sensor data. [Figure 5a] This figure shows a schematic diagram of an example of a graphical annotation that highlights posterior capsule rupture. [Figure 5b] This figure shows a schematic diagram of an example of a graphical annotation that highlights posterior capsule rupture. [Figure 6a] This figure shows a schematic diagram of an example of graphical annotations that highlight retinal holes, overlaid on a visual representation of camera sensor data and a visual representation of OCT sensor data. [Figure 6b] This figure shows a schematic diagram of an example of graphical annotations that highlight abnormalities in a corneal graft, overlaid on a visual representation of camera sensor data and a visual representation of OCT sensor data. [Figure 6c]This figure shows a schematic diagram of an example of graphical annotations that highlight open incisions, overlaid on a visual representation of camera sensor data and a visual representation of OCT sensor data. [Figure 7] This figure shows a schematic diagram of an example of a graphical annotation that highlights the relative distance between the tip of an instrument and anatomical features. [Figure 8] This diagram shows a flowchart of an example method for an ophthalmic microscope system. [Figure 9] This is a schematic diagram of a system that includes a microscope and a computer system. [Modes for carrying out the invention]

[0030] Next, various examples will be explained more fully by referring to the attached drawings, which show several examples. In the drawings, the thickness of lines, layers, and / or areas may be exaggerated for clarity.

[0031] Figure 1a shows a block diagram of an example system 110 for an ophthalmic microscope system (shown in Figure 1c). Optionally, the system includes an interface 112. System 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the system includes an interface 112. One or more processors are connected to one or more storage devices and optional interfaces. Generally, the functionality of the system is provided by one or more processors, together with optional interfaces (for exchanging information) and / or one or more storage devices (for storing data).

[0032] The system is configured to acquire intraoperative sensor data of the eye from at least one imaging device 120;142;150 of the ophthalmic microscope system (as shown in Figures 1b and / or 1c) via interface 112, for example. The system is configured to process the intraoperative sensor data using a machine learning model. The machine learning model is trained to output information about one or more anatomical features of the eye based on the intraoperative sensor data. The system is configured to generate display signals for display devices 130;130a;130b;130c of the ophthalmic microscope system (as shown in Figures 1b and / or 1c) based on the information about one or more anatomical features of the eye. The display signals include a visual guidance overlay to guide the user of the ophthalmic microscope system regarding one or more anatomical features of the eye.

[0033] Figure 1a shows system 110 alone. However, system 110 may be connected to one or more optional components of an ophthalmic microscope system, as shown in Figure 1b. Figure 1b shows a schematic diagram of an example system for an ophthalmic microscope system in the context of components of an ophthalmic microscope system. For example, as shown in Figure 1b, system 110 may be connected to at least one imaging device of an ophthalmic microscope system, such as an intraoperative optical coherence tomography (intraoperative OCT or iOCT) device 120, a microscope imaging sensor 142, and an endoscope 150 of the ophthalmic microscope system. For example, the imaging sensor 142 may be incorporated into the microscope (140, as shown in Figure 1c), while the OCT device 120 and the endoscope are used directly in or within the eye 160.

[0034] Figure 1c shows a schematic diagram of an example of an ophthalmic microscope system 100. The ophthalmic microscope system comprises at least one imaging device, e.g., an OCT device 120, an imaging sensor 142 of a microscope 140, or an endoscope 150, display devices 130a; 130b; 130c and system 110. Figure 1c shows three potential display devices: a head-up display 130a, an eyepiece display 130b of a microscope 140, and a display 130c that is mounted on the microscope (or the microscope's holding structure). As the name suggests, the ophthalmic microscope system further includes a microscope 140.

[0035] In general, microscopes such as the microscope 140 shown in Figure 1c are optical instruments suitable for examining objects that may be too small for human visual inspection (alone). For example, a microscope can provide optical magnification of a sample. In modern microscopes, optical magnification is often provided by an imaging sensor such as a camera or the imaging sensor 142 of the microscope 140. In other words, the microscope 140 may be a digital microscope or an optical-digital hybrid microscope. Alternatively, a purely optical approach can be taken. The microscope 140 may further include one or more optical magnification components used to magnify the view on the sample, such as an objective lens (i.e., a lens). In the context of this application, the term “ophthalmic microscope system” is used to encompass parts of a system that are used in conjunction with a microscope, such as system 110, OCT device 120, display 130a to 130c or endoscope 150, but are not part of the actual microscope (which includes optical components and is therefore also referred to as “optics carrier”).

[0036] The microscope system shown in Figure 1c is an ophthalmic microscope system, which is a surgical microscope system for use during ophthalmic surgery, i.e., eye surgery. The ophthalmic microscope system 100 shown in Figure 1c includes a number of optional components, for example, a base unit 105 (which includes system 110) having a (rotating) stand, and a (robot or manual) arm 170 that holds the microscope 140 in place and is coupled to the base unit 105 and the microscope 140. Since this disclosure relates to ophthalmic (surgical) microscopes and ophthalmic microscope systems for use in eye surgery, the sample observed through the microscope is the patient's eye 160 or at least a part of the eye.

[0037] Various examples of the present disclosure are used to generate a visual guidance overlay for guiding a user of an ophthalmic microscope system, such as a surgeon, by annotating one or more anatomical features of the eye with respect to one or more anatomical features of the eye. As a result, this visual guidance overlay is based on machine learning-based analysis of intraoperative sensor data from at least one imaging sensor. Thus, the system is configured to acquire intraoperative sensor data of the eye from at least one imaging device 120;142;150 of the ophthalmic microscope system. For example, the system may be configured to receive intraoperative sensor data via interface 112, i.e., at least one imaging device may be configured to actively provide intraoperative sensor data to the system. Alternatively, the system may be configured to read intraoperative sensor data from at least one imaging device or from a memory located outside the system and outside at least one imaging sensor device.

[0038] In general, the proposed systems are designed for use during surgery and are therefore designed to accommodate the use of “intraoperative” sensor data. Thus, intraoperative sensor data may be sensor data generated during the surgical procedure. For example, intraoperative sensor data may be sensor data collected after the initial incision has been made, rather than sensor data collected before the start of the surgical procedure, such as during preparation for the surgical procedure. In various examples, intraoperative sensor data is continuously updated, and as the surgical procedure progresses, the intraoperative sensor data records the progress of the surgical procedure. Thus, it is also possible to continuously acquire, for example, receive or read new intraoperative sensor data, thereby continuously regenerating the visual guidance overlay based on the latest intraoperative sensor data. As a result, the system may be configured to acquire intraoperative sensor data as a continuously updated stream of intraoperative sensor data. Thus, as will be described in more detail below, the system may be configured to update the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. In practice, intraoperative sensor data and the resulting visual guidance overlay can represent the eye as observed in (near) real time (with a delay of up to 500ms, for example).

[0039] As described above, the ophthalmic microscope system may include various imaging devices such as an iOCT device 120, an imaging sensor 142, or an endoscope 150. The intraoperative sensor data may include, in correspondence, one or more of the intraoperative optical coherence tomography sensor data from the intraoperative optical coherence tomography device 120 of the ophthalmic microscope system, the intraoperative imaging sensor data from the imaging sensor 142 of the microscope 140 of the ophthalmic microscope system, and the intraoperative endoscopic sensor data from the endoscope 150 of the ophthalmic microscope system. In some cases, the intraoperative sensor data may include two or more sets of intraoperative sensor data from, for example, two or more imaging sensors.

[0040] Intraoperative sensor data is processed using machine learning to determine information about one or more anatomical features of the eye based on the intraoperative sensor data. Therefore, the method is designed to recognize and digitally track anatomical (tissue) features using image recognition based on intraoperative sensor data, which can be provided by cameras (such as imaging sensors in microscope systems), intraoperative optical coherence tomography (iOCT) systems, or other imaging equipment. By referencing a library of image and video datasets available for training machine learning models, machine learning (ML) and especially deep learning (DL) based software may be able to identify, locate, and quantify anatomical or pathological features, such as the cornea, iris, anterior chamber angle, and posterior capsule. From intraoperative video from surgical cameras, iOCT, or other forms of imaging equipment, the software may be suitable for making real-time inferences about the features currently being observed. Therefore, the proposed concept provides a method for identifying features of interest in the eye, such as the posterior capsule, retinal membrane, or macular hole, and using digital augmentation to highlight and track these features on a surgical display. In other words, various examples can be used to perform digital visual tagging of organizational characteristics.

[0041] Machine learning may refer to algorithms and statistical models that a computer system can use to perform a particular task without using explicit instructions, rather than relying on models and inference. For example, machine learning may use data transformations that are inferred from the analysis of historical data and / or training data, rather than rule-based data transformations. For example, image content may be analyzed using a machine learning model or a machine learning algorithm. For a machine learning model to analyze image content, the machine learning model may be trained with training images as input and training content information as output. By training a machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize image content, so that image content not included in the training data becomes recognizable using the machine learning model. The same principle may be used in the same way for other types of sensor data: by training a machine learning model with training sensor data and a desired output, the machine learning model "learns" the transformation between sensor data and output, which can then be used to provide output based on the non-trained sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain feature vectors that can be used as input to a machine learning model.

[0042] In the context of this disclosure, a machine learning model is trained to output information about one or more anatomical features of an eye based on intraoperative sensor data. In other words, the intraoperative sensor data is provided to the input side of the machine learning model, and the information about one or more anatomical features of an eye is provided to the output side of the machine learning model. Thus, the machine learning model converts the intraoperative sensor data into information about one or more anatomical features. To perform this conversion, the machine learning model is trained with training data.

[0043] A machine learning model may be trained using training input data. The example above uses a training method called "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, each of which may contain multiple input data values ​​and multiple desired output values; that is, each training sample is associated with a desired output value. By specifying both the training samples and the desired output values, the machine learning model "learns" during training which output values ​​to provide based on input samples similar to the provided samples. In addition to supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack corresponding desired output values. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). A classification algorithm may be used if the output is limited to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. A regression algorithm may be used if the output may have any numerical value (within a range). Similarity learning algorithms may resemble both classification and regression algorithms, but they are based on learning from examples using a similarity function that measures how similar or related two objects are.

[0044] In the proposed concept, a machine learning model is trained to process intraoperative sensor data representing one or more anatomical features for the purpose of deriving and outputting the characteristics of one or more anatomical features. For example, one or more anatomical features may include one or more of at least one layer of the eye, at least one histological structure of the eye, at least one graft, and at least one pathological feature. Generally, there are at least four common categories of information about one or more anatomical features that can be provided by the machine learning model, these categories are identity or classification of one or more anatomical features, location of one or more anatomical features, quantification of one or more anatomical features, and abnormality in one or more anatomical features.

[0045] To facilitate or enable the analysis of one or more anatomical features by a machine learning model, the machine learning model may be trained to detect and segment one or more anatomical features within intraoperative sensor data. In other words, the machine learning model may be trained to segment one or more anatomical features for the purpose of distinguishing individual anatomical features within intraoperative sensor data. For example, supervised learning may be used to train the machine learning model to perform segmentation using a sample of sensor data from the aforementioned sensor as training input data, and a manually segmented version of this sample as the desired output. The segmented anatomical features may then be processed separately (or simultaneously) by the machine learning model. In some examples, the machine learning model may include two or more submodels, namely a first submodel that performs image segmentation to isolate one or more anatomical features, and one or more second submodels that perform further analysis of the segmented anatomical features.

[0046] For example, a machine learning model may be trained to determine the identity of one or more anatomical features. In machine learning, this task is referred to as a “classification” task. In other words, a machine learning model may be trained to classify one or more anatomical features, i.e., to output information regarding the classification of one or more anatomical features in intraoperative sensor data. As outlined above, supervised learning can be used to train a machine learning model to perform classification. For example, training samples representing anatomical features may be provided as training inputs, and information regarding the classification of the anatomical features represented in the training samples may be provided as the desired output of training the machine learning model. In practice, feature recognition is performed on intraoperative sensor data, providing a method for identifying features of the eye of interest, such as the posterior capsule, retinal membrane, or macular hole. As suggested above, one of the machine learning models or a second submodel of the machine learning model may be trained to process each anatomical feature separately based on the segmentation of one or more anatomical features.

[0047] In various examples, the location and / or extent of one or more anatomical features may be output by a machine learning model. For example, the location and / or extent of one or more anatomical features may be output based on the segmentation of one or more anatomical features. In other words, the machine learning model may be trained to output one or more points representing the location and / or extent of one or more anatomical features as a result of segmentation. Similarly, quantification of one or more anatomical features may be performed during a post-processing task, which may be used, for example, to determine the number and / or size of each segmented anatomical feature based on the segmentation of one or more anatomical features.

[0048] As described above, in some examples, intraoperative sensor data includes intraoperative optical coherence tomography (OCT) sensor data. OCT is often used to scan layers of the eye and provide depth analysis of those layers. These layers are represented by intraoperative OCT sensor data and can be segmented and identified by a machine learning model. As a result, the machine learning model may be trained to output information about one or more layers of the eye based on the intraoperative OCT sensor data. To facilitate the processing of different types of intraoperative sensor data, the intraoperative sensor data may be input to the machine learning model as image data. For example, intraoperative OCT sensor data may be converted to image data and provided to the machine learning model as image data.

[0049] In this context, each layer of the eye may be considered a distinct anatomical feature of the eye. As a result, as outlined above, a machine learning model may be trained to output information regarding the classification of one or more layers corresponding to one or more anatomical features of the eye. Similarly, a machine learning model may be trained to output one or more points representing the location and / or extent of one or more layers corresponding to one or more layers corresponding to one or more anatomical features of the eye.

[0050] In some examples, machine learning models are further used for anomaly detection. In other words, a machine learning model may be trained to output information about one or more anomalies relating to one or more anatomical features of the eye. In this context, anomaly detection may be used to identify at least one of two types of anomalies: anomalies relating to anatomical features treated as part of a surgical procedure, such as a hole in the retina shown in relation to Figure 6a or a misaligned corneal graft shown in relation to Figure 6b, and anomalies relating to anatomical features that are undesirable or necessary byproducts of a surgical procedure, such as an incision, as shown in Figure 6c. Again, supervised learning can be used to train a machine learning model to output information about one or more anomalies relating to one or more anatomical features of the eye. For example, a sample of sensor data representing anomalies may be used as a training sample, and the location and / or classification of the anomalies may be used as the desired output for training the machine learning model. As a result, information about one or more anomalies may include information about the location and / or classification of one or more anomalies. For example, one of one or more second submodels may be trained to perform anomaly detection separately for segmented anatomical features.

[0051] In general, machine learning models may be used to perform classification, localization, quantification, or anomaly detection of various different types of anatomical features. However, in many cases, only a subset of anatomical features may be of interest to a user, such as a surgeon. The user may have the software automatically detect all tissue features, or manually select the features of interest that should be identified. When only a subset of features is of interest, the output of the machine learning model may be filtered to include only the anatomical features of interest, or another input may be provided to the machine learning model that represents the features of interest to that model. This input may be considered during the training of the machine learning model.

[0052] While the training of machine learning models has been described in the context of ophthalmic microscope systems and corresponding systems, methods, and computer programs, the training of the machine learning model may be completed before the machine learning model is loaded into the system and used to process intraoperative sensor data. In other words, the machine learning model may be a pre-trained machine learning model trained by entities outside the ophthalmic microscope system.

[0053] The output of a machine learning model is used to generate a display signal along with a visual guidance overlay. In other words, the system is configured to generate a display signal for a display device of an ophthalmic microscope system based on information about one or more anatomical features of the eye. Generally, the display signal may be a signal for driving (e.g., controlling) the display. For example, the display signal may include video data and / or control commands for driving the display. For example, the display signal may be provided through one of the one or more interfaces 112 of the system. Thus, the system 110 may include a video interface 112 suitable for providing video signals to a touchscreen display.

[0054] The display signal includes a visual guidance overlay. As the name suggests, the visual guidance overlay may be superimposed on one or more other visual components of the display signal. For example, the display signal may further include a visual representation of intraoperative sensor data. In other words, the system may be configured to superimpose a visual guidance overlay on the visual representation of intraoperative sensor data in the display signal. Furthermore, the position of one or more elements of the visual guidance overlay, such as visual indicators, may coincide with the corresponding portions of the intraoperative sensor data.

[0055] The system is configured to generate a visual guidance overlay as part of the display signal, which is suitable for guiding the user of the ophthalmic microscope system with respect to one or more anatomical features of the eye. This may be done by annotating intraoperative sensor data with respect to one or more anatomical features, and the visual guidance overlay includes annotations of intraoperative sensor data. In other words, the visual guidance overlay may include annotations of one or more anatomical features of the eye that are suitable for guiding the user of the ophthalmic microscope system during surgical procedures. For example, digital augmentation can be used to highlight and track anatomical features on the surgical display.

[0056] For example, annotation may be performed by including one or more visual indicators in a visual guidance overlay. In other words, the system may be configured to generate a visual guidance overlay having one or more indicators from a plurality of visual indicators. One or more visual indicators may be used to annotate one or more anatomical features. Thus, for example, one or more visual indicators may be superimposed on one or more anatomical features in the display signal such that the position of the visual indicators in the visual guidance overlay coincides with the position of the corresponding anatomical features shown in the representation of the intraoperative sensor data. Such an approach is applicable to various types of visual indicators. For example, the plurality of visual indicators may include one or more of the following: an overlay for highlighting one or more surfaces of one or more anatomical features, an overlay for highlighting one or more edges of one or more anatomical features, one or more directional indicators, and one or more indicators related to one or more anomalies relating to one or more anatomical features. These types of visual indicators may be directly superimposed on the corresponding anatomical features shown in the visual representation of the intraoperative sensor data. In some examples, the plurality of visual indicators may include text annotations for at least a subset of one or more anatomical features. As shown in Figures 2a, 2b, 3a, and 4a, such text annotations may be superimposed near each anatomical feature shown in a visual representation of intraoperative sensor data, which is linked by visual elements, for example, or they may be shown as general information as part of a user interface that is part of a display signal.

[0057] As described above, intraoperative sensor data can be continuously updated during surgical procedures. Correspondingly, the system may be configured to update the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. For example, the system may be configured to periodically regenerate the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. For example, the continuously updated stream of intraoperative sensor data may include a series of samples of intraoperative sensor data. The system may be configured to process at least a subset of samples from this series of samples using a machine learning model (for example, every nth sample or according to a predetermined frequency) and regenerate the visual guidance display based on the latest output of the machine learning model.

[0058] The following sections provide several examples illustrating the use of visual indicators to guide surgeons during surgical procedures.

[0059] As outlined with respect to machine learning models, a machine learning model may be trained to output information regarding the classification of one or more anatomical features in intraoperative sensor data. This classification may be used to input into a visual guidance overlay to provide textual or graphical annotations for each anatomical feature. In other words, the system may be configured to generate a visual guidance overlay with visual indicators related to the classification of one or more anatomical features. Examples of such visual indicators related to classification are shown in Figures 2a to 4b.

[0060] Figures 2a and 2b show schematic diagrams of examples of anatomical feature annotations superimposed on a visual representation of camera sensor data. In Figures 2a and 2b, the software is used to identify, locate, and quantify pathological features in the posterior and anterior regions. From intraoperative video from surgical cameras, iOCT, or other forms of imaging equipment, the software can make real-time inferences about the features currently being observed. For example, Figure 2a shows annotations of visible anatomical features in a camera image of the eye. In Figure 2a, text annotations can be superimposed on the camera image, as indicated by the numbers below. For example, Figure 2a shows annotations for the constriction groove 201, pupil 202, collarette 203, crypt 204, ciliary body region 205, pupillary region 206, and radial groove 207. In Figure 2b, anatomical features, and in particular abnormalities related to anatomical features such as hemorrhage and aneurysms, are annotated as text annotations. Figure 2b shows annotations for soft exudates 211, hemorrhages 212, microaneurysms 213, vascular structures 214, hard exudates 215, optic discs 216, microhemorrhages 217, and macula 218.

[0061] Using a combination of inputs from both a camera, iOCT, or other imaging equipment, the software can interpret anatomical details such as the size and depth of structures. An example application of the proposed concept relates to the detection of the posterior capsule during cataract surgery, which helps guide the surgeon's workflow and prevents accidental tearing and rupture of the capsule when performing hydrodissection, phacoemulsification, and lens placement. This detection of the posterior capsule can be performed based on OCT sensor data, which can be used to distinguish different layers of the eye. Thus, the system may be configured to generate a visual guidance overlay with visual indicators that highlight or annotate at least a subset of one or more layers of the eye. In practice, information about one or more layers of the eye, generated by a machine learning model based on intraoperative OCT sensor data, may be used to generate a visual guidance overlay with visual indicators that highlight or annotate at least a subset of one or more layers of the eye.

[0062] Figures 3a to 3c show the detection of the posterior capsule during surgical procedures to prevent accidental tearing and rupture of the capsule. In particular, Figure 3a shows a schematic diagram of an example of text annotation of anatomical features superimposed on a visual representation of intraoperative OCT sensor data. In Figure 3a, text annotation is used to annotate the anterior capsule 301, IOL 302, and posterior capsule 303.

[0063] In Figure 3b, a different approach is selected. Figure 3b shows a schematic diagram of an example of graphical annotation of anatomical features superimposed on a visual representation of OCT sensor data. In Figure 3b, two visual indicators highlighting the edges of anatomical features are included in a visual guidance overlay, namely line 311 highlighting the edge of the IOL and line 312 highlighting the edge of the posterior capsule.

[0064] Figure 3c shows an example of a display signal containing two visual representations 321;323 of intraoperative sensor data. Figure 3c is a schematic diagram showing an example of a visual representation of camera sensor data and an example of a visual representation of OCT sensor data shown side by side. On the left is the visual representation 321 of intraoperative imaging sensor data from a microscope imaging sensor (hereinafter referred to as the "camera view"), and on the right is the visual representation 323 of intraoperative OCT sensor data (hereinafter referred to as the "OCT view"). The current scan line 322 of the OCT is superimposed on the camera view 321. As will be apparent to those skilled in the art, OCT is a scanning technique for obtaining a three-dimensional scan of a target. However, the visual representation 323 shows only a two-dimensional visual representation of the cross-section of the three-dimensional scan. The OCT scan line 322 represents the position where the cross-section is shown in the visual representation 323 of the intraoperative OCT sensor data. The scan line can be moved by the user, for example, by moving the scan line directly or via the slider control shown below the visual representation 323 of the intraoperative OCT sensor data.

[0065] Various tissue structures can be identified in real time during surgery. To prevent information overload, in some examples, the user can optionally select to view only specific features of interest related to their workflow step. In other words, the system may be configured to generate a visual guidance overlay based on the selection of a subset of multiple visual indicators. For example, one or more subsets of anatomical features, and therefore subsets of multiple visual indicators, may be selected by the user. Additionally or alternatively, the user may select a subset of the type or cause of the visual indicators to determine the multiple visual indicators. Thus, the selection of a subset of multiple visual indicators may be based on user input to the ophthalmic microscope system. This can be done via an interactive graphical user interface or via the microscope's handle and footswitches.

[0066] The selected tissue features may then be digitally augmented onto a surgical display, which may be done by tracing across the margins, providing anatomical information, highlighting structures, or providing directional information or other forms of information to guide the surgical workflow.

[0067] Figures 4a and 4b illustrate an example of how a user can choose to view only the features of a specific area of ​​interest relevant to their workflow step. Figure 4a shows a schematic diagram of examples of text and graphical annotations for multiple layers of the eye. Figure 4a shows the layers, namely the internal limiting membrane (ILM) 401, the retinal nerve fiber layer (RNFL) 402, the retinal ganglion cell layer (GCL) 403, the internal plexiform layer (IPL) 404, the internal granular layer (INL) 405, the external plexiform layer (OPL) 406, the external granular layer (ONL) 407, the external limiting membrane (ELM) 408, the photoreceptor layer (PR) 409, the retinal pigment epithelial cells (RPE) 410, Bruch's membrane (BM) 411, the choroidal capillary plate (CC) 412, and the choroidal parenchyma (CS) 413. For example, the acronym for each layer may be shown on either side of the highlighted layer. For example, a user may select layers of interest by selecting or deselecting them in a diagram similar to, for example, the one shown in Figure 4a.

[0068] Alternatively, the layer of interest, or more generally, the anatomical features of interest, may be automatically selected based on the progress of the surgical procedure. For example, the system may be configured to determine the selection of a subset of multiple visual indicators based on the progress of an ophthalmic surgical procedure performed with the help of an ophthalmic microscope system. For example, the system may be configured to track the progress of an ophthalmic surgical procedure, for example, based on a preoperative plan, and select a predetermined set of visual indicators from a subset of multiple visual indicators.

[0069] The results of user or system selections are shown in Figure 4b. Figure 4b shows a schematic diagram of an example of graphical annotations of multiple layers of the eye superimposed on a visual representation of OCT sensor data. In Figure 4a, only a subset of these layers (photoreceptor layer 424, internal limiting membrane 425, retinal nerve fiber layer 426, and outer granular layer 427) are highlighted by linear traces. Figure 4b further shows the camera view 421 and OCT view 423 in the eye and indicates the location of the OCT scan lines 422.

[0070] Additionally, warnings can be provided on the screen to alert to potential complications, for example, due to abnormalities in tissue structure or potential errors in surgical technique. Thus, as described above, a machine learning model may be trained to output information about one or more abnormalities relating to one or more anatomical features of the eye. The system may be configured to generate a visual guidance overlay with visual indicators relating to one or more abnormalities. In other words, the visual guidance overlay may include visual indicators that warn or notify the user of one or more abnormalities. For example, the visual indicators may include one or more of a warning message, a warning pictogram, and / or visual indicators that outline the location of the abnormality. Thus, the system may be configured to include an alarm for one or more abnormalities within the display signal. Alternatively or additionally, the system may be configured to output the alarm via an output device of the ophthalmic microscope system, such as a warning light or speaker.

[0071] As shown in Figures 5a and 5b, warnings can be provided on the screen to alert for potential complications resulting from abnormal tissue structures. Useful applications of such warnings include the detection of abnormal deformation of shape or microtears in the posterior capsule, which indicate a high risk of posterior capsule rupture when performing hydrodissection, phacoemulsification, and lens placement. Figures 5a and 5b show schematic diagrams of examples of graphical annotations highlighting posterior capsule rupture. Figure 5a shows two views of the eye, namely camera view 501 and OCT view 505, where the OCT scan line 502 is superimposed on camera view 501. In the camera view, two superimposed lines 503;504 are shown, highlighting the abnormal tissue structure, in this case the first and second edges of the rupture in the posterior capsule. Corresponding ruptures at two different OCT scan positions are highlighted by lines 506;507 in OCT view 505. Optionally, a warning message 508 “Posterior capsule rupture” may be shown in addition to one or more warning pictograms 509.

[0072] Furthermore, the system can provide surgical confirmation and navigation guidance once a specific procedure is completed. For example, holes remaining in the retina can be detected through image recognition using iOCT and highlighted on the surgical display for easy tracking. Another example relates to corneal transplantation, where the proposed system can detect whether the graft is in the correct orientation and accurately aligned across the transplantation site. Figures 6a to 6c show corresponding examples.

[0073] Figure 6a shows an example of surgical confirmation and navigation guidance based on whether a specific procedure has been completed. In this example, a hole remaining in the retina can be detected through image recognition using iOCT and highlighted on a surgical display for easy tracking. Figure 6a shows a schematic diagram of an example of graphical annotations highlighting holes in the retina, superimposed on a visual representation of camera sensor data and a visual representation of OCT sensor data. Figure 6a shows two views of the eye, namely camera view 601 and OCT view 607, similar to Figures 4b to 5b. The OCT scan lines 602 are superimposed on camera view 601. In Figure 6a, the first and second holes in the retina are highlighted by circles 603;606a. The second hole in OCT view 607 is highlighted by the corresponding circle 606b. In addition, directional markers 604;605 are shown for the two holes. For example, the directional markers may be used to guide the surgeon toward the location of the holes.

[0074] As can be seen in Figure 6a, two or more sets of intraoperative sensor data are acquired from two or more imaging devices. In some cases, an anomaly may be detected in one set of intraoperative sensor data, and an indicator may be superimposed on the visual representation of the other set of intraoperative sensor data. For example, in Figure 6a, an anomaly is detected in the intraoperative OCT sensor data and is superimposed on the visual representation of the intraoperative OCT sensor data and intraoperative imaging sensor data from the microscope's imaging sensor. In other words, the intraoperative sensor data may include first intraoperative sensor data from a first imaging device (e.g., intraoperative OCT sensor data) and second intraoperative sensor data from a second imaging device (e.g., intraoperative imaging sensor data). The system may be configured to generate a display signal having a first visual representation of the first intraoperative sensor data and a second visual representation of the second intraoperative sensor data. For example, the camera view 601 in Figure 6a may show a second visual representation of the second intraoperative sensor data, and the OCT view 607 in Figure 6a may show a first visual representation of the first intraoperative sensor data. The system may be configured to superimpose, for example, visual indicators of anomalies 603;604;605;606a detected by a machine learning model based on the first intraoperative sensor data onto the corresponding positions of the second visual representation 601 of the second intraoperative sensor data in the display signal, in addition to visual indicators of anomalies 606b superimposed on the visual representation 607 of the first intraoperative sensor data. Another example of this concept is shown in Figure 6b.

[0075] Figure 6b shows another example of surgical confirmation and navigation guidance based on whether a particular procedure has been completed. In this example, the system recognizes and prompts if the corneal graft is not positioned correctly or precisely aligned. Figure 6b shows a schematic diagram of an example of graphical annotation highlighting abnormalities in the corneal graft, which is superimposed on the visual representation of camera sensor data and OCT sensor data. Figure 6b shows the camera view 611 and the first and second OCT views 616;617. Since two OCT devices are used, the first scan line 612 of the first OCT and the second scan line 614 of the second OCT are shown. In this example, triangles 613a;615a are used to highlight the first and second abnormalities in the corneal graft superimposed on the camera view, while triangles 613b;615b highlight the same abnormalities in the first and second OCT views 616;617. For example, corresponding visual indicators (i.e., triangles) superimposed on different visual representations of intraoperative sensor data may have the same shape, the same color, and / or the same line pattern.

[0076] Figure 6c shows another example of surgical confirmation and navigation guidance based on whether a specific procedure has been completed. In this example, the system recognizes and prompts if the incision is open and further hydration is needed. Figure 6c shows a schematic diagram of an example of a graphical annotation highlighting an open incision, which is superimposed on the visual representation of the camera sensor data and the visual representation of the OCT sensor data. Figure 6c shows the camera view 621, the first OCT view 625, and the second OCT view 627. Figure 6c further shows the corresponding first and second OCT scan lines 622;623. In Figure 6c, an ellipse 624a highlighting the open incision is superimposed on the camera view, and a corresponding ellipse 624b highlighting the open incision is superimposed on the OCT view. In addition, the warning message 626 “Hydration is needed and the wound is not closed” is displayed.

[0077] In some examples, instrument detection may be incorporated to track the relative distance between the instrument tip and tissue structure. For example, the shape profile of the instrument may be identified to track the relative distance between the instrument tip and tissue structure. This allows the surgeon to perform precise, visually guided manipulation tasks while tracking the depth and distance of their instrument relative to the tissue feature of interest.

[0078] Therefore, the system may be configured to detect the presence of one or more surgical instruments in the intraoperative sensor data. For example, the system may be configured to detect the presence of one or more surgical instruments using an object detection algorithm, which is, for example, a visual object matching algorithm or a machine learning model trained to detect one or more instruments in the intraoperative sensor data (e.g., using supervised learning-based training of a machine learning model). For example, the machine learning model used to process the intraoperative sensor data may be further trained to detect and locate one or more instruments in the intraoperative sensor data.

[0079] The system may further be configured to determine the distance between one or more detected surgical instruments and one or more anatomical features. For example, intraoperative sensor data used to detect one or more surgical instruments, such as the intraoperative sensor data shown in Figure 7, may have a scale known relative to one or more surgical instruments, or the size of one or more surgical instruments may be used to determine the scale of the intraoperative sensor data. The system may be configured to determine the distance between one or more detected surgical instruments and one or more anatomical features based on the scale of the intraoperative sensor data and based on the distance (e.g., in pixels) between one or more detected surgical instruments and one or more anatomical features in a visual representation of the intraoperative sensor data. Alternatively, the distance can be associated with the distance (e.g., in pixels) between one or more detected surgical instruments and one or more anatomical features in a visual representation of each intraoperative sensor data. The system may be configured to generate a visual guidance overlay with a visual indicator representing the distance between one or more detected surgical instruments and one or more anatomical features. For example, to improve the visibility of the distance between one or more detected surgical instruments, a visual indicator representing the distance between one or more detected surgical instruments may include a numerical representation of the distance, or the visual indicator may increase the contrast of the visual representation and / or highlight the edges and / or anatomical edges of one or more surgical instruments. In some examples, if this distance is below a threshold, the visual indicator representing the distance may include a proximity warning.

[0080] Figure 7 shows an example of how instrument detection can be incorporated to track the relative distance between the instrument tip and tissue structure. Figure 7 shows a schematic diagram of an example of graphical annotation that highlights the relative distance between the instrument tip and anatomical features. Figure 7 shows camera view 701 and OCT view 704. The instrument tip 705 is detected in intraoperative OCT sensor data and highlighted in the visual representation of visual sensor data. Circles 702 and 703 are used to highlight the instrument tip in the camera view and OCT view. For example, Figure 7 shows circle 702a highlighting the tip of a first instrument in the camera view, the corresponding circle 702b highlighting the tip of the first instrument in the OCT view, circle 703a highlighting the tip of a second instrument in the camera view, and the corresponding circle 703b highlighting the tip of a second instrument in the OCT view. Similar to the examples shown in Figures 6a to 6c, corresponding visual indicators, such as circles, that outline the tip of the same instrument may have the same color, shape, and / or line pattern.

[0081] The display signal is generated for the display device of the ophthalmic microscope system. Accordingly, the system may be configured to provide the display signal to the display device of the ophthalmic microscope system. For example, the surgical display may be in the form of a 3D head-up surgical monitor, a digital viewer mounted on the head or attached to the microscope, or an image loading device into the microscope eyepiece. Thus, as shown in Figure 1c, the display device may be one of the following: a head-up display 130a (i.e., a display that the surgeon looks at straight ahead rather than looking down at the surgical site), a head-mounted display (not shown), such as virtual reality goggles or augmented reality glasses or mixed reality glasses, a display 130c attached to the microscope of the ophthalmic microscope system, and an eyepiece display 130b of the microscope of the ophthalmic microscope system.

[0082] One or more interfaces 112 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​by a specified code, within a module, between modules, or between modules of different entities. For example, one or more interfaces 112 may include interface circuits configured to receive and / or transmit information. In embodiments, one or more processors 114 may be implemented using any means for processing, such as one or more processing units, one or more processing units, a processor operable with appropriately adapted software, a computer, or a programmable hardware component. In other words, the described functions of one or more processors 114 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, and the like. In at least some embodiments, one or more storage devices 116 may include at least one element from a group of computer-readable recording media, such as magnetic recording media or optical recording media, for example, hard disk drives, flash memory, solid state disks (SSDs), floppy disks, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or network storage.

[0083] Further details and embodiments of the system and ophthalmic microscope system are referred to in relation to the proposed concept or one or more examples described above or below (e.g., Figures 8 to 9). The system and ophthalmic microscope system may include one or more additional optional features corresponding to one or more embodiments of the proposed concept or one or more examples described above or below.

[0084] Figure 8 shows a flowchart of an example of a corresponding (computer-implemented) method for an ophthalmic microscope system, for example, an ophthalmic microscope system introduced in relation to Figures 1a to 7. For example, this method may be carried out by system 110 introduced in relation to Figures 1a to 7. This method includes acquiring intraoperative sensor data of the eye from at least one imaging device of the ophthalmic microscope system 810. This method includes processing the intraoperative sensor data using a machine learning model 820. The machine learning model is trained to output information about one or more anatomical features of the eye based on the intraoperative sensor data. This method includes generating a display signal 830 based on the information about one or more anatomical features of the eye. The display signal includes a visual guidance overlay to guide the user of the ophthalmic microscope system regarding one or more anatomical features of the eye.

[0085] Optionally, this method may include one or more further features, for example, one or more features introduced by the systems introduced in relation to Figures 1a to 7 or by the surgical microscope system.

[0086] Further details and embodiments of the method are referred to in relation to the proposed concept or one or more examples described above or below (e.g., Figures 1a to 7, Figure 9). The method may include one or more additional optional features corresponding to one or more embodiments of the proposed concept or one or more examples described above or below.

[0087] Some embodiments relate to microscopes that include systems such as those described in relation to one or more of Figures 1 to 8. Alternatively, the microscope may be part of a system such as those described in relation to one or more of Figures 1 to 8, or may be connected to a system such as those described in relation to one or more of Figures 1 to 8. Figure 9 shows a schematic diagram of a system 900 configured to carry out the methods described herein. The system 900 includes a microscope 910 and a computer system 920. The microscope 910 is configured to take images and is connected to the computer system 920. The computer system 920 is configured to carry out at least a portion of the methods described herein. The computer system 920 may be configured to run machine learning algorithms. The computer system 920 and the microscope 910 may be separate entities or may be integrated within a common housing. The computer system 920 may be part of the central processing system of the microscope 910, and / or the computer system 920 may be part of a dependent component of the microscope 910, such as a sensor, actor, camera, or illumination unit of the microscope 910.

[0088] The computer system 920 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed to various locations such as local clients and / or one or more remote server farms and / or data centers). The computer system 920 may include any circuit or combination of circuits. In one embodiment, the computer system 920 may include one or more processors, which can be of any kind. As used herein, the processor may be intended to be any kind of computing circuit, such as a microprocessor for a microscope or microscopic component (e.g., a camera), a microcontroller, a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multicore processor, a field-programmable gate array (FPGA), or any other kind of processor or processing circuit. Other types of circuits that may be included in the computer system 920 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (communication circuits, etc.) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 920 may also include one or more storage devices that may contain one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 920 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touchscreen, voice recognition device, or any other device which enables a user of the system to input information into and receive information from the computer system 920.

[0089] Some or all of the steps may be performed by a hardware device (or by using a hardware device), such as a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most critical steps may be performed by such a device.

[0090] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is feasible using a non-transient recording medium, which is a digital recording medium, etc., that stores electronically readable control signals and cooperates (or can cooperate) with a programmable computer system to carry out each method. Examples include floppy disks, hard disk drives (HDDs), solid-state disks (SSDs), DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs, or FLASH memory. Thus, the digital recording medium may be computer-readable.

[0091] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so as to carry out any of the methods described herein.

[0092] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to perform one of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.

[0093] Another embodiment includes a computer program stored in a machine-readable carrier for carrying out any of the methods described herein.

[0094] Therefore, in other words, embodiments of the present invention are computer programs having program code for carrying out any of the methods described herein when the computer program is executed on a computer.

[0095] Accordingly, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for carrying out any of the methods described herein when executed by a processor. The data carrier, digital recording medium, or recording medium is typically tangible and / or non-transient. Another embodiment of the present invention is an apparatus, such as those described herein, comprising a processor and a recording medium.

[0096] Therefore, another embodiment of the present invention is a data stream or signal sequence representing a computer program for carrying out any of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0097] Another embodiment includes processing means, for example, a computer or programmable logic device configured or adapted to carry out any of the methods described herein.

[0098] Another embodiment includes a computer having an installed computer program for carrying out any of the methods described herein.

[0099] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for carrying out any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0100] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to carry out any of the methods described herein. Generally, the methods are advantageously carried out by any hardware device.

[0101] The embodiments may be based on the use of machine learning models or machine learning algorithms. Referring to Figures 1a to 7, two learning approaches, namely supervised learning and semi-supervised learning, were discussed.

[0102] In addition to supervised or semi-supervised learning, unsupervised learning may be used to train machine learning models. In unsupervised learning, input data (only) may be provided, and unsupervised learning algorithms may be used to find structure in the input data (for example, by grouping or clustering the input data, or by finding commonalities in the data). Clustering is the process of assigning input data containing multiple input values ​​into multiple subsets (clusters), so that input values ​​within the same cluster are similar according to one or more (pre-defined) similarity criteria, but are not similar to input values ​​in another cluster.

[0103] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as “software agents”) are trained to take actions in their surroundings. A reward is calculated based on the actions taken. Reinforcement learning is based on training one or more software agents to choose actions that result in software agents that perform better on a given task, with cumulative rewards increasing (as revealed by the increase in rewards).

[0104] Furthermore, several techniques may be applied as part of a machine learning algorithm. For example, feature representation learning may be used. In other words, a machine learning model may be trained at least partially using feature representation learning, and / or a machine learning algorithm may include feature representation learning components. A feature representation learning algorithm, which may be called a representation learning algorithm, may not only store information in its own input but may also transform the information to make it useful, often as a preprocessing step before performing classification or prediction. Feature representation learning may be based, for example, on principal component analysis or cluster analysis.

[0105] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide the identification of input values ​​that raise suspicion by being significantly different from the majority of the input or training data. In other words, a machine learning model may be trained with anomaly detection, at least in part, and / or a machine learning algorithm may include anomaly detection components.

[0106] In some examples, a machine learning algorithm may use a decision tree as its predictive model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by branches of the decision tree, and the output values ​​corresponding to these items may be represented by leaves of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. When discrete values ​​are used, the decision tree may be represented as a classification tree, and when continuous values ​​are used, the decision tree may be represented as a regression tree.

[0107] Correlation rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model may be based on one or more correlation rules. Correlation rules are created by identifying relationships between variables in a large amount of data. A machine learning algorithm may identify and / or utilize one or more correlational rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply knowledge.

[0108] Machine learning algorithms are typically based on machine learning models. In other words, the term “machine learning algorithm” may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term “machine learning model” may refer to a set of data structures and / or rules that represent learned knowledge (for example, based on training performed by a machine learning algorithm). In embodiments, usage of a machine learning algorithm may mean usage of one underlying machine learning model (or multiple underlying machine learning models). Usage of a machine learning model may mean that a machine learning model and / or a set of data structures / rules that are a machine learning model are trained by a machine learning algorithm.

[0109] For example, a machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or brain. An ANN consists of multiple interconnected nodes and multiple junctions, so-called edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (simply) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (nonlinear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weights" of the edges or nodes that provide the input. The weights of nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0110] Alternatively, a machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with a relevant learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing inputs with multiple training input values ​​belonging to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, a machine learning model may be a Bayesian network, which is a stochastic directed acyclic graphical model. A Bayesian network may use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, a machine learning model may be based on a search algorithm and a genetic algorithm, which is a heuristic method that mimics the process of natural selection.

[0111] As used herein, the term "and / or" includes all possible combinations of one or more of the related items and may be abbreviated as " / ".

[0112] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus. [Explanation of Symbols]

[0113] 100 Ophthalmic Microscope Systems 105 Base Unit 110 System 112 Interfaces 114 processors 116 Storage Devices 120 OCT equipment 130 Display devices 130a Head-Up Display 130b Eyepiece Display Display that can be attached to a 130c microscope 140 Microscopes 142 Optical imaging sensors 150 Endoscopes 160 eyes 170 Arm 201-218 Text annotations on anatomical features 301 Front pouch 302 IOL 303 Posterior pouch 311 Lines that highlight the edges of IOLs 312 Lines highlighting the posterior capsule 321 Camera View 322 OCT scan lines 323 OCT View 401-413 Layers of the eye 421 Camera View 422 OCT scan lines 423 OCT View 424-427 Lines that highlight the eye layer 501 Camera View 502 OCT scan lines 503, 504 Lines that highlight the edges of abnormal tissue structures. 505 OCT View Lines highlighting abnormal tissue structures in OCT (Optical Coherence Tomography) 506, 507 508 Warning message: "Posterior capsule rupture" 509 Warning pictograms 601 Camera View 602 OCT scan lines 603,606a Circles that highlight holes in the retina in the camera view. 604, 605 Direction markers Circle highlighting a hole in the retina in the 606b OCT view. 607 OCT View 611 Camera View 612,614 OCT scan lines 613a;615a Triangles that highlight corneal graft abnormalities in the camera view. 613b;615b Triangles highlighting corneal graft abnormalities in the OCT view 616,617 OCT views 621 Camera View 622,623 OCT scan lines 624a Ellipse that highlights open incisions in the camera view Ellipses highlighting open incisions in the 624b OCT view. 625,627 OCT views 626 Warning message: "Hydration is needed, and the wound is not closed." 701 Camera View 702a, 703a Circles that highlight the tip of the device in the camera view. Circles that highlight the tip of the instrument in the 702b, 703b OCT view. 704 OCT View 705 Tip of the instrument 810 Acquire intraoperative sensor data. 820 Processing intraoperative sensor data using machine learning models. 830 Generating display signals with visual guidance overlays. 900 System 910 Microscope 920 Computer Systems

Claims

1. A system (110;920) for an ophthalmic microscope system (100;900), The system includes one or more processors (114) and one or more storage devices (116), The aforementioned system, The steps include acquiring intraoperative sensor data of the eye from at least one imaging device (120; 142; 150) of the ophthalmic microscope system, A step of processing the intraoperative sensor data using a machine learning model, wherein the machine learning model is trained to output information about one or more anatomical features of the eye based on the intraoperative sensor data; A step of generating a display signal for a display device (130a; 130b; 130c) of the ophthalmic microscope system based on the information relating to one or more anatomical features of the eye, wherein the display signal includes a visual guidance overlay for guiding the user of the ophthalmic microscope system with respect to one or more anatomical features of the eye; A step of generating a visual guidance overlay comprising one or more visual indicators from a plurality of visual indicators, wherein the plurality of visual indicators includes one or more directional indicators; The steps include generating the visual guidance overlay based on the selection of a subset of the plurality of visual indicators, The steps of determining the selection based on the progress of an ophthalmic surgical procedure performed with the help of the ophthalmic microscope system and It is configured to do the following: System (110; 920).

2. The visual guidance overlay includes annotations of one or more anatomical features of the eye, suitable for guiding the user of the ophthalmic microscope system during a surgical procedure. The system according to claim 1.

3. The system is configured to acquire the intraoperative sensor data as a continuously updated stream of intraoperative sensor data. The system is configured to update the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. The system according to claim 1.

4. The system is configured to superimpose the visual guidance overlay on the visual representation of the intraoperative sensor data within the display signal. The system according to claim 1.

5. The aforementioned intraoperative sensor data includes intraoperative optical coherence tomography sensor data. The machine learning model is trained to output information about one or more layers of the eye based on the intraoperative optical coherence tomography sensor data. The system is configured to generate the visual guidance overlay, which includes visual indicators that highlight or annotate at least a subset of the one or more layers of the eye. The system according to claim 1.

6. The machine learning model is trained to output information regarding the classification of one or more anatomical features in the intraoperative sensor data. The system is configured to generate the visual guidance overlay, which includes a visual indicator related to the classification of one or more anatomical features. The system according to claim 1.

7. The machine learning model is trained to output information about one or more abnormalities relating to one or more anatomical features of the eye. The system is configured to generate the visual guidance overlay which includes a visual indicator for one or more anomalies. The system according to claim 1.

8. The intraoperative sensor data includes first intraoperative sensor data from a first imaging device and second intraoperative sensor data from a second imaging device. The system is configured to generate a display signal comprising a first visual representation of the first intraoperative sensor data and a second visual representation of the second intraoperative sensor data. The system is configured to superimpose a visual indicator of an anomaly detected by the machine learning model, based on the first intraoperative sensor data, onto the corresponding position of the second visual representation of the second intraoperative sensor data in the display signal. The system according to claim 7.

9. The system is configured to detect the presence of one or more surgical instruments in the intraoperative sensor data. The system is configured to determine the distance between the detected one or more surgical instruments and the one or more anatomical features. The system is configured to generate the visual guidance overlay, which includes a visual indicator representing the distance between the detected one or more surgical instruments and the one or more anatomical features. The system according to claim 1.

10. The intraoperative sensor data includes one or more of the following: intraoperative optical coherence tomography sensor data from the intraoperative optical coherence tomography device (120) of the ophthalmic microscope system; intraoperative imaging sensor data from the imaging sensor (142) of the microscope (140; 910) of the ophthalmic microscope system; and intraoperative endoscopic sensor data from the endoscope (150) of the ophthalmic microscope system. The system according to claim 1.

11. An ophthalmic microscope system (100; 900), The system comprises at least one imaging device (120; 142; 150), a display device (130a; 130b; 130c), and the system according to any one of claims 1 to 10 (110; 920), Ophthalmic microscope system (100; 900).

12. A method for an ophthalmic microscope system (100; 900), wherein the method is The steps include (810) acquiring intraoperative sensor data of the eye from at least one imaging device of the ophthalmic microscope system, A step (820) of processing the intraoperative sensor data using a machine learning model, wherein the machine learning model is trained to output information relating to one or more anatomical features of the eye based on the intraoperative sensor data, Step (830) of generating a display signal based on information relating to one or more anatomical features of the eye, wherein the display signal includes a visual guidance overlay for guiding the user of the ophthalmic microscope system with respect to one or more anatomical features of the eye; A step of generating a visual guidance overlay comprising one or more visual indicators from a plurality of visual indicators, wherein the plurality of visual indicators includes one or more directional indicators; The steps include generating the visual guidance overlay based on the selection of a subset of the plurality of visual indicators, The steps of determining the selection based on the progress of an ophthalmic surgical procedure performed with the help of the ophthalmic microscope system and A method that includes this.

13. It is a computer program, The computer program has program code for carrying out the method described in claim 12 when it is executed on a processor, Computer program.

Images