Ophthalmic microscope system, corresponding system, method and computer program product

Abstract

Examples relate to an ophthalmic microscope system, corresponding systems, methods and computer programs for the ophthalmic microscope system. The system includes one or more processors and one or more storage devices. The system is configured to acquire intraoperative sensor data of an eye from at least one imaging device of the ophthalmic microscope system. The system is configured to process the intraoperative sensor data with a machine learning model. The machine learning model is trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data. The system is configured to generate a display signal for a display device of the ophthalmic microscope system based on the information regarding the one or more anatomical features of the eye. The display signal includes a visual guidance overlay for guiding a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye.

Description

[Technical field]

[0001] Examples relate to ophthalmic microscope systems, corresponding systems, methods and computer programs for ophthalmic microscope systems, and more particularly, but not exclusively, to concepts for generating visual guidance overlays for guiding users of ophthalmic microscope systems. [Background technology]

[0002] Visualization of tissue structures is a major focus in surgical microscopy. However, visualization of such tissue structures through the eyepieces of each surgical microscope system can have its own challenges. For example, in cataract surgery performed with an ophthalmic microscope system, ophthalmologists often rely on the so-called red reflex, which provides ideal contrast to visualize the capsule, lens, and anterior chamber structures. The red reflex can provide the necessary contrast between the lens and the posterior capsule of the eye, which provides information about the depth at which the surgeon is working within the eye. However, in a cloudy cataract, the transmission of red reflex light can be blocked by the opacification of the cataract lens, which limits the intensity of the red reflex observed.

[0003] The posterior aspect of the lens capsule presents a barrier between the anterior and posterior segments during cataract surgery. Accidental tearing of the posterior capsule during cataract surgery complicates lens removal, impedes insertion of a lens implant, and increases the incidence of postoperative problems. However, surgeons can have difficulty measuring the depth and strength of the membranous collagen structure, especially when red-reflex illumination is lacking.

[0004] Retinal surgery at the posterior segment of the eye often involves peeling procedures, such as the peeling of the epiretinal membrane (ERM) or inner limiting membrane (ILM) to treat a variety of vitreoretinal diseases, including macular holes, macular purging, epiretinal membranes, diabetic macular edema, and retinal detachment. Because both retinal membranes are translucent and only a few microns thick, surgeons often use toxic dyes, such as trypan blue or indocyanine green (ICG), to stain and visualize these membranes.

[0005] Removal of the vitreous (fluid) is another important workflow step to prevent recurrence of retinal detachment in patients. Surgeons sometimes use steroids to stain the clear vitreous white to ensure complete removal of the vitreous pocket in the eye. However, these steroids can be toxic to patients as well, and surgeons often try to limit or completely avoid the use of these steroids in surgery, when possible.

[0006] Observation at the posterior segment can be further complicated by insufficient or poor illumination from internal illumination systems and translucent tissue features, making it difficult to distinguish ocular features.

[0007] Image recognition, recognition intelligence and machine learning have also been used to analyze images in the medical field, particularly images showing ocular structures, to identify said ocular structures, but such systems are only used on static data, for example to annotate still images of ocular structures.

[0008] Some intraocular lens (IOL) guidance systems have some image recognition capabilities that are limited to identifying pupil size and scleral vascular tortuosity and thickness to enable correct positioning of an intraocular lens during cataract surgery. However, such systems are limited to preoperative planning and alignment of toric IOLs and generally do not provide the ability to identify and track ocular features of interest intraoperatively. Summary of the Invention [Problem to be solved by the invention]

[0009] It may be desirable to improve the concept of ophthalmic microscope systems. [Means for solving the problem]

[0010] This need is addressed by the subject matter of the independent claims.

[0011] Various embodiments of the present disclosure are based on the finding that machine learning-based analysis of intraoperative sensor data of at least one imaging sensor of an ophthalmic microscope system can be employed to guide a surgeon during a surgical procedure. Anatomical features are identified and tracked in the intraoperative sensor data using machine learning, for example, to classify and locate the anatomical features and / or to detect anomalies related to the anatomical features. Visual overlays are generated that are used to highlight or annotate anatomical features of interest in a visual representation of the intraoperative sensor data to guide the surgeon during surgery. For example, some anatomical structures may be highlighted, for example, to highlight the edge of the posterior capsule as mentioned above or to annotate anomalies such as a hole in the retina or a misorientation of the corneal graft. In effect, the proposed concept provides a method for identifying ocular features of interest, such as the posterior capsule, retinal membrane, or macular hole, using digital augmentation to highlight and track one or more features in a surgical display.

[0012] Various examples of the present disclosure relate to a system for an ophthalmic microscope system. The system includes one or more processors and one or more storage devices. The system is configured to acquire intraoperative sensor data of an eye from at least one imaging device of the ophthalmic microscope system. The system is configured to process the intraoperative sensor data using a machine learning model. The machine learning model is trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data. The system is configured to generate a display signal for a display device of the ophthalmic microscope system based on the information regarding the one or more anatomical features of the eye. The display signal includes a visual guidance overlay for guiding a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye. For example, the machine learning model may be used to track anatomical features (i.e., eye features) of the eye in real time, whereas the visual guidance overlay may be used to annotate or highlight at least some anatomical features to assist a surgeon during surgery.

[0013] Generally, the surgeon may be guided by annotating one or more anatomical features of the eye, for example, by providing text annotations, by highlighting the anatomical features, by highlighting / tracking edges of the one or more anatomical features, or by highlighting abnormalities with respect to the one or more anatomical features. Thus, the visual guidance overlay may include annotations of one or more anatomical features of the eye suitable for guiding a user of the ophthalmic microscope system during a surgical procedure.

[0014] During ophthalmic surgery, the condition of the eye changes continuously due to manipulations by the surgeon. Thus, the intraoperative sensor data may be updated and continuously processed, and the visual guidance overlay may be updated accordingly. In other words, the system may be configured to acquire the intraoperative sensor data as a continuously updated stream of intraoperative sensor data. The system may be configured to update the visual guidance overlay based on the continuously updated stream of intraoperative sensor data.

[0015] The visual guidance overlay is used to guide a user of the ophthalmic microscope system with respect to one or more anatomical features of the eye. Thus, the visual guidance overlay may be superimposed on the intraoperative sensor data, or rather, on a visual representation of the intraoperative sensor data. In other words, the system may be configured to superimpose the visual guidance overlay on a visual representation of the intraoperative sensor data in the display signal.

[0016] As described above, one or more anatomical features may be annotated in the visual guidance overlay. The system may be configured to generate a visual guidance overlay comprising one or more visual indicators of a plurality of visual indicators. For example, the plurality of visual indicators include one or more of a text annotation of at least a subset of the one or more anatomical features, an overlay for highlighting one or more surfaces of the one or more anatomical features, an overlay for highlighting one or more edges of the one or more anatomical features, one or more directional indicators, and one or more indicators associated with one or more anomalies related to the one or more anatomical features. For example, the text annotation may be used during a surgical procedure to label one or more anatomical features, to describe anomalies related to the one or more anatomical features, or to describe a subsequent task. For example, an overlay for highlighting a surface or edge of an anatomical feature may assist the surgeon in distinguishing anatomical features in a visual representation of intraoperative sensor data. A directional operator may guide the surgeon towards a location of a further operation to be performed. Similarly, an indicator associated with anomalies may be used to alert the surgeon to the anomaly and to guide the surgeon towards a location of a further operation to be performed.

[0017] However, different anatomical features may be of interest to the surgeon during different surgical procedures or different stages of the surgical procedure. Thus, only a subset of the anatomical features may be considered for the visual guidance overlay. In other words, the system may be configured to generate the visual guidance overlay based on a selection of a subset of the plurality of visual indicators. For example, the selection may be based on an input of a user of the ophthalmic microscope system. In other words, a user, e.g., a surgeon, may select an anatomical feature or a category of anatomical features for the visual guidance overlay. Additionally or alternatively, the system may be configured to determine this selection based on the progress of the ophthalmic surgical procedure performed with the aid of the ophthalmic microscope system. In other words, an automated system may be used to track the progress of the surgical procedure and adjust the one or more anatomical features considered based on the progress of the surgical procedure, which may reduce overhead for the surgeon so that the surgeon can focus on the surgical procedure.

[0018] A primary tool during ophthalmic procedures is optical coherence tomography (OCT), which is used to obtain a depth profile of the layers of the eye in one or more scan lines. For example, the intraoperative sensor data may include intraoperative optical coherence tomography sensor data. A machine learning model may be trained to output information regarding one or more layers of the eye based on the intraoperative optical coherence tomography sensor data. The system may be configured to generate a visual guidance overlay comprising visual indicators that highlight or annotate at least a subset of the one or more layers of the eye. For example, layers of the eye may be difficult to visually distinguish via intraoperative imaging sensor data, so OCT is used to guide the surgeon in depth during surgery. Annotating / highlighting features shown in the intraoperative OCT sensor data may facilitate this navigation and alert the surgeon to abnormalities.

[0019] In some examples, the machine learning model is trained to output information regarding a classification of one or more anatomical features in the intraoperative sensor data. The system may be configured to generate a visual guidance overlay comprising a visual indicator associated with the classification of the one or more anatomical features. For example, this may be used to assist a surgeon in distinguishing anatomical features in a visual representation of the intraoperative sensor data.

[0020] As described above, the proposed concept can be used to detect and highlight anomalies. Thus, a machine learning model may be trained to output information regarding one or more anomalies regarding one or more anatomical features of the eye. The system may be configured to generate a visual guidance overlay with a visual indicator regarding the one or more anomalies. For example, a warning message may be shown when an anomaly is detected. In other words, the system may be configured to include an alert regarding the one or more anomalies in the display signal or output an alert via an output device of the ophthalmic microscope system. Additionally or alternatively, the location of the anomaly may be highlighted in a visual representation of the intraoperative sensor data, or the anomaly may be added to a list of tasks to be performed by the surgeon.

[0021] In some examples, the system may include multiple imaging devices, such as a microscope imaging sensor of an ophthalmic microscope system and an OCT system as described above. In many cases, intraoperative sensor data of one of the imaging devices may be better suited to detect a given anatomical feature. For example, the intraoperative OCT sensor data may be particularly well suited to detect and differentiate between layers of the eye. For example, if an abnormality is detected with respect to one of the layers of the eye in the intraoperative OCT sensor data, the abnormality may not only be highlighted in the visual representation of the intraoperative OCT sensor data, but also (or only) in the corresponding location of the visual representation of the intraoperative sensor data of the microscope imaging sensor. In more general terms, the intraoperative sensor data may include first intraoperative sensor data from a first imaging device and second intraoperative sensor data from a second imaging device. The system may be configured to generate a display signal comprising a first visual representation of the first intraoperative sensor data and a second visual representation of the second intraoperative sensor data. The system may be configured to overlay a visual indicator of an abnormality detected by the machine learning model based on the first intraoperative sensor data over a corresponding location of the second visual representation of the second intraoperative sensor data in the display signal. For example, visual indicators that correspond (e.g., have the same shape, color, and / or line style) may be overlaid over both visual representations, allowing the surgeon to recognize the correspondence between the visual indicators.

[0022] Another aspect that may be used to guide the surgeon to facilitate surgery is the tracking of a surgical instrument relative to one or more anatomical features. For example, the distance between the surgical instrument and the anatomical feature may be tracked, allowing the surgeon to operate close to the edge of the anatomical feature without causing undesired incisions. Thus, the system may be configured to detect the presence of one or more surgical instruments in the intraoperative sensor data. The system may be configured to determine the distance between the detected one or more surgical instruments and the one or more anatomical features. The system may be configured to generate a visual guidance overlay comprising a visual indicator representing the distance between the detected one or more surgical instruments and the one or more anatomical features.

[0023] As outlined in the above description, different types of intraoperative sensor data can be processed by the proposed concept. For example, the intraoperative sensor data may include one or more of intraoperative optical coherence tomography sensor data of an intraoperative optical coherence tomography device of an ophthalmic microscope system, intraoperative imaging sensor data of an imaging sensor of a microscope of an ophthalmic microscope system, and intraoperative endoscopic sensor data of an endoscope of an ophthalmic microscope system. For example, different types of intraoperative sensor data are particularly suitable for detecting different types of anatomical features.

[0024] The generated display signal may be output via a display device of the ophthalmic microscope system. For example, the system may be configured to provide the display signal to a display device of the ophthalmic microscope system. For example, the display device may be one of a head-up display, a head-mounted display, a display mounted on a microscope of the ophthalmic microscope system, and an eyepiece display of the microscope of the ophthalmic microscope system.

[0025] Various aspects of the present disclosure relate to a corresponding ophthalmic microscope system including at least one imaging device, a display device, and the system presented above. For example, the at least one imaging device may include at least one of an intraoperative optical coherence tomography device, a microscopic imaging sensor, and an endoscope. For example, intraoperative sensor data of different imaging devices may be particularly suitable for detecting different types of anatomical features.

[0026] Various aspects of the present disclosure relate to a corresponding method for an ophthalmic microscope system. The method includes acquiring intraoperative sensor data of an eye from at least one imaging device of the ophthalmic microscope system. The method includes processing the intraoperative sensor data with a machine learning model. The machine learning model is trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data. The method includes generating a display signal based on the information regarding the one or more anatomical features of the eye. The display signal includes a visual guidance overlay to guide a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye.

[0027] Various aspects of the present disclosure also relate to corresponding computer programs having program codes for performing the above-mentioned methods, when the computer program runs on a processor.

[0028] Some examples of apparatus and / or methods are now described, by way of example only, and with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0029] [Figure 1a] FIG. 1 illustrates a block diagram of an example system for an ophthalmic microscope system. [Figure 1b] FIG. 1 illustrates a schematic diagram of an example system for an ophthalmic microscope system in the context of the components of the ophthalmic microscope system. [Figure 1c]FIG. 1 shows a schematic diagram of an example ophthalmic microscope system. [Figure 2a] FIG. 13 shows a schematic diagram of example annotations of anatomical features overlaid on a visual representation of camera sensor data. [Figure 2b] FIG. 13 shows a schematic diagram of example annotations of anatomical features overlaid on a visual representation of camera sensor data. [Figure 3a] FIG. 13 shows a schematic diagram of an example of text annotations of anatomical features overlaid on a visual representation of OCT sensor data. [Figure 3b] FIG. 13 shows a schematic diagram of an example of graphical annotation of anatomical features overlaid on a visual representation of OCT sensor data. [Figure 3c] FIG. 1 shows a schematic diagram of an example visual representation of camera sensor data and an example visual representation of OCT sensor data shown side-by-side. [Figure 4a] FIG. 1 shows a schematic diagram of examples of textual and graphical annotations of multiple layers of the eye. [Figure 4b] FIG. 13 shows a schematic diagram of an example of graphical annotation of multiple layers of the eye overlaid on a visual representation of OCT sensor data. [Figure 5a] FIG. 13 shows a schematic diagram of an example of a graphical annotation highlighting a posterior capsular rupture. [Figure 5b] FIG. 13 shows a schematic diagram of an example of a graphical annotation highlighting a posterior capsular rupture. [Figure 6a] FIG. 13 shows a schematic diagram of an example of a graphical annotation highlighting a retinal hole superimposed on a visual representation of camera sensor data and a visual representation of OCT sensor data. [Figure 6b] FIG. 13 shows a schematic diagram of an example of graphical annotations highlighting anomalies in a corneal implant superimposed on a visual representation of camera sensor data and a visual representation of OCT sensor data. [Figure 6c]FIG. 13 shows a schematic diagram of an example of a graphical annotation highlighting an open incision superimposed on a visual representation of camera sensor data and a visual representation of OCT sensor data. [Figure 7] FIG. 13 shows a schematic diagram of an example of a graphical annotation highlighting the relative distance between an instrument tip and an anatomical feature. [Figure 8] FIG. 1 shows a flowchart of an example method for an ophthalmic microscope system. [Figure 9] FIG. 1 shows a schematic diagram of a system including a microscope and a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Various examples will now be described more fully with reference to the accompanying drawings, in which several examples are shown and in which the thickness of lines, layers and / or regions may be exaggerated for clarity.

[0031] FIG. 1a shows a block diagram of an example of a system 110 for an ophthalmic microscope system (shown in FIG. 1c). Optionally, the system includes an interface 112. The system 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the system includes an interface 112. The one or more processors are connected to one or more storage devices and optional interfaces. In general, the functionality of the system is provided by one or more processors, e.g., together with optional interfaces (for exchanging information) and / or one or more storage devices (for saving data).

[0032] The system is configured to acquire intraoperative sensor data of the eye from at least one imaging device 120; 142; 150 (as shown in FIG. 1b and / or FIG. 1c) of an ophthalmic microscope system, for example, via an interface 112. The system is configured to process the intraoperative sensor data using a machine learning model. The machine learning model is trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data. The system is configured to generate a display signal for a display device 130; 130a; 130b; 130c (as shown in FIG. 1b and / or FIG. 1c) of the ophthalmic microscope system based on the information regarding the one or more anatomical features of the eye. The display signal includes a visual guidance overlay to guide a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye.

[0033] In FIG. 1a, the system 110 is shown alone. However, the system 110 may be connected to one or more optional components of an ophthalmic microscope system, as shown in FIG. 1b. FIG. 1b shows a schematic diagram of an example of a system for an ophthalmic microscope system in the context of the components of the ophthalmic microscope system. For example, as shown in FIG. 1b, the system 110 may be connected to at least one imaging device of an ophthalmic microscope system, such as an intraoperative optical coherence tomography (intraoperative OCT or iOCT) device 120, an imaging sensor 142 of the microscope, and an endoscope 150 of the ophthalmic microscope system. For example, the imaging sensor 142 may be integrated into the microscope (140, as shown in FIG. 1c), while the OCT device 120 and the endoscope are used directly at or in the eye 160.

[0034] FIG. 1c shows a schematic diagram of an example of an ophthalmic microscope system 100. The ophthalmic microscope system comprises at least one imaging device, such as an OCT device 120, an imaging sensor 142 of a microscope 140, or an endoscope 150, display devices 130a; 130b; 130c, and a system 110. In FIG. 1c, three potential display devices are shown: a head-up display 130a, an eyepiece display 130b of the microscope 140, and a display 130c attached to the microscope (or a holding structure of the microscope). As the name suggests, the ophthalmic microscope system further includes a microscope 140.

[0035] Generally, a microscope, such as the microscope 140 shown in FIG. 1c, is an optical instrument suitable for inspecting objects that may be too small for a human to inspect visually (alone). For example, a microscope may provide optical magnification of a sample. In modern microscopes, optical magnification is often provided for an imaging sensor, such as a camera or imaging sensor 142 of the microscope 140. In other words, the microscope 140 may be a digital microscope or a combined optical-digital microscope. Alternatively, a purely optical approach may be taken. The microscope 140 may further include one or more optical magnification components, such as an objective lens (i.e., a lens), that are used to magnify the view on the sample. In the context of this application, the term "ophthalmic microscope system" is used to encompass parts of the system that are not part of the actual microscope (which includes optical components and is therefore also referred to as "optics carrier"), but are used in conjunction with the microscope, such as the system 110, the OCT device 120, the displays 130a to 130c, or the endoscope 150.

[0036] The microscope system shown in Fig. 1c is an ophthalmic microscope system, which is a surgical microscope system for use during ophthalmic surgery, i.e. eye surgery. The ophthalmic microscope system 100 shown in Fig. 1c includes a number of optional components, such as a base unit 105 (containing the system 110) having a (rotatable) stand, and an arm 170 (robotic or manual) that holds the microscope 140 in place and is coupled to the base unit 105 and the microscope 140. Since the present disclosure relates to ophthalmic (surgical) microscopes and ophthalmic microscope systems for use in eye surgery, the sample observed through the microscope is a patient's eye 160 or at least a part of the eye.

[0037] Various examples of the present disclosure are used to generate a visual guidance overlay for guiding a user of an ophthalmic microscope system, e.g., a surgeon, by annotating one or more anatomical features of the eye with respect to one or more anatomical features of the eye. As a result, the visual guidance overlay is based on a machine learning-based analysis of intraoperative sensor data of at least one imaging sensor. Thus, the system is configured to obtain intraoperative sensor data of the eye from at least one imaging device 120; 142; 150 of the ophthalmic microscope system. For example, the system may be configured to receive the intraoperative sensor data via the interface 112, i.e., the at least one imaging device may be configured to actively provide the intraoperative sensor data to the system. Alternatively, the system may be configured to read the intraoperative sensor data from the at least one imaging device or from a memory that is external to the system and external to the at least one imaging sensor device.

[0038] In general, the proposed system is designed to be used during surgery and is therefore tailored to the use of "intraoperative" sensor data. Thus, the intraoperative sensor data may be sensor data generated during a surgical procedure. For example, the intraoperative sensor data may be sensor data collected after an initial incision has been made, rather than sensor data collected before the start of a surgical procedure, such as during preparation for the surgical procedure. In various examples, the intraoperative sensor data is continuously updated, and as the surgical procedure progresses, the intraoperative sensor data records the progress of the surgical procedure. Thus, the intraoperative sensor data may also be continuously newly acquired, e.g., received or read, such that the visual guidance overlay may be continuously regenerated based on the latest intraoperative sensor data. As a result, the system may be configured to acquire the intraoperative sensor data as a continuously updated stream of intraoperative sensor data. Thus, as described in more detail below, the system may be configured to update the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. In practice, the intraoperative sensor data and the visual guidance overlay generated on top of it can represent the eye being observed in (near) real time (e.g., with a delay of up to 500 ms).

[0039] As discussed above, the ophthalmic microscope system may include various imaging devices, such as an iOCT device 120, an imaging sensor 142, or an endoscope 150. The intraoperative sensor data may correspondingly include one or more of intraoperative optical coherence tomography sensor data of an intraoperative optical coherence tomography device 120 of the ophthalmic microscope system, intraoperative imaging sensor data of an imaging sensor 142 of a microscope 140 of the ophthalmic microscope system, and intraoperative endoscopic sensor data of an endoscope 150 of the ophthalmic microscope system. In some cases, the intraoperative sensor data may include two or more sets of intraoperative sensor data, for example from two or more imaging sensors.

[0040] Based on the intraoperative sensor data, the intraoperative sensor data is processed using machine learning to determine information about one or more anatomical features of the eye. Thus, a method is provided for recognizing and digitally tracking anatomical (tissue) features using image recognition based on the intraoperative sensor data, which can be provided by a camera (such as the imaging sensor of a microscope system), an intraoperative optical coherence tomography (iOCT) system, or other imaging equipment. By referencing a library of image and video datasets available for training machine learning models, software based on machine learning (ML) and especially deep learning (DL) may be able to identify, localize and quantify anatomical or pathological features, such as, for example, differentiation of the cornea, iris, anterior chamber angle, posterior capsule, etc. From the intraoperative video from the surgical camera, iOCT or other form of imaging equipment, the software may be suitable to make inferences in real time about the features currently being observed. Thus, the proposed concept provides a method for identifying an ocular feature of interest, such as the posterior capsule, retinal membrane or macular hole, and using digital augmentation to highlight and track this feature in a surgical display. In other words, various embodiments can perform digital visual tagging of tissue features.

[0041] Instead of relying on models and inference, machine learning may refer to algorithms and statistical models that a computer system may use to perform a particular task without the use of explicit instructions. For example, machine learning may use data transformations inferred from analysis of past data and / or training data instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or using a machine learning algorithm. For a machine learning model to analyze image content, the machine learning model may be trained with training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize image content, such that image content not included in the training data becomes recognizable using the machine learning model. The same principle may be used for other types of sensor data in a similar manner: by training the machine learning model with training sensor data and a desired output, the machine learning model "learns" a transformation between sensor data and output that can be used to provide an output based on the non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata and / or image data) may be pre-processed to obtain feature vectors that are used as input to a machine learning model.

[0042] In the context of the present disclosure, the machine learning model is trained to output information about one or more anatomical features of the eye based on the intraoperative sensor data. In other words, the intraoperative sensor data is provided to the input side of the machine learning model, and the information about one or more anatomical features of the eye is provided to the output side of the machine learning model. Thus, the machine learning model converts the intraoperative sensor data into information about one or more anatomical features. To perform this conversion, the machine learning model is trained with training data.

[0043] The machine learning model may be trained using training input data. The above example uses a training method called "supervised learning". In supervised learning, the machine learning model is trained using multiple training samples, where each sample may include multiple input data values ​​and multiple desired output values, i.e., each training sample is associated with a desired output value. By specifying both the training samples and the desired output value, the machine learning model "learns" during training which output value to provide based on input samples that are similar to the provided sample. Besides supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on supervised learning algorithms (e.g. classification algorithms, regression algorithms or similarity learning algorithms). Classification algorithms may be used when the output is restricted to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. Regression algorithms may be used when the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two objects are.

[0044] In the proposed concept, a machine learning model is trained to process intraoperative sensor data representing one or more anatomical features for the purpose of deriving and outputting characteristics of the one or more anatomical features. For example, the one or more anatomical features may include one or more of at least one layer of the eye, at least one tissue structure of the eye, at least one implant, and at least one pathological feature. In general, there are (at least) four general categories of information about the one or more anatomical features that may be provided by the machine learning model, which are identity or classification of the one or more anatomical features, location of the one or more anatomical features, quantification of the one or more anatomical features, and abnormalities in the one or more anatomical features.

[0045] To facilitate or enable analysis of one or more anatomical features by the machine learning model, the machine learning model may be trained to detect and segment one or more anatomical features in the intraoperative sensor data. In other words, the machine learning model may be trained to perform segmentation of one or more anatomical features in order to distinguish individual anatomical features in the intraoperative sensor data. For example, supervised learning may be used to train the machine learning model to perform segmentation using samples of sensor data from the above-mentioned sensors as training input data and manually segmented versions of the samples as the desired output. The segmented anatomical features may then be processed separately (or simultaneously) by the machine learning model. In some examples, the machine learning model may include two or more sub-models, i.e., a first sub-model that performs image segmentation to isolate one or more anatomical features, and one or more second sub-models that perform further analysis of the segmented anatomical features.

[0046] For example, the machine learning model may be trained to determine the identity of one or more anatomical features. In machine learning, this task is expressed as a "classification" task. In other words, the machine learning model may be trained to classify one or more anatomical features, i.e., to output information regarding the classification of one or more anatomical features in the intraoperative sensor data. As outlined in the above description, supervised learning may be used to train the machine learning model to perform classification. For example, training samples representing anatomical features may be provided as training inputs, and information regarding the classification of the anatomical features represented in the training samples may be provided as a desired output of training the machine learning model. In practice, feature recognition is performed on the intraoperative sensor data, providing a method for identifying ocular features of interest, such as the posterior capsule, retinal membrane, or macular hole. As alluded to above, the machine learning model or one of the second sub-models of the machine learning model may be trained to process each anatomical feature separately based on the segmentation of one or more anatomical features.

[0047] In various examples, the position and / or extent of one or more anatomical features are output by the machine learning model. For example, the position and / or extent of one or more anatomical features may be output based on the segmentation of one or more anatomical features. In other words, the machine learning model may be trained to output one or more points representing the position and / or extent of one or more anatomical features as a result of the segmentation. Similarly, quantification of one or more anatomical features may be performed based on the segmentation of one or more anatomical features during a post-processing task that may be used to determine, for example, the number and / or size of each segmented anatomical feature.

[0048] As mentioned above, in some examples, the intraoperative sensor data includes intraoperative optical coherence tomography sensor data. Optical coherence tomography is often used to scan layers of the eye, providing a depth analysis of the eye layers. These layers may be represented by the intraoperative OCT sensor data and segmented and identified by the machine learning model. As a result, the machine learning model may be trained to output information regarding one or more layers of the eye based on the intraoperative optical coherence tomography sensor data. To facilitate processing of different types of intraoperative sensor data, the intraoperative sensor data may be input to the machine learning model as image data. For example, the intraoperative OCT sensor data may be converted to image data and provided to the machine learning model as image data.

[0049] In this context, each layer of the eye may be considered as a separate anatomical feature of the eye. As a result, as outlined in the above description, the machine learning model may be trained to output information regarding the classification of one or more layers corresponding to one or more anatomical features of the eye. Similarly, the machine learning model may be trained to output one or more points representing the position and / or extent of one or more layers corresponding to one or more anatomical features of the eye.

[0050] In some examples, the machine learning model is further used for anomaly detection. In other words, the machine learning model may be trained to output information about one or more anomalies related to one or more anatomical features of the eye. In this context, anomaly detection may be used to identify at least one of two types of anomalies: anomalies related to anatomical features that are treated as part of a surgical procedure, such as a hole in the retina shown in connection with FIG. 6a, or a misoriented corneal graft shown in connection with FIG. 6b, and anomalies related to anatomical features that are undesirable or required by-products of a surgical procedure, such as an incision wound, as shown in FIG. 6c. Again, supervised learning may be used to train the machine learning model to output information about one or more anomalies related to one or more anatomical features of the eye. For example, samples of sensor data representative of anomalies may be used as training samples, and the location of the anomaly and / or the classification of the anomaly may be used as a desired output for training the machine learning model. As a result, the information about the one or more anomalies may include information about the location and / or classification of the one or more anomalies. For example, one second sub-model of the one or more second sub-models may be trained to perform anomaly detection separately on the segmented anatomical features.

[0051] In general, machine learning models may be used to perform classification, localization, quantification, or anomaly detection of various different types of anatomical features. However, in many cases, only a subset of anatomical features may be of interest to a user, e.g., a surgeon. The user may allow the software to automatically detect all tissue features, or may manually select the features of interest to be identified. If only a subset of features is of interest, the output of the machine learning model may be filtered to include (only) the anatomical features of interest, or another input may be provided to the machine learning model, indicating the features that are of interest to the machine learning model. This input may be taken into account in training the machine learning model.

[0052] Although the training of the machine learning model has been described in the context of an ophthalmic microscope system and corresponding systems, methods, and computer programs, the training of the machine learning model may be completed before the machine learning model is loaded into the system and used to process intraoperative sensor data. In other words, the machine learning model may be a pre-trained machine learning model that is trained by an entity external to the ophthalmic microscope system.

[0053] The output of the machine learning model is used to generate a display signal together with a visual guidance overlay. In other words, the system is configured to generate a display signal for a display device of the ophthalmic microscope system based on information about one or more anatomical features of the eye. In general, the display signal may be a signal for driving (e.g., controlling) a display. For example, the display signal may include video data and / or control instructions for driving the display. For example, the display signal may be provided through one interface of one or more interfaces 112 of the system. Thus, the system 110 may include a video interface 112 suitable for providing a video signal to a touch screen display.

[0054] The display signal includes a visual guidance overlay. As the name suggests, the visual guidance overlay may be superimposed over one or more other visual components of the display signal. For example, the display signal may further include a visual representation of the intraoperative sensor data. In other words, the system may be configured to superimpose the visual guidance overlay over the visual representation of the intraoperative sensor data in the display signal. Furthermore, the location of one or more elements of the visual guidance overlay, such as visual indicators, may coincide with a corresponding portion of the intraoperative sensor data.

[0055] The system is configured to generate a visual guidance overlay as part of the display signal, the visual guidance overlay being suitable for guiding a user of the ophthalmic microscope system with respect to one or more anatomical features of the eye. This may be done by annotating intraoperative sensor data with respect to the one or more anatomical features, the visual guidance overlay including annotations of the intraoperative sensor data. In other words, the visual guidance overlay may include annotations of one or more anatomical features of the eye suitable for guiding a user of the ophthalmic microscope system during a surgical procedure. For example, digital augmentation may be used to highlight and track the anatomical features in the surgical display.

[0056] For example, annotation may be performed by including one or more visual indicators in the visual guidance overlay. In other words, the system may be configured to generate a visual guidance overlay having one or more indicators of the plurality of visual indicators. The one or more visual indicators may be used to annotate one or more anatomical features. Thus, for example, one or more visual indicators may be superimposed on one or more anatomical features in the display signal such that the position of the visual indicator in the visual guidance overlay matches the position of the corresponding anatomical feature shown in the representation of the intraoperative sensor data. Such an approach is applicable to various types of visual indicators. For example, the plurality of visual indicators may include one or more of an overlay for highlighting one or more surfaces of the one or more anatomical features, an overlay for highlighting one or more edges of the one or more anatomical features, one or more directional indicators, and one or more indicators related to one or more anomalies with respect to the one or more anatomical features. These types of visual indicators may be superimposed directly on the corresponding anatomical feature shown in the visual representation of the intraoperative sensor data. In some examples, the plurality of visual indicators may include a text annotation of at least a subset of the one or more anatomical features. As shown in Figures 2a, 2b, 3a and 4a, such text annotations may be shown, for example, superimposed near each anatomical feature shown in the visual representation of the intraoperative sensor data to which they are linked by visual elements, or may be shown as general information as part of a user interface that is part of the display signal.

[0057] As described above, the intraoperative sensor data can be continuously updated during a surgical procedure. Correspondingly, the system may be configured to update the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. For example, the system may be configured to periodically regenerate the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. For example, the continuously updated stream of intraoperative sensor data may include a series of samples of the intraoperative sensor data. The system may be configured to process at least a subset of the samples of the series of samples with a machine learning model (e.g., every nth sample or according to a predefined frequency) and regenerate the visual guidance display based on the latest output of the machine learning model.

[0058] Below, several examples are provided that illustrate the use of visual indicators to guide the surgeon during a surgical procedure.

[0059] As outlined with respect to the machine learning model, the machine learning model may be trained to output information regarding a classification of one or more anatomical features in the intraoperative sensor data. This classification may be used to populate a visual guidance overlay to provide textual or graphical annotations of each anatomical feature. In other words, the system may be configured to generate a visual guidance overlay comprising visual indicators associated with the classification of one or more anatomical features. Examples of such visual indicators associated with the classification are shown in Figures 2a through 4b.

[0060] 2a and 2b show schematic diagrams of examples of annotations of anatomical features superimposed on a visual representation of camera sensor data. In 2a and 2b, software is used to identify, locate and quantify pathological features in the posterior and anterior segments. From intraoperative video from a surgical camera, iOCT or other forms of imaging equipment, the software can make inferences in real time about the features currently being observed. For example, in 2a, annotations of anatomical features visible in a camera image of the eye are shown. In 2a, text annotations can be superimposed on the camera image, as represented by the following numbers. For example, 2a shows annotations of the contractile sulcus 201, pupil 202, collarette 203, crypt 204, ciliary body region 205, pupil region 206 and radial sulcus 207. In 2b, anatomical features and abnormalities related to the anatomical features, particularly hemorrhages and aneurysms, are annotated as text annotations. FIG. 2b shows annotations of soft exudates 211, hemorrhages 212, microaneurysms 213, vasculature 214, hard exudates 215, optic disc 216, microhemorrhages 217 and macula 218.

[0061] Using a combination of inputs from both the camera, the iOCT or other imaging equipment, the software can perform interpretation of anatomical details, such as the size and depth of the structures. An example application of the proposed concept relates to the detection of the posterior capsule during cataract surgery, which helps guide the surgeon's workflow to prevent accidental tearing and rupture of the capsule when performing hydrodissection, phacoemulsification and lens placement. This detection of the posterior capsule can be performed based on OCT sensor data, which can be used to distinguish different layers of the eye. Thus, the system may be configured to generate a visual guidance overlay comprising visual indicators that highlight or annotate at least a subset of one or more layers of the eye. In practice, information regarding one or more layers of the eye generated by a machine learning model based on intraoperative OCT sensor data may be used to generate a visual guidance overlay comprising visual indicators that highlight or annotate at least a subset of one or more layers of the eye.

[0062] 3a-c show detection of the posterior capsule during a surgical procedure to prevent accidental tearing and rupture of the capsule. In particular, FIG. 3a shows a schematic diagram of an example of text annotation of anatomical features superimposed on a visual representation of intraoperative OCT sensor data. In FIG. 3a, text annotations are used to annotate the anterior capsule 301, the IOL 302, and the posterior capsule 303.

[0063] A different approach is chosen in Fig. 3b, which shows a schematic diagram of an example of graphical annotation of anatomical features superimposed on a visual representation of OCT sensor data. In Fig. 3b, two visual indicators highlighting the edges of anatomical features are included in the visual guidance overlay: line 311 highlighting the edge of the IOL and line 312 highlighting the edge of the posterior capsule.

[0064] In Fig. 3c, an example of a display signal is shown that includes two visual representations 321; 323 of intraoperative sensor data. Fig. 3c shows a schematic diagram of an example of a visual representation of camera sensor data and an example of a visual representation of OCT sensor data shown side by side. On the left side, a visual representation 321 of intraoperative imaging sensor data of a microscope imaging sensor is shown (hereinafter referred to as "camera view"), and on the right side, a visual representation 323 of intraoperative OCT sensor data is shown (hereinafter referred to as "OCT view"). Overlaid on top of the camera view 321 is a current scan line 322 of the OCT. As will be clear to those skilled in the art, OCT is a scanning technique for obtaining a three-dimensional scan of a target. However, the visual representation 323 shows only a two-dimensional visual representation of a cross-section of the three-dimensional scan. The OCT scan line 322 represents the location where the cross-section is shown in the visual representation 323 of the intraoperative OCT sensor data. The scan lines can be moved by the user, for example, by moving the scan lines directly or via a slider control shown below the visual representation 323 of the intra-operative OCT sensor data.

[0065] Various tissue structures can be identified intraoperatively and in real-time. To prevent information overload, in some examples, the user can additionally select to view only specific features of interest that are relevant to his or her workflow step. In other words, the system may be configured to generate a visual guidance overlay based on a selection of a subset of the plurality of visual indicators. For example, a subset of one or more anatomical features, and thus a subset of the plurality of visual indicators, may be selected by the user. Additionally or alternatively, the user may select a subset of types or causes of visual indicators to determine the plurality of visual indicators. Thus, the selection of the subset of the plurality of visual indicators may be based on an input of a user of the ophthalmic microscope system. This may be done via an interactive graphical user interface or via the handle and foot switch of the microscope.

[0066] The selected tissue features may then be digitally augmented onto the surgical display, either by tracing over edges, providing anatomical information, highlighting structures, providing directional information or other forms of information to guide the surgical workflow.

[0067] In Figures 4a-b, examples are shown of how a user can select to only view features of particular interest that are relevant to their workflow step. Figure 4a shows a schematic diagram of an example of textual and graphical annotations of multiple layers of the eye. Figure 4a shows the layers: Inner limiting membrane (ILM) 401, Retinal nerve fiber layer (RNFL) 402, Retinal ganglion cell layer (GCL) 403, Inner plexiform layer (IPL) 404, Inner nuclear layer (INL) 405, Outer plexiform layer (OPL) 406, Outer nuclear layer (ONL) 407, Outer limiting membrane (ELM) 408, Photoreceptor layer (PR) 409, Retinal pigment epithelium (RPE) 410, Bruch's membrane (BM) 411, Choriocapillaris (CC) 412, and Choroid stroma (CS) 413. For example, the acronyms of each layer may be shown on either side of the highlighted layer. For example, a user may select layers of interest by, for example, selecting or deselecting those layers in a view similar to that shown in FIG. 4a.

[0068] Alternatively, the layer of interest, or more generally the anatomical feature of interest, may be automatically selected based on the progression of a surgical procedure. For example, the system may be configured to determine the selection of the subset of the plurality of visual indicators based on the progression of an ophthalmic surgical procedure performed with the aid of the ophthalmic microscope system. For example, the system may be configured to track the progression of the ophthalmic surgical procedure and select a predefined set of visual indicators for the subset of the plurality of visual indicators, e.g., based on a preoperative plan.

[0069] The result of the user or system selection is shown in FIG. 4b. FIG. 4b shows a schematic diagram of an example of a graphical annotation of multiple layers of the eye superimposed on a visual representation of the OCT sensor data. In FIG. 4a, only a subset of these layers (photoreceptor layer 424, inner limiting membrane 425, retinal nerve fibers 426 and outer nuclear layer 427) are highlighted by linear tracing. FIG. 4b further shows a camera view 421 and an OCT view 423 of the eye, and indicates the location of the OCT scan line 422.

[0070] Additionally, a warning may be provided on the screen to provide a warning for possible complications, for example due to abnormalities in tissue structure or potential errors in surgical technique. Thus, as described above, the machine learning model may be trained to output information regarding one or more abnormalities regarding one or more anatomical features of the eye. The system may be configured to generate a visual guidance overlay comprising a visual indicator regarding the one or more abnormalities. In other words, the visual guidance overlay may include a visual indicator that warns or notifies the user of the one or more abnormalities. For example, the visual indicator may include one or more of a warning message, a warning pictogram, and / or a visual indicator outlining the location of the abnormality. Thus, the system may be configured to include a warning for the one or more abnormalities in the display signal. Alternatively or additionally, the system may be configured to output the warning via an output device of the ophthalmic microscope system, such as a warning light or a speaker.

[0071] As shown in Fig. 5a and Fig. 5b, a warning can be provided on the screen to provide warning of possible complications due to abnormal tissue structure. A useful application of such a warning is in the detection of abnormal deformation of the shape or the detection of micro-tears of the posterior capsule, which indicates a high risk of posterior capsule rupture when performing hydrodissection, phacoemulsification and lens placement. Fig. 5a and Fig. 5b show a schematic diagram of an example of a graphical annotation highlighting a posterior capsule rupture. Fig. 5a shows two views of the eye, a camera view 501 and an OCT view 505, where an OCT scan line 502 is superimposed on the camera view 501. In the camera view, two lines 503; 504 are shown superimposed, which highlight the first and second edges of the abnormal tissue structure, in this case the posterior capsule, of the rupture. The corresponding rupture in the two different OCT scan positions is highlighted by lines 506; 507 in the OCT view 505. Optionally, an additional warning message 508 "POSITOR CAPSULE RUPTURE" may be displayed along with one or more warning pictograms 509.

[0072] Furthermore, the system can provide surgical confirmation and navigation guidance when a particular procedure is completed. For example, remaining holes in the retina can be detected through image recognition with iOCT and highlighted on the surgical display for easy tracking. Another example concerns corneal transplant procedures, where the proposed system can detect whether the graft is in the correct orientation and aligned correctly across the transplant site. Figures 6a to 6c show corresponding examples.

[0073] FIG. 6a shows an example of surgical confirmation and navigation guidance based on whether a particular procedure is completed. In this example, holes remaining in the retina can be detected through image recognition using iOCT and highlighted on the surgical display for easy tracking. FIG. 6a shows a schematic diagram of an example of a graphical annotation highlighting holes in the retina superimposed on a visual representation of the camera sensor data and a visual representation of the OCT sensor data. FIG. 6a shows two views of the eye, similar to FIGS. 4b-5b, namely a camera view 601 and an OCT view 607. An OCT scan line 602 is superimposed on the camera view 601. In FIG. 6a, a first hole and a second hole in the retina are highlighted by circles 603; 606a. A corresponding circle 606b highlights the second hole in the OCT view 607. Additionally, directional markers 604; 605 are shown for the two holes. For example, the directional markers may be used to guide the surgeon towards the location of the holes.

[0074] As can be seen in FIG. 6a, two or more sets of intraoperative sensor data are acquired from two or more imaging devices. In some cases, an abnormality may be detected in one set of intraoperative sensor data and an indicator may be superimposed on a visual representation of the other set of intraoperative sensor data. For example, in FIG. 6a, an abnormality is detected in the intraoperative OCT sensor data and superimposed on a visual representation of the intraoperative OCT sensor data and the intraoperative imaging sensor data of the microscope imaging sensor. In other words, the intraoperative sensor data may include first intraoperative sensor data (e.g., intraoperative OCT sensor data) from a first imaging device and second intraoperative sensor data (e.g., intraoperative imaging sensor data) from a second imaging device. The system may be configured to generate a display signal having a first visual representation of the first intraoperative sensor data and a second visual representation of the second intraoperative sensor data. For example, the camera view 601 of Fig. 6a may show a second visual representation of the second intraoperative sensor data, and the OCT view 607 of Fig. 6a may show a first visual representation of the first intraoperative sensor data. The system may be configured to superimpose visual indicators 603;604;605;606a of anomalies detected by the machine learning model based on the first intraoperative sensor data over corresponding locations of the second visual representation 601 of the second intraoperative sensor data in the display signal, in addition to the visual indicator 606b of anomalies superimposed over the visual representation 607 of the first intraoperative sensor data. Another example of this concept is shown in Fig. 6b.

[0075] FIG. 6b shows another example of surgical confirmation and navigation guidance based on whether a particular procedure has been completed. In this example, the system recognizes and prompts if the corneal graft is not oriented correctly or aligned correctly. FIG. 6b shows a schematic diagram of an example of graphical annotation highlighting anomalies in the corneal graft, which is superimposed on a visual representation of the camera sensor data and a visual representation of the OCT sensor data. FIG. 6b shows a camera view 611 and a first and a second OCT view 616; 617. Since two OCT devices are used, a first scan line 612 of the first OCT and a second scan line 614 of the second OCT are shown. In this example, triangles 613a; 615a are used to highlight a first anomaly and a second anomaly in the corneal graft superimposed on the camera view, while triangles 613b; 615b highlight the same anomaly in the first and second OCT views 616; 617. For example, corresponding visual indicators (ie, triangles) that are overlaid on different visual representations of the intraoperative sensor data may have the same shape, the same color, and / or the same line pattern.

[0076] FIG. 6c shows another example of surgical confirmation and navigation guidance based on whether a particular procedure is completed. In this example, the system recognizes and prompts when an incision is open and requires further hydration. FIG. 6c shows a schematic diagram of an example of a graphical annotation highlighting an open incision, which is superimposed on a visual representation of the camera sensor data and a visual representation of the OCT sensor data. FIG. 6c shows a camera view 621, a first OCT view 625, and a second OCT view 627. FIG. 6c further shows corresponding first and second OCT scan lines 622;623. In FIG. 6c, an ellipse 624a highlighting the open incision is superimposed on the camera view, and a corresponding ellipse 624b highlighting the open incision is superimposed on the OCT view. In addition, a warning message 626 "Hydration required, wound not closed" is displayed.

[0077] In some examples, instrument detection may be incorporated, for example, to track the relative distance between the instrument tip and tissue structures. For example, the shape profile of the instrument may be identified to track the relative distance between the instrument tip and tissue structures. This allows the surgeon to perform precise, visually guided manipulation tasks while tracking the depth and distance of his or her instrument relative to tissue features of interest.

[0078] Thus, the system may be configured to detect the presence of one or more surgical instruments in the intraoperative sensor data. For example, the system may be configured to detect the presence of one or more surgical instruments using an object detection algorithm, such as a visual object matching algorithm or a machine learning model that has been trained to detect one or more instruments in the intraoperative sensor data (e.g., using supervised learning-based training of the machine learning model). For example, the machine learning model used to process the intraoperative sensor data may be further trained to detect and locate one or more instruments in the intraoperative sensor data.

[0079] The system may further be configured to determine a distance between the detected surgical instrument(s) and the one or more anatomical features. For example, the intraoperative sensor data used to detect the one or more surgical instruments, such as the intraoperative sensor data shown in FIG. 7, may have a known scale relative to the one or more surgical instruments, or the size of the one or more surgical instruments may be used to determine the scale of the intraoperative sensor data. The system may be configured to determine a distance between the detected surgical instrument(s) and the one or more anatomical features based on the scale of the intraoperative sensor data and based on a distance (e.g., in pixels) between the detected surgical instrument(s) and the one or more anatomical features in the visual representation of the intraoperative sensor data. Alternatively, the distance may be associated with a distance (e.g., in pixels) between the detected surgical instrument(s) and the one or more anatomical features in the visual representation of each intraoperative sensor data. The system may be configured to generate a visual guidance overlay comprising a visual indicator representing a distance between the detected surgical instrument(s) and the one or more anatomical features. For example, to improve visibility of the distance between the detected surgical instrument(s), the visual indicator representing the distance between the detected surgical instrument(s) may include a numerical representation of the distance, or the visual indicator may increase the contrast of the visual representation and / or highlight an edge of the surgical instrument(s) and / or an anatomical edge(s). In some examples, the visual indicator representing the distance may include a proximity warning if the distance is below a threshold.

[0080] FIG. 7 illustrates an example of how instrument detection can be incorporated to track the relative distance between the instrument tip and tissue structures. FIG. 7 illustrates a schematic diagram of an example of a graphical annotation highlighting the relative distance between the instrument tip and an anatomical feature. FIG. 7 illustrates a camera view 701 and an OCT view 704. An instrument tip 705 is detected in the intraoperative OCT sensor data and highlighted in the visual representation of the visual sensor data. Circles 702-703 are used to highlight the instrument tip in the camera view and the OCT view. For example, FIG. 7 illustrates a circle 702a highlighting the tip of a first instrument in the camera view, a corresponding circle 702b highlighting the tip of the first instrument in the OCT view, a circle 703a highlighting the tip of a second instrument in the camera view, and a corresponding circle 703b highlighting the tip of the second instrument in the OCT view. Similar to the examples shown in Figures 6a to 6c, corresponding visual indicators, such as circles, that outline the tip of the same instrument may have the same color, the same shape and / or the same line pattern.

[0081] The display signal is generated for a display device of the ophthalmic microscope system. Correspondingly, the system may be configured to provide the display signal to the display device of the ophthalmic microscope system. For example, the surgical display may be in the form of a 3D head-up surgical monitor, a head-mounted or microscope-mounted digital viewer, or an image insertion into a microscope eyepiece. Thus, as shown in FIG. 1c, the display device may be one of a head-up display 130a (i.e., a display that the surgeon looks at while looking straight ahead, rather than down at the surgical site), a head-mounted display (not shown), such as virtual reality goggles or augmented reality glasses or mixed reality glasses, a display 130c attached to the microscope of the ophthalmic microscope system, and a microscope eyepiece display 130b of the ophthalmic microscope system.

[0082] The one or more interfaces 112 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, the one or more interfaces 112 may include an interface circuit configured to receive and / or transmit information. In an embodiment, the one or more processors 114 may be implemented using any means for processing, such as one or more processing units, one or more processing devices, a processor operable with software that is accordingly adapted, a computer or a programmable hardware component. In other words, the described functions of the one or more processors 114 may also be implemented in software, which is then executed in one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. In at least some embodiments, the one or more storage devices 116 may include at least one member of a group of computer-readable recording media, such as magnetic or optical recording media, e.g., a hard disk drive, flash memory, a solid-state disk (SSD), a floppy disk, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or network storage.

[0083] Further details and aspects of the system and ophthalmic microscope system are referred to in connection with the proposed concept or one or more examples described above or below (e.g., Figures 8-9). The system and ophthalmic microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0084] FIG. 8 shows a flowchart of an example of a corresponding (computer-implemented) method for an ophthalmic microscope system, such as the ophthalmic microscope system introduced in relation to FIGS. 1a to 7. For example, the method may be implemented by the system 110 introduced in relation to FIGS. 1a to 7. The method includes acquiring 810 intraoperative sensor data of an eye from at least one imaging device of the ophthalmic microscope system. The method includes processing 820 the intraoperative sensor data with a machine learning model. The machine learning model is trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data. The method includes generating 830 a display signal based on the information regarding the one or more anatomical features of the eye. The display signal includes a visual guidance overlay to guide a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye.

[0085] Optionally, the method may include one or more further features, for example one or more features introduced by the system introduced in relation to Figures 1a to 7 or a surgical microscope system.

[0086] Further details and aspects of the method are mentioned in relation to the proposed concept or one or more examples above or below (e.g., Figures 1a to 7, 9). The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above or below.

[0087] Some embodiments relate to a microscope including a system as described in relation to one or more of the figures of Figures 1 to 8. Alternatively, the microscope may be part of a system as described in relation to one or more of the figures of Figures 1 to 8 or may be connected to a system as described in relation to one or more of the figures of Figures 1 to 8. Figure 9 shows a schematic diagram of a system 900 configured to perform the methods described herein. The system 900 includes a microscope 910 and a computer system 920. The microscope 910 is configured to capture images and is connected to the computer system 920. The computer system 920 is configured to perform at least some of the methods described herein. The computer system 920 may be configured to execute machine learning algorithms. The computer system 920 and the microscope 910 may be separate entities, but may be integrated in one common housing. The computer system 920 may be part of a central processing system of the microscope 910 and / or the computer system 920 may be part of a subordinate part of the microscope 910, such as a sensor, actor, camera or lighting unit of the microscope 910.

[0088] The computer system 920 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 920 may include any circuit or combination of circuits. In one embodiment, the computer system 920 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, for example, of a microscope or a microscope component (e.g., a camera). Other types of circuits that may be included in computer system 920 may be custom circuits, application specific integrated circuits (ASICs), such as one or more circuits (such as communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 920 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives handling removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), and the like.Computer system 920 may also include a display device, one or more speakers and a keyboard and / or controller which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that enables a user of the system to input information to and receive information from computer system 920.

[0089] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the crucial steps may be performed by such an apparatus.

[0090] According to certain implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be performed by a non-transitory recording medium, such as a digital recording medium, for example a floppy disk, a hard disk drive (HDD), a solid state disk (SSD), a DVD, a Blu-ray, a CD, a ROM, a PROM and an EPROM, an EEPROM or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Thus, the digital recording medium can be computer readable.

[0091] Some embodiments of the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform any of the methods described herein.

[0092] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which is operable to perform any of the methods when the computer program product is run on a computer, the program code may for example be stored on a machine readable carrier.

[0093] Another embodiment comprises the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0094] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0095] Therefore, another embodiment of the present invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, including a processor and a recording medium.

[0096] A further embodiment of the invention is therefore also a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0097] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0098] Another embodiment comprises a computer having the computer program installed thereon for performing any of the methods described herein.

[0099] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server to transfer the computer program to the receiver.

[0100] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

[0101] The embodiments may be based on the use of machine learning models or algorithms. With reference to Figures 1a to 7, two learning approaches have been discussed: supervised learning and semi-supervised learning.

[0102] In addition to supervised or semi-supervised learning, unsupervised learning may be used to train machine learning models. In unsupervised learning, input data may be (only) provided, and unsupervised learning algorithms may be used to find structure in the input data (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data containing multiple input values ​​into multiple subsets (clusters), such that input values ​​in the same cluster are similar according to one or more (predefined) similarity criteria, but are not similar to input values ​​contained in another cluster.

[0103] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as "software agents") are trained to take actions in their surroundings. Based on the actions taken, rewards are calculated. Reinforcement learning is based on training one or more software agents to select actions that result in an increasing cumulative reward (as manifested by an increasing reward), resulting in the software agent becoming better at a given task.

[0104] Furthermore, some techniques may be applied to parts of the machine learning algorithm. For example, feature representation learning may be used. In other words, the machine learning model may be trained at least in part with feature representation learning, and / or the machine learning algorithm may include a feature representation learning component. A feature representation learning algorithm, which may be referred to as a representation learning algorithm, may not only preserve information in its input, but may also transform the information to make it useful, often as a pre-processing step before performing classification or prediction. Feature representation learning may be based on, for example, principal component analysis or cluster analysis.

[0105] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide identification of input values ​​that raise suspicion by differing significantly from the majority of the input or training data. In other words, a machine learning model may be trained at least in part with anomaly detection and / or a machine learning algorithm may include an anomaly detection component.

[0106] In some examples, the machine learning algorithm may use a decision tree as a predictive model. In other words, the machine learning model may be based on a decision tree. In a decision tree, an observation about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to this item may be represented by a leaf of the decision tree. The decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0107] Association rules are another technique that may be used in machine learning algorithms. In other words, a machine learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. A machine learning algorithm may identify and / or utilize one or more association rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply the knowledge.

[0108] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a set of data structures and / or rules that represent learned knowledge (e.g., based on training performed by a machine learning algorithm). In embodiments, the use of machine learning algorithm may refer to the use of an underlying machine learning model (or underlying machine learning models). The use of machine learning model may refer to the machine learning model and / or the set of data structures / rules that are the machine learning model being trained by a machine learning algorithm.

[0109] For example, the machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or the brain. An ANN contains a number of interconnected nodes and a number of junctions, so-called edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may convey information from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g. the sum of its inputs). The inputs of a node may be used in a function based on the "weights" of the edges or nodes that provide the inputs. The weights of the nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0110] Alternatively, the machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that may be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing input with multiple training input values ​​that belong to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine learning model may be based on a genetic algorithm, which is a heuristic method that mimics search algorithms and the process of natural selection.

[0111] As used in this specification, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0112] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus. [Explanation of symbols]

[0113] 100 Ophthalmic Microscope System 105 Base Unit 110 System 112 Interface 114 processors 116 Storage Devices 120 OCT equipment 130 Display device 130a Head-up display 130b Eyepiece Display 130c Microscope-mounted display 140 Microscope 142 Optical Imaging Sensor 150 Endoscope 160 eyes 170 Arm 201-218 Text annotation of anatomical features 301 Anterior capsule 302 IOL 303 Posterior capsule 311 Lines highlighting the edge of the IOL 312 Line highlighting the posterior capsule 321 Camera Views 322 OCT scan lines 323 OCT Views 401~413 Eye Layer 421 Camera Views 422 OCT scan lines 423 OCT Views 424~427 Lines highlighting the layers of the eye 501 Camera Views 502 OCT scan lines 503,504 Lines highlighting the edges of abnormal tissue structures 505 OCT Views 506,507 Lines highlighting abnormal tissue structures in OCT 508 Warning message "Posterior capsule rupture" 509 Warning pictogram 601 Camera Views 602 OCT scan lines 603,606a Circles highlighting holes in the retina in the camera view 604,605 ​​Directional markers 606b Circle highlighting hole in retina in OCT view 607 OCT Views 611 Camera Views 612,614 OCT scan lines 613a;615a Triangles highlighting corneal graft abnormalities in the camera view 613b;615b Triangle highlighting corneal graft abnormalities in OCT view 616,617 OCT views 621 Camera Views 622,623 OCT scan lines 624a Ellipse highlighting open incision in camera view 624b Ellipse highlighting open incision in OCT view 625,627 OCT views 626 Warning message "Hydration required, wound not closed" 701 Camera Views 702a, 703a Circles highlighting the tip of the instrument in the camera view 702b,703b Circle highlighting the tip of the instrument in the OCT view 704 OCT views 705 Instrument Tip 810 Acquiring intraoperative sensor data 820 Processing intraoperative sensor data using machine learning models 830 Generating a display signal with visual guidance overlay 900 System 910 Microscope 920 Computer Systems

Claims

1. A system (110; 920) for an ophthalmic microscope system (100; 900), comprising: The system includes one or more processors (114) and one or more storage devices (116); the system is configured to acquire ophthalmic intra-operative sensor data from at least one imaging device (120; 142; 150) of the ophthalmic microscope system; the system is configured to process the intraoperative sensor data with a machine learning model, the machine learning model being trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data; the system is configured to generate a display signal for a display device (130a; 130b; 130c) of the ophthalmic microscope system based on the information regarding the one or more anatomical features of the eye, the display signal including a visual guidance overlay for guiding a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye. System (110; 920).

2. the visual guidance overlay including annotations of the one or more anatomical features of the eye suitable for guiding a user of the ophthalmic microscopy system during a surgical procedure. The system of claim 1 .

3. the system is configured to acquire the intraoperative sensor data as a continuously updated stream of intraoperative sensor data; and updating the visual guidance overlay based on the continuously updated stream of intraoperative sensor data. The system of claim 1 .

4. the system is configured to overlay the visual guidance overlay over a visual representation of the intraoperative sensor data in the display signal. The system of claim 1 .

5. the system is configured to generate the visual guidance overlay comprising one or more visual indicators of a plurality of visual indicators including one or more of: a text annotation of at least a subset of the one or more anatomical features, an overlay for highlighting one or more surfaces of the one or more anatomical features, an overlay for highlighting one or more edges of the one or more anatomical features, one or more directional indicators, and one or more indicators related to one or more anomalies related to the one or more anatomical features. The system of claim 1 .

6. the system is configured to generate the visual guidance overlay based on a selection of a subset of the plurality of visual indicators; the selection is based on input from a user of the ophthalmic microscope system; or the system is configured to determine the selection based on a progress of an ophthalmic surgical procedure performed with the aid of the ophthalmic microscope system. The system of claim 5.

7. the intraoperative sensor data includes intraoperative optical coherence tomography sensor data; the machine learning model is trained to output information regarding one or more layers of the eye based on the intraoperative optical coherence tomography sensor data; the system is configured to generate the visual guidance overlay comprising a visual indicator that highlights or annotates at least a subset of the one or more layers of the eye. The system of claim 1 .

8. the machine learning model is trained to output information regarding a classification of the one or more anatomical features in the intraoperative sensor data; the system is configured to generate the visual guidance overlay comprising a visual indicator associated with the classification of the one or more anatomical features. The system of claim 1 .

9. the machine learning model is trained to output information regarding one or more abnormalities related to the one or more anatomical features of the eye; the system is configured to generate the visual guidance overlay comprising a visual indicator related to the one or more anomalies. The system of claim 1 .

10. the intraoperative sensor data includes first intraoperative sensor data from a first imaging modality and second intraoperative sensor data from a second imaging modality; the system is configured to generate the display signal comprising a first visual representation of the first intra-operative sensor data and a second visual representation of the second intra-operative sensor data; the system is configured to overlay a visual indicator of an anomaly detected by the machine learning model based on the first intraoperative sensor data over a corresponding location of the second visual representation of the second intraoperative sensor data in the display signal.

10. The system of claim 9.

11. the system is configured to detect a presence of one or more surgical instruments in the intraoperative sensor data; the system is configured to determine a distance between the detected one or more surgical instruments and the one or more anatomical features; the system is configured to generate the visual guidance overlay comprising a visual indicator representing the distance between the detected one or more surgical instruments and the one or more anatomical features. The system of claim 1 .

12. The intraoperative sensor data includes one or more of intraoperative optical coherence tomography sensor data of an intraoperative optical coherence tomography device (120) of the ophthalmic microscope system, intraoperative imaging sensor data of an imaging sensor (142) of a microscope (140; 910) of the ophthalmic microscope system, and intraoperative endoscopic sensor data of an endoscope (150) of the ophthalmic microscope system. The system of claim 1 .

13. An ophthalmic microscope system (100; 900), comprising: comprising at least one imaging device (120; 142; 150), a display device (130a; 130b; 130c) and the system (110; 920) according to any one of claims 1 to 12, Ophthalmic microscope system (100; 900).

14. A method for an ophthalmic microscope system (100; 900), said method comprising the steps of: acquiring (810) intraoperative ophthalmic sensor data from at least one imaging device of the ophthalmic microscope system; processing (820) the intraoperative sensor data with a machine learning model, the machine learning model being trained to output information regarding one or more anatomical features of the eye based on the intraoperative sensor data; generating (830) a display signal based on information regarding the one or more anatomical features of the eye, the display signal including a visual guidance overlay for guiding a user of the ophthalmic microscope system regarding the one or more anatomical features of the eye; The method includes:

15. A computer program comprising: The computer program has a program code for performing the method of claim 14 when the computer program is run on a processor. Computer program.

Images