Surgical microscope system, system and method for microscope of surgical microscope system, and computer program

Abstract

To provide a surgical microscope system, a system and a method for the microscope of the surgical microscope system, and a computer program.SOLUTION: The system is configured to acquire imaging sensor data from at least one optical imaging sensor 122 of the microscope, to decide information on the area of interest of a user on the basis of the input of the user of the surgical microscope system, to decide an anatomical interest feature in the area of interest, and to detect the position of the anatomical interest feature in the imaging sensor data. The system is configured to trigger the autofocus function of the microscope, to focus on the position of the anatomical interest feature.SELECTED DRAWING: Figure 1a

Description

Technical Field

[0001] The examples relate to a surgical microscope system and a system, method, and computer program for a microscope of a surgical microscope system.

Background Art

[0002] Autofocus is a function commonly used in current optical cameras. For example, surgical microscopes often provide an autofocus function. However, in surgical microscopes, the normal autofocus function generally uses a pre-defined image area as the focus reference. In many cases, the area of interest to the surgeon often does not exist within such autofocus areas. Therefore, there is a possibility that the camera may focus on an area other than what the surgeon intended.

[0003] Smartphones and mirrorless interchangeable lens cameras provide an extended autofocus function that attempts to search for and focus on a human face. Even when the face moves within the image, autofocus can continue to search for the face and adjust the focus accordingly.

[0004] In surgical microscopes and other medical optical imaging devices, the area of interest is not as universal as a human face. Rather, the area of interest can be any anatomical or morphological structure, such as blood vessels or surgical cavities, respectively. Therefore, it is impossible to pre-define a spatial pattern as in the case of a face and search for it within the image to set an autofocus target.

[0005] There may be a desire to improve the autofocus function of surgical microscopes.

Summary of the Invention

Means for Solving the Problems

[0006] This request is addressed by the subject matter of the independent claim.

[0007] The proposed concept is based on the observation that during surgery, surgeons focus on a small area (i.e., the actual surgical site) and have little interest in other parts of the surrounding area. This area is defined or described by one or more anatomical features, and the surgeon performs surgery either on or between these anatomical features. In the proposed concept, the surgeon can trigger the proposed function by pointing to an area in the image, causing the image focus to automatically adjust to this area. The surgeon can provide a command, which identifies an anatomical feature within the area to be followed. For example, the system may be commanded to follow a user-specified blood vessel in a given area, or the command may be vague, and the proposed function may make some assumptions about which features are implied in the indicated area, such as selecting a surgical cavity if the area indicated by the surgeon contains one. Thus, the proposed concept enables the surgeon to intelligently and meaningfully select an autofocus area, allowing the autofocus to follow specific image features rather than a spatially fixed area of ​​the image.

[0008] Various examples of this disclosure relate to a system for a microscope in a surgical microscope system. The system comprises one or more processors and one or more storage devices. The system is configured to acquire imaging sensor data from at least one optical imaging sensor of the microscope. The system is configured to determine information about the user's area of ​​interest based on user input of the surgical microscope system. The system is configured to determine anatomical features of interest within the area of ​​interest. The system is configured to detect the location of anatomical features of interest in the imaging sensor data. The system is configured to trigger the autofocus function of the microscope to focus on the location of the anatomical features of interest. By determining the area of ​​interest based on user input, sufficient information can be provided to the detection algorithm to determine potential features of interest, which can then be tracked and used as targets for the autofocus function, so that the potential features of interest remain in focus throughout the surgical procedure.

[0009] Generally, the system can be configured to track the location of anatomical features of interest across multiple frames of imaging sensor data. By tracking features across multiple frames, the autofocus function can be triggered to adjust the focus based on the current location of the anatomical features of interest.

[0010] For example, the system may be configured to trigger the autofocus function when the location of an anatomical feature of interest is shifted relative to the field of view of the imaging sensor data by at least a predetermined time interval. In other words, if the location of an anatomical feature of interest is shifted, not merely temporarily (for example, if the field of view is altered by a surgeon, or if the surgeon moves the feature of interest), the autofocus function can be used to adjust the focus to the current location of the anatomical feature of interest.

[0011] While surgical microscope systems generally feature powerful processors for image processing, the efficiency of the proposed concept can be improved by limiting the detection of anatomical feature locations to only a subset of frames, for example, every nth frame. For instance, the system can be configured to detect the location of anatomical features of interest in many or every other frame of the imaging sensor data.

[0012] In various examples, anatomical features of interest are detected using software algorithms that may be based on machine learning models. By limiting the imaging sensor data to the area of ​​interest, the effort required by the software algorithm can be reduced and accuracy can be improved. For example, the system may be configured to locate the area of ​​interest in the imaging sensor data. The system may also be configured to determine anatomical features of interest within a portion of the imaging sensor data representing the area of ​​interest.

[0013] Different anatomical features within an area of ​​interest may be identified by a (machine learning-based) algorithm, and each anatomical feature of interest can be selected from among the identified anatomical features. For example, the system can be configured to perform image segmentation on at least a portion of the imaging sensor data representing the area of ​​interest to determine at least one feature present within that portion of the imaging sensor data representing the area of ​​interest. In addition to or instead of this, the system may be configured to perform object detection on at least a portion of the imaging sensor data representing the area of ​​interest to identify at least one feature present within that portion of the imaging sensor data representing the area of ​​interest. The system can be configured to determine anatomical features of interest based on at least one feature present within that portion of the imaging sensor data representing the area of ​​interest. For example, an existing image segmentation machine learning model or object detection machine learning model can be used to determine anatomical features present in the area of ​​interest, and from among them, anatomical features of interest can be selected.

[0014] For example, during neurosurgery, there will be various features that are potentially of interest to the surgeon. For instance, the system can be configured to perform object detection to identify at least one of blood vessels, vascular bifurcations, hemorrhages, and tumors within at least a portion of the imaging sensor data representing the area of ​​interest.

[0015] In addition, non-anatomical features can also be detected and used to define non-anatomical features of interest, and in some cases, can be used in place of anatomical features of interest. For example, the system may be configured to perform object detection to identify at least one of clips and stitches as a non-anatomical feature of interest. The system may be configured to detect the location of non-anatomical features of interest in the imaging sensor data. The system may be configured to perform the autofocus function of the microscope based further on the location of the non-anatomical features of interest. In some cases, non-anatomical features such as clips or stitches are correctly positioned on a portion of the surgical site that the surgeon intends to operate on, making them useful targets for the autofocus function.

[0016] In some cases, an anatomical feature of interest may be part of a larger anatomical feature. For example, an anatomical feature may be a branch within a blood vessel, which is part of a larger blood vessel. Thus, this larger anatomical feature may be subdivided into smaller anatomical features, and the extent of a given anatomical feature within the larger anatomical feature is defined by the adjacent anatomical features. For example, the system can be configured to determine the extent of an anatomical feature of interest based on the extent of one or more features located adjacent to the feature on which the anatomical feature of interest is based.

[0017] In some cases, an area of ​​interest cannot be defined by a single anatomical feature of interest, but may relate to an area between features or a group of features. For example, if the area of ​​interest indicated by the user relates to two or more features or an area between two or more features, the system may be configured to determine an anatomical feature of interest based on two or more features, and to determine the location of the anatomical feature of interest based on the locations of two or more features.

[0018] In this case, the features that define the boundary of the area of ​​interest may be located outside the area of ​​interest. Therefore, the system can be configured to select two or more features from those located within and outside the area of ​​interest when the area of ​​interest indicated by the user relates to an area between two or more features.

[0019] Various options exist for the user to specify the area of ​​interest. For example, the system can be configured to determine the area of ​​interest by detecting a pointer manipulated by the user within the imaging sensor data. Alternatively, or in addition to this, the system may be configured to determine the area of ​​interest based on user input signals acquired through the user interface of the surgical microscope system. The pointer or user interface allows for very precise specification of the area of ​​interest.

[0020] Alternatively, or in addition to this, the system is configured to determine the area of ​​interest based on a voice description of anatomical features acquired via the voice command system of the surgical microscope system. In this way, the area of ​​interest can be defined without the surgeon having to move instruments away from the surgical site.

[0021] The options listed above allow surgeons to explicitly define their area of ​​interest. However, in some cases, the area of ​​interest may be implicitly defined, for example, by inferring it from the surgeon's actions. For example, the system can be configured to determine the area of ​​interest by determining a portion of the surgical site being operated on by the user in the imaging sensor data. Alternatively, or in addition to this, the system may be configured to determine the area of ​​interest using an eye-tracking mechanism. Alternatively, or in addition to this, the system may be configured to determine the area of ​​interest from a predetermined image area (preferably the center of the field of view) after the user has completed field of view alignment. In these cases, the effort required of the surgeon is reduced because the user does not need to explicitly define the area of ​​interest.

[0022] In some examples, some image processing may be performed to facilitate the definition of the area of ​​interest. For example, the system may be configured to perform image segmentation and / or object detection to determine multiple features in the imaging sensor data. The system may be configured to determine a visual representation of the multiple features and provide a display signal containing this visual representation to the display device of the surgical microscope system. The system may be configured to take user input in response to the visual representation of the multiple features. By providing a visual representation of the multiple features, the user can select from the detected features to define the area of ​​interest.

[0023] Surgical microscope systems are often adapted to detect fluorescence emitted by fluorophores injected into blood vessels or tissues. These fluorescence emissions can be used to clearly identify anatomical features of interest. Additional imaging information, such as information obtained from multispectral imaging, can also be used to clearly identify anatomical features of interest. For example, imaging sensor data may include a first component having color imaging data and a second component having at least one of hyperspectral imaging data, multispectral imaging data, or fluorescence imaging data. The system can be configured to determine and / or detect the location of anatomical features of interest based on at least the second component.

[0024] The determined anatomical (or non-anatomical) features of interest can be, for example, highlighted and presented to the user so that the user can be assured that the "correct" features are used for the autofocus function. For example, the system can be configured to generate a digital view based on imaging sensor data. The system can be configured to highlight areas of interest and / or anatomical (or non-anatomical) features of interest within the digital view. The system can be configured to provide a display signal, including the digital view, to the display device of a surgical microscope system.

[0025] Various examples relate to surgical microscope systems comprising a microscope having an optical imaging sensor and the system presented above.

[0026] Various examples relate to a method for a microscope of a surgical microscope system corresponding to the above. The method includes obtaining imaging sensor data from at least one optical imaging sensor of the microscope. The method includes determining information regarding an area of interest of a user based on an input of the user of the surgical microscope system. The method includes determining an anatomical feature of interest within the area of interest. The method includes detecting the position of the anatomical feature of interest in the imaging sensor data. The method includes triggering an autofocus function of the microscope to focus on the position of the anatomical feature of interest.

[0027] Various examples relate to a computer program having program code for performing the above method when executed on a processor.

[0028] Some examples of the apparatus and / or method will be described below by way of example only, with reference to the accompanying drawings.

Brief Description of the Drawings

[0029] [Figure 1a] FIG. is a block diagram of an example of a system for a microscope of a surgical microscope system. [Figure 1b] FIG. is a schematic diagram of an example of a surgical microscope system. [Figure 1c] FIG. is a diagram of an anatomical feature of interest related to the blood vessels of the brain. [Figure 1d] FIG. is a diagram of an anatomical feature of interest related to the blood vessels of the brain. —— [Figure 1e] FIG. is a diagram of an anatomical feature of interest related to the blood vessels of the brain. [Figure 2] FIG. shows a flowchart of an example of a method for a microscope of a surgical microscope system. [Figure 3] FIG. is a schematic diagram of an example of a digital view on a surgical site of the brain. [Figure 4] FIG. is a schematic diagram of a system including a microscope and a computer system.

Modes for Carrying Out the Invention

[0030] Next, various examples will be explained in more detail with reference to the attached drawings, which illustrate several examples. In the drawings, the thickness of lines, layers, and / or areas may be exaggerated for clarity.

[0031] Figure 1a shows a block diagram of an example of system 110 for a microscope 120 of surgical microscope system 100 (shown in more detail in Figure 1b). Surgical microscope system 100 comprises a microscope 120, which is a digital microscope, i.e., an optical imaging sensor 122 coupled to system 110. Generally, microscopes, such as microscope 120, are optical instruments suitable for examining objects that are too small to be examined with the human eye (alone). For example, a microscope can provide optical magnification of a sample. In modern microscopes, optical magnification is often provided by a camera or imaging sensor, such as the optical imaging sensor 122 of microscope 120. Microscope 120 may further comprise one or more optical magnification components, such as objective lenses (i.e., lenses), used to magnify the view on the sample.

[0032] In addition to the optical components that are part of the microscope 120, the surgical microscope system 100 further comprises a computer system, system 110. The system comprises one or more processors 114 and one or more storage devices 116. Optionally, the system may further comprise one or more interfaces 112. As shown in Figure 1a, one or more processors 114 are coupled to one or more storage devices 116 and one or more interfaces (as optional). Generally, the functionality of the system is provided by one or more processors 114, in conjunction with one or more interfaces 112 (for example, to exchange information with the microscope's optical imaging sensor 122 or the surgical microscope system's display device 130) and / or one or more storage devices 116 (for storing and / or retrieving information). System 110 is configured to acquire imaging sensor data from at least one of the microscope's optical imaging sensors 122. System 110 is configured to determine information about the user's area of ​​interest based on user input from the surgical microscope system. System 110 is configured to determine anatomical features of interest within the area of ​​interest. System 110 is configured to detect the location of anatomical features of interest in the imaging sensor data. System 110 is configured to trigger the microscope's autofocus function to focus on the location of the anatomical features of interest. Clearly, System 110 is a system for processing imaging sensor data within the surgical microscope system 100, and / or for controlling the microscope 120 and / or other components of the surgical microscope system 100.

[0033] Generally, a microscope system such as surgical microscope system 100 is a system that includes a microscope 120, additional components that operate together with the microscope, such as system 110 (a computer system adapted to control the surgical microscope system and, for example, process imaging sensor data from the microscope), additional sensors, displays, etc.

[0034] Various different types of microscopes exist. When a microscope is used in the medical or biological fields, the object observed through the microscope may be a sample of organic tissue, for example, placed in a petri dish or present in a part of a patient's body. In this case, microscope 120 is a microscope in a surgical microscope system, i.e., a microscope used during surgical procedures such as oncological surgical procedures or tumor surgery. Therefore, the object observed through the microscope and shown as a composite image may be a sample of organic tissue from a patient, and in particular may be the surgical site where the surgeon operates during the surgical procedure. In the following, the object being imaged, i.e., the surgical site, will be assumed to be the surgical site of the brain during neurosurgery. However, the proposed concept is also applicable to other types of surgery such as cardiac surgery or ophthalmology.

[0035] Figure 1b shows a schematic diagram of an example of a surgical microscope system 100 comprising a system 110 and a microscope 120 (having an optical imaging sensor 122). The surgical microscope system 100 shown in Figure 1b includes a base unit 105 (including the system 110) having a (rotating) stand, an eyepiece display 130a positioned on the microscope 120, an auxiliary display 130b positioned on the base unit 105, and a plurality of optional means components such as a (robot or manual) arm 140 that holds the microscope 120 in place and is coupled to the base unit 105 and the microscope 120. Generally, these optional and non-optional means components can be coupled to the system 110, which can be configured to control and / or interact with each component.

[0036] The proposed concept is based on two main components: obtaining user input to define an area of ​​interest, and using the user-specified area of ​​interest to determine anatomical features of interest, which are then tracked so that autofocus remains applicable to the anatomical features of interest even if the field of view changes or the features of interest move.

[0037] The proposed concept begins with determining an area of ​​interest based on user input. Various options exist for performing this task. For example, a user can use a pointer, such as a finger, surgical instrument, or a dedicated pointer device (e.g., having active visual markers such as light, or passive visual markers such as 2D codes), to point to or surround one or more anatomical features and indicate an area of ​​interest (the user indicates an area of ​​interest surrounding a point or circle). Pointing and / or surrounding can be detected using a machine learning model trained to determine pointing and / or surrounding actions in imaging sensor data, or using a deterministic algorithm configured to determine pointing or surrounding actions based on imaging sensor data (e.g., based on active or passive visual markers, or by performing object detection on the fingertip). The system can be configured to detect a pointer manipulated by the user within the imaging sensor data and determine an area of ​​interest. For example, the system can be configured to determine an area of ​​interest around a (single) point if the user points to a (single) location within the field of view. For example, if a user points to a feature (e.g., an anatomical or non-anatomical feature), the area of ​​interest can surround that feature. The system can be configured to determine the area of ​​interest to coincide with the area surrounded by the user when the user surrounds an area within the field of view. For example, if a user surrounds a feature or a set of adjacent features, the area of ​​interest can surround that feature or that set of features. If a user points to multiple locations, the system can be configured to determine whether those points intersect the same anatomical feature and then set an area of ​​interest around that part (each part) of the anatomical feature. For example, if a user points multiple times along a long, narrow blood vessel, the area of ​​interest can surround each part of the blood vessel pointed to by the surgeon (as shown, for example, in Figure 1e).

[0038] A similar approach can be taken when a user defines an area of ​​interest via a user interface, such as a touch panel, of a surgical microscope system. For example, the system may be configured to determine the area of ​​interest based on user input signals acquired through the user interface of the surgical microscope system. In this case as well, the user can define the area of ​​interest using one or more points or circles.

[0039] To support the selection process, imaging sensor data can be analyzed to determine and distinguish features in the imaging sensor data, such as anatomical features (e.g., blood vessels, tumors, branches, tissue segments, etc.) or non-anatomical features (e.g., clips or stitches). For this purpose, one or both of the following machine learning-based techniques, namely image segmentation and object detection, may be used. The system may be configured to perform image segmentation and / or object detection to determine multiple features in the imaging sensor data. In object detection, the locations of one or more predefined objects (i.e., objects on which each machine learning model is trained) in the imaging sensor data, along with the object classification (if the machine learning model is trained to detect multiple different types of objects), are output by the machine learning model. Generally, the locations of one or more predefined objects are given as a set of locations that form a bounding box, i.e., a rectangular shape surrounding each detected object. In image segmentation, the locations of features (i.e., segments of the imaging sensor data that have similar attributes, e.g., belonging to the same object) are output by the machine learning model. Generally, the location of a feature is provided as a pixel mask. That is, the location of the pixels belonging to a feature is output for each feature.

[0040] For both object detection and image segmentation, machine learning models trained to perform their respective tasks are used. For example, to train a machine learning model trained to perform object detection, multiple or samples of imaging sensor data may be provided as training input samples, a corresponding list of bounding box coordinates may be provided as the desired output of the training, and a supervised learning-based training algorithm may be used to perform the training using the multiple training input samples and the corresponding desired output. For example, to train a machine learning model trained to perform image segmentation, multiple or samples of imaging sensor data may be provided as training input samples, a corresponding pixel mask may be provided as the desired output of the training, and a supervised learning-based training algorithm may be used to perform the training using the multiple training input samples and the corresponding desired output. In some examples, both object detection and image segmentation may be performed using the same machine learning model. In this case, the two types of desired outputs described above may be used in parallel during training, and the machine learning model may be trained to output both bounding boxes and pixel masks. Machine learning models can be used not only to support the selection of areas of interest, but also to determine anatomical features of interest, as will be shown in later stages.

[0041] The system can be configured to determine the visual representation of multiple features. For example, the system may be configured to determine the visual representation using overlays that highlight and / or depict multiple features, for example, using contours around features (based on, for example, a pixel mask or bounding box) and / or using color overlays that are overlaid over each feature (based on, for example, a pixel mask or bounding box). For example, if object detection is being performed, a description of a feature may be included next to the feature. In addition, a grid may be displayed, which can be useful for a voice-based user interface. For example, the visual representation may be included in a digital view of the surgical site being observed through the microscope, or may be overlaid on the digital view, for example. The system can be configured to provide a display signal containing the visual representation (or the digital view having the visual representation) to a display device 130 of the surgical microscope system (e.g., an eyepiece display 130a or an auxiliary display 130b). The user can use the visual representation to select an area of ​​interest, for example, by pointing to or surrounding one or more features. Therefore, the system may be configured to acquire user input in response to visual representations of multiple features.

[0042] This visual representation can be particularly useful when a voice-based user interface is used to select an area of ​​interest. For example, a system may be configured to determine an area of ​​interest based on a voice description of an anatomical feature obtained via a voice command system of a surgical microscope system. The system may be configured to process the voice description using a voice processing algorithm to associate the described anatomical feature with one of several features identified using object detection and / or image segmentation. For example, if the voice description includes a reference to a grid cell, the system may be configured to select an anatomical feature that adapts to the description shown in the grid cell and determine an area of ​​interest based on the selected anatomical feature. Alternatively, or in addition to this, a natural language description (e.g., "the largest branch on the lower right side" or "a portion of the blood vessel between the two branches shown on the right side of the figure") may be used, and the system may be configured to select an anatomical feature based on this natural language description and determine an area of ​​interest based on the selected anatomical feature.

[0043] In the above, the area of ​​interest is explicitly selected by the user. Alternatively, the selection of the area of ​​interest can be performed implicitly using a system that interprets the user's (e.g., surgeon's) actions to determine the area of ​​interest. For example, the system may be configured to determine the area of ​​interest by determining a portion of the surgical site being operated on by the user in the imaging sensor data. For example, the system may be configured to perform object detection to detect the position of one or more surgical instruments in the imaging sensor data, determine a portion of the surgical site being operated on by the user based on the position of one or more surgical instruments, and determine the area of ​​interest around the position of one or more surgical instruments (e.g., around the side of one or more surgical instruments facing the surgical site). In addition to or instead of this, the system may also be configured to determine the area of ​​interest by using an eye-tracking mechanism to track the user's gaze, for example through the eyepiece of a microscope, and determining the area of ​​interest which is the area where the gaze is focused for a predetermined time interval. Alternatively, or in addition to the above, the system may be configured to determine an area of ​​interest from a given image area (within the field of view, for example, a central area within the field of view that covers at least 10% and up to 50% of the field of view) after the user has finished aligning the field of view (i.e., after the user has changed the position of the microscope or adjusted the magnification of the microscope).

[0044] Once the area of ​​interest is determined, anatomical features of interest within that area are selected. Specifically, the system may be configured to locate the area of ​​interest (and therefore the current field of view) in the imaging sensor data and to determine anatomical features of interest within the portion of the imaging sensor data representing the area of ​​interest, using image processing techniques such as object detection or image segmentation as described above. In other words, image processing techniques are used to automatically select anatomical features of interest without additional user input, for example, in addition to the selection of the area of ​​interest.

[0045] As outlined above, at least one of the two techniques, "object detection" and "image segmentation," can be used to analyze imaging sensor data, determine features (e.g., anatomical or non-anatomical features) in the imaging sensor data, and select anatomical features from among the determined features. For this purpose, past determinations of features across the imaging sensor data (as an optional means) may be used, or object detection or image segmentation may be performed on a subset of the imaging sensor data corresponding to an area of ​​interest (and several potential surrounding areas, as will become apparent below). Thus, the system can be configured to perform image segmentation on at least a portion of the imaging sensor data representing an area of ​​interest to determine at least one feature present within that portion of the imaging sensor data representing the area of ​​interest, and to determine anatomical features of interest based on at least one (e.g., intersecting) feature present within that portion of the imaging sensor data representing the area of ​​interest (e.g., a pixel mask). Alternatively, or in addition to the above, the system may be configured to perform object detection on at least a portion of the image sensor data representing an area of ​​interest to identify at least one feature (e.g., a bounding box) that exists (e.g., intersects) within the portion of the image sensor data representing the area of ​​interest, and to determine an anatomical feature of interest based on the identified at least one feature present within the portion of the image sensor data representing the area of ​​interest. In other words, if the object detection output of an image segmentation machine learning model indicates that one or more features intersect with an area of ​​interest in the image sensor data, then such feature (or features) can be used as a feature of interest. If multiple points are provided by the user, features or combinations of features that intersect with all or most of those points may be selected as the anatomical feature of interest.

[0046] In some examples, the features used to select anatomical features of interest may be limited to specific groups of features. For example, a system may be configured to perform object detection to identify at least one of the following within at least a portion of imaging sensor data representing an area of ​​interest: blood vessels, vascular bifurcations, hemorrhages, tumors, surgical cavities, tissue of a predetermined color, discolored tissue, raised areas, and depressed areas. Thus, a machine learning model trained to perform object detection may be trained to detect at least one of the following in imaging sensor data: blood vessels, vascular bifurcations, hemorrhages, tumors, surgical cavities, tissue of a predetermined color, discolored tissue, raised areas, and depressed areas. Similarly, a machine learning model trained to perform image segmentation may be trained to perform image segmentation on at least one of the following in imaging sensor data: blood vessels, vascular bifurcations, hemorrhages, tumors, surgical cavities, tissue of a predetermined color, discolored tissue, raised areas, and depressed areas. Therefore, anatomical features of interest may be selected based on the output of one or more machine learning models trained to output information such as bounding boxes or pixel masks that represent specific features such as the aforementioned blood vessels, bifurcations, hemorrhages, tumors, surgical cavities, tissues of a given color, discolored tissues, raised areas, or depressed areas.

[0047] The following provides some examples of how the proposed method can be applied to imaging sensor data collected during neurosurgery. Figures 1c–1e show diagrams of anatomical features of interest relating to blood vessels in the brain. For example, the system may identify arteries, veins, and unique shape identifiers, morphological features such as branching, and color features for these. Figure 1c shows a blood vessel in the brain with multiple branches. In neurosurgery, such branching of blood vessels is often of particular importance. Therefore, as shown in Figure 1d, the system may be configured to use object detection to identify branching points 150–158 of a blood vessel (shown as bounding boxes around the branching points). If the area of ​​interest intersects with one of the bounding boxes, that branching point may be selected as an anatomical feature of interest. In some examples, the system may be configured to identify low-contrast or invisible information within the area of ​​interest. For example, if a surgeon points to a vein multiple times, the system can understand that it is around a vein and not a nearby artery, or if a surgeon clicks on fully oxygenated blood (microbleed) on the surface of a cavity, the system can identify a surgical cavity, taking into account the blood saturation and the shape of this element (microbleed). In addition, additional imaging information, such as imaging sensor data from hyperspectral or fluorescence imaging, may also be considered. For example, imaging sensor data may include a first component having color imaging data (e.g., from white light reflectance imaging) and a second component having at least one of hyperspectral imaging data, multispectral imaging data, and fluorescence imaging data. The second component may include information that is not present in or indistinguishable in the first component, for example, by mixing with color information from adjacent wavelength bands, or by excluding or eliminating fluorescence emission from the white light reflectance image (by excluding the respective fluorescence emission wavelength bands from the illumination used to perform white light reflectance imaging). This additional information can be used to distinguish or select anatomical features of interest.Therefore, the system can be configured to determine and / or detect the location of anatomical features of interest by distinguishing them from other visible features in the imaging sensor data, for example, using fluorescence emission or color information separated in the hyper or multispectral imaging sensor data based on at least a second component.

[0048] In some examples, the system can also consider adjacent structures, such as branches of the same vessel, that do not directly exist within a user-defined area; that is, a segment can be understood by the algorithm as a portion of a vessel between two characteristic branches or curves. For example, as shown in Figure 1e, a large curve 160 can be selected as an anatomical feature of interest (which is part of a larger vessel) by, for example, pointing multiple times along the path of the curve or by pointing to the center of the curve. Alternatively, a portion of a vessel 162 between branch points 164;166 may be selected. Thus, the system can be configured to determine the range of an anatomical feature of interest based on the range of one or more features located adjacent to the feature on which the anatomical feature of interest is based. For example, the system may be configured to limit the range of a feature of interest based on adjacent features, for example, by limiting the portion of a vessel 162 selected as an anatomical feature of interest based on the locations of two branch points 164;166 located adjacent to a portion of a vessel.

[0049] In some cases, the area between two characteristic features can be considered the area of ​​interest. If no characteristic features are present, the area of ​​interest, and therefore the anatomical feature of interest, can be designated as an area spatially oriented relative to other features, even if it is outside the area of ​​interest. For example, the system can be configured to determine the anatomical feature of interest based on two or more features and the location of the anatomical feature of interest based on the locations of two or more features, if the area of ​​interest indicated by the user relates to two or more features or an area between two or more features. Furthermore, the system can be configured to select two or more features from features located within the area of ​​interest and features located outside the area of ​​interest, if the area of ​​interest indicated by the user relates to an area between two or more features. For example, the area of ​​interest may be understood by an algorithm as the tissue between two vertically oriented blood vessels that are near but outside the area of ​​interest indicated by the user. Secondary optical imaging sensors with a wider field of view can also be used for this purpose, i.e., to select features outside the area of ​​interest. In some cases, the image analysis / processing techniques described above can be used to extract low-contrast or invisible features, or features that require quantification. For example, the system can be configured to perform image processing to measure parameters related to blood vessels / capillaries, such as density, branching, and curvature, and to use these parameters to segment and identify features of anatomical interest. For instance, a tissue area with high-density capillaries may be a feature of anatomical interest.

[0050] In some cases, the system may also consider foreign objects such as clips or stitches. For example, the system can be configured to perform object detection to identify at least one of clips and stitches as a non-anatomical feature of interest. Thus, a machine learning model trained to perform object detection and / or image segmentation can be trained on imaging sensor data exhibiting non-anatomical features (as training input samples) and corresponding bounding box coordinates and / or pixel masks as desired outputs. The system can be configured to detect the location of non-anatomical features of interest in the imaging sensor data and to perform the microscope's autofocus function based further on the location of the non-anatomical features of interest. Non-anatomical features of interest can be used in addition to or instead of anatomical features of interest, and for example, the system can support the use of both anatomical and non-anatomical features with respect to autofocus capabilities. For example, if the user points to an anatomical feature of interest or the area of ​​interest contains a (more prominent) anatomical feature, the autofocus function can be performed based on this anatomical feature; if the user points to a non-anatomical feature of interest or the area of ​​interest contains a (more prominent) non-anatomical feature, the autofocus function can be performed based on the non-anatomical feature. In other words, if the user selection is (clearly) related to non-anatomical features, the autofocus function may be performed based on non-anatomical features; otherwise, the autofocus function may be performed based on anatomical features.

[0051] Once an anatomical (or non-anatomical) feature of interest is determined, the autofocus function is directed towards that feature. Specifically, the system is configured to detect the location of the anatomical feature of interest in the imaging sensor data and trigger the microscope's autofocus function to focus on the location of the anatomical or non-anatomical feature of interest. In other words, the system may be configured to set the focus used by the autofocus system to the location of the (anatomical or non-anatomical) feature of interest.

[0052] In various examples, a precise determination of features of interest is made, and the features of interest are tracked across the entire frame, allowing the autofocus function to be re-activated once the field of view changes (e.g., due to the user moving the microscope or changing the microscope's magnification) or the location of the features of interest changes. Thus, the system can be configured to track the location of anatomical or non-anatomical features of interest across multiple frames of imaging sensor data, for example, by performing object detection or image segmentation on multiple frames and determining the correspondence between features in subsequent frames (e.g., based on the shape of the features indicated by the pixel mask output by image segmentation). However, this process is computationally intensive, and the differences between subsequent frames are usually small, so the tracking can be performed at a frame rate lower than the frame rate of the imaging sensor data. Thus, the system can be configured to detect the location of anatomical features of interest in many or every other frame of the imaging sensor data (e.g., every nth frame, n ∈ {2, 3, 4, 6, 10, 12, 15, 24, 30, 45, 60}). The system can be configured to trigger the autofocus function if the position of an anatomical or non-anatomical feature of interest shifts relative to the field of view of the imaging sensor data for at least a predetermined time interval, for example, at least 1 second, at least 2 seconds, at least 5 seconds, or at least 10 seconds (e.g., when the field of view shifts or when the anatomical feature moves). In other words, if the position of an anatomical or non-anatomical feature of interest changes not only temporarily but also in relation to the field of view, the autofocus function can be reactivated based on the position of the anatomical or non-anatomical feature of interest. The system may be configured to detect a feature of interest that has shifted relative to the field of view, for example, by comparing the position of the feature of interest in consecutive frames.

[0053] In some cases, features of interest can change during surgical procedures. For example, if the feature of interest is a tumor, the tumor may be removed; if the feature of interest is a blood vessel, the vessel may be deformed; and if the feature of interest is the intersection of two blood vessels, one of the vessels may be removed. In such cases, the anatomical feature of interest can be updated and / or redefined. In other words, the system can be configured to repeat the determination of an anatomical feature of interest if a previously determined feature of interest is removed or deformed. In this case, the area surrounding the previously determined anatomical feature of interest can be used as the area of ​​interest, and a new anatomical feature of interest can be determined within that area of ​​interest. For example, if the anatomical feature of interest is a blood vessel and the vessel is deformed at some point, the new deformed shape can be used in the definition of the anatomical feature of interest. Alternatively or in addition to this, different types of dynamic anatomical feature of interest definitions may be used. For example, if the anatomical feature of interest starts as the point where an artery and a vein intersect, and the vein may be removed from the field of view during surgery, the anatomical feature of interest can be dynamically redefined as the arterial portion after a particular branch. In some cases, smart and dynamic adaptation of anatomical features of interest can be performed by taking into account the area the surgeon is working in. For example, when a surgeon is digging into tissue towards a deeper tumor, the algorithm can follow a "point of action".

[0054] The selected anatomical or non-anatomical features of interest, and therefore the points used with respect to the autofocus function, can be communicated to the user of the surgical microscope system (e.g., a surgeon). As outlined above, a digital view of the surgical site can be generated based on imaging sensor data, which shows the surgical site along with additional information, such as the settings of the surgical microscope system. Therefore, the system can be configured to generate a digital view based on imaging sensor data. In addition, the selected anatomical or non-anatomical features of interest may be highlighted, for example, by displaying a color overlay over the anatomical or non-anatomical features of interest, or by displaying a visual indicator representing the focus. The system can be configured to highlight areas of interest (e.g., as contours) and / or features of interest (e.g., as another contour, color overlay, or visual indicator representing the focus) in the digital view by adding an overlay with a contour, color overlay, or visual indicator. The system can be configured to provide a display signal including the digital view to the display device 130 of the surgical microscope system.

[0055] In various examples, visual representations of multiple features and / or digital views of the surgical site are generated and provided to the display 130 of the surgical microscope system as part of a display signal. The visual representation or digital view can be viewed by a user of the surgical microscope system, e.g., a surgeon. For this purpose, the display signal may be provided to the display of the microscope system, e.g., an auxiliary display 130b or an eyepiece display 130a. Thus, the system can be configured to generate a display signal for the display device 130 of the microscope system, which is based on a digital view or visual representation. For example, the display signal may be a signal for driving (e.g., controlling) the display device 130. For example, the display signal may include video data and / or control commands for driving the display. For example, the display signal can be provided via one of one or more interfaces 112 of the system. Thus, system 110 may have a video interface 112 suitable for providing a display signal to the display 130 of the microscope system 100.

[0056] In the proposed microscope system, imaging sensor data is provided using at least one optical imaging sensor. Therefore, the optical imaging sensor 122 is configured to generate imaging sensor data. For example, at least one optical imaging sensor 122 of the microscope 120 includes, or may include, an imaging sensor 122 based on an active pixel sensor (APS) or a charge-coupled device (CCD). For example, in an APS imaging sensor, light is recorded at each pixel using a photodetector and active amplifier at the pixel. APS imaging sensors are often based on complementary metal-oxide-semiconductor (CMOS) or scientific CMOS (S-CMOS) technology. In a CCD imaging sensor, incident photons are converted into electronic charges at the semiconductor-oxide interface and then move between capacitance bins within the imaging sensor by an imaging sensor circuit for imaging. The processing system 110 can be configured to acquire (i.e., receive or read) image sensor data from the optical image sensor. The image sensor data can be acquired by receiving the image sensor data from the optical image sensor (for example, via interface 112), by reading the image sensor data from the memory of the optical image sensor (for example, via interface 112), or by reading the image sensor data from the storage device 116 of the system 110 after the image sensor data has been written to the storage device 116 by the optical image sensor or another system or processor.

[0057] One or more interfaces 112 of system 110 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information that may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, one or more interfaces 112 may include interface circuits configured to receive and / or transmit information. One or more processors 114 of system 110 can be implemented using any means for processing, such as one or more processing units, one or more processing devices, processors, computers, or programmable hardware components that can operate with software adapted accordingly. In other words, the functions of one or more processors 114 described above can be implemented as software, in which case the software runs on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, and the like. One or more storage devices 116 of system 110 may include at least one element of a group of computer-readable storage media, such as a hard disk drive, flash memory, floppy disk, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or magnetic or optical storage media such as network storage devices.

[0058] Further details and embodiments of the surgical microscope system will be referred to in relation to the proposed concept or one or more examples above or below (e.g., Figures 2 to 4). The surgical microscope system may feature one or more additional optional means corresponding to one or more embodiments of the proposed concept or one or more examples above or below.

[0059] Figure 2 shows a flowchart of an example of a corresponding method for a microscope in a surgical microscope system. The method includes acquiring imaging sensor data from at least one optical imaging sensor of the microscope (210). The method includes determining information about the user's area of ​​interest based on user input of the surgical microscope system (220). The method includes determining anatomical features of interest within the area of ​​interest (230). The method includes detecting the location of the anatomical features of interest in the imaging sensor data (240). The method includes triggering the autofocus function of the microscope to focus on the location of the anatomical features of interest (250).

[0060] For example, the method can be implemented by a surgical microscope system introduced in relation to one of Figures 1a to 1e. The features introduced in relation to the surgical microscope systems in Figures 1a to 1e may also be included in the corresponding method.

[0061] Further details and embodiments of the method are referred to in relation to the proposed concept or one or more examples above or below (e.g., Figures 1a-1e, 3-4). The method may include one or more additional optional features corresponding to one or more embodiments of the proposed concept or one or more examples above or below.

[0062] Various examples in this disclosure relate to concepts for adaptive autofocus.

[0063] Generally, surgeons need to manually focus on their region of interest each time they move the microscope. The proposed concept provides autofocus for a region of interest that the surgeon has manually defined (only once). Therefore, the surgeon does not need to spend time adjusting the focus each time they move the microscope. As a result, the region of interest can always be in focus.

[0064] Figure 3 shows a schematic diagram of an example of a digital view of a brain surgical site. Figure 3 shows the current focus 310 set before the focus is adjusted based on the determined feature of interest. Based on focus 3, region 320 is in focus. Figure 3 further shows a feature of interest 330 corresponding to a curved region of a blood vessel. The lock around the highlighted feature of interest indicates that the autofocus system is locked onto this feature of interest. Figure 3 further shows a first user interface element 340 that can be triggered to lock the focus function and a second user interface element 350 that can be triggered to lock the central function.

[0065] The proposed concept is based on four components. In the first component, the surgeon identifies the area of ​​interest that needs to be kept in focus. In the second component, the algorithm converts local information, i.e., the region of interest (ROI) or area of ​​interest, into features of interest (FOI) to be searched for in the image, as described in relation to Figures 1a and 2. In the third component, the algorithm searches for the identified FOI in each frame and segments the relevant image portion as a focal ROI. In the fourth component, the focal ROI is transmitted to the autofocus function.

[0066] The following provides some implementation examples for the four basic components listed above. For example, with respect to the first component, a surgeon can define an ROI using a pointing device, such as a computer mouse or touch panel. Alternatively, or in addition to this, the surgeon may use an object in the field of view, such as the tip or fingertip of a surgical tool. In some examples, a microscope system may use an eye-tracking device, which allows the surgeon to view the image area and indicate the ROI. For example, a defined ROI can be defined using a standard (i.e., geometric) shape (e.g., a circle or a square), or it may be defined using a freeform line that outlines the desired structure.

[0067] In some examples, the user can point or click on a desired feature once or multiple times, for example, by clicking along a blood vessel. In addition to or instead of this, the surgeon may use voice commands, for example, "follow the aortic bifurcation on the upper left." In some examples, the microscope can segment the image based on different criteria, and the surgeon can select one of the segments, for example, vascular segmentation and vascular features (e.g., bifurcation, deformation, color), surgical cavities, color / discoloration, and raised areas. For example, segmentation may also use information other than that shown in the color image, such as fluorescence, multispectral / hyperspectral images, and preoperative data (e.g., it is known that the tumor is deep in the tissue). For example, image segmentation may be performed or assisted by the surgeon's surgical activity, for example, the area the surgeon is cutting, touching, or manipulating.

[0068] After an ROI is selected by the surgeon, the system can perform the second component of the proposed concept, i.e., identify features of interest (FOIs). For example, the system can also identify arteries, veins, and unique shape identifiers for them, as well as morphological and color features such as branching. The system can identify low-contrast or invisible information within the ROI. For example, if a surgeon clicks multiple times on a vein, the system can understand that it is around a vein and not a nearby artery, or if a surgeon clicks on fully oxygenated blood (microbleed) on the surface of a cavity, the algorithm can identify the surgical cavity, taking into account the blood saturation and the shape of this element (microbleed). In some examples, the system can also consider adjacent structures, such as branches of the same vessel, that are not directly present within the user-defined area; i.e., a segment can be understood by the algorithm as a vascular portion between two characteristic branches or vascular curves. In some examples, the system can consider foreign bodies such as clips or stitches.

[0069] If no distinctive features exist, it may be possible to define an ROI, and therefore an FOI, as an area spatially oriented relative to other features, even if it is outside the ROI. For example, an ROI can be understood by an algorithm as tissue between two vertically oriented blood vessels that are near but outside the user-indicated ROI. A secondary camera with a wider field of view (FoV) can also be used for this purpose. In some cases, image analysis / processing techniques can be used to extract low-contrast or invisible features, or features that require quantification. For example, image processing can measure parameters related to blood vessels / capillaries, such as density, branching, and curvature, and these parameters can be used to segment and identify FOIs. A tissue area with high-density capillaries may be considered an FOI. In general, an FOI can be described by any one or more features. The results of the FOI identification process can be visualized to ensure the validity of the recognition. The surgeon can verify this, or take action only if it is incorrect. For example, the system can use specific symbols (e.g., arrows indicating the size of a discolored patch, arcs indicating a curve in a blood vessel, or the presence of a fluorescent signal) to indicate the type of feature the algorithm uses for tracking. This allows surgeons to better control the effectiveness of the tracking.

[0070] In the third component of the system, the algorithm searches for FOIs. For example, the algorithm can search for FOIs using a dynamic FOI description. For example, an FOI may be searchable in each frame, but the description of the FOI may be redefined (i.e., updated) periodically (e.g., every n frames). For example, if the FOI is a blood vessel and the vessel deforms at some point, the new deformed shape may be used in the definition. Alternatively or in addition to this, different types of dynamic FOI definitions may be used. For example, in addition to the dynamic FOI definitions identified above, different types of FOIs can provide the ability to change how FOIs are defined. For example, an FOI may start as a point where an artery intersects with a vein, but the vein may be removed from the FOV during surgery. In this case, the FOI can be dynamically redefined as the arterial portion after a particular branch. In some examples, smart and dynamic FOI adaptation can be performed by considering the area the surgeon is working in. For example, when the surgeon is working deep into the tissue toward a deeper tumor, the algorithm can follow a "point of action". Similar to the definition of FOIs, the results of the FOI search process can be visualized (for example, by outlining the current ROI used for autofocus). This can be done continuously, periodically (e.g., a 1-second flash every 15 seconds), when there is a change in parameters (e.g., when the length of an anatomical structure changes), or when the recognized feature changes (e.g., when the algorithm stops tracking vascular crossings and starts tracking branching). In some examples, the proposed system can demonstrate the algorithm's certainty in FOI identification. This allows surgeons to redefine ROIs / FOIs before critical parts of surgery, thus avoiding the major inconvenience of autofocus failure during critical surgical steps.

[0071] Regarding the fourth component, namely the communication of ROIs to the autofocus subsystem, the following implementation approaches can be used. For example, the focus ROI may be transmitted with a time delay to avoid errors caused by instantaneous misinterpretation of the FOI by the algorithm. For example, if the FOI is covered by another object (e.g., tissue, tool, gauze), this will not cause the algorithm to "gone hazy." For example, the communication frequency may be influenced by the certainty / confidence of FOI recognition. For example, if the algorithm is certain (i.e., the FOI is found with high certainty), the focus ROI can be updated instantaneously, but if the certainty is low, it may wait until the confidence increases or a warning is issued. In some examples, ROIs may have different weights for ROI zones with different confidence levels. For example, a well-recognized tissue structure may have a weight of 90%, while other tissue areas with ambiguous recognition (e.g., due to weak fluorescence signals) may have a weight of 30%. This means that autofocus can prioritize focusing on zones with high confidence.

[0072] Further details and embodiments of the concept for adaptive autofocus are referred to in relation to the proposed concept or one or more examples described above or below (e.g., Figures 1a-2, Figure 4). The concept for adaptive autofocus may feature one or more embodiments of the proposed concept or one or more additional optional means corresponding to one or more examples above or below.

[0073] Some embodiments relate to a microscope that includes a system such as those described in relation to one or more of Figures 1 to 3. Alternatively, the microscope may be part of a system such as those described in relation to one or more of Figures 1 to 3, or may be connected to a system such as those described in relation to one or more of Figures 1 to 3. Figure 4 shows a schematic diagram of a system 400 configured to perform the methods described herein. The system 400 includes a microscope 410 and a computer system 420. The microscope 410 is configured to take images and is connected to the computer system 420. The computer system 420 is configured to perform at least a portion of the methods described herein. The computer system 420 may be configured to perform machine learning algorithms. The computer system 420 and the microscope 410 may be separate entities, but may be integrated within a single common housing. The computer system 420 may be part of the central processing system of the microscope 410, and / or the computer system 420 may be part of a dependent component of the microscope 410, such as a sensor, actor, camera, or illumination unit of the microscope 410.

[0074] The computer system 420 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed to various locations such as local clients and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which may be of any kind. As used herein, a processor may be intended to be any kind of computing circuit, such as a microprocessor for a microscope or microscopic component (e.g., a camera), a microcontroller, a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multicore processor, a field-programmable gate array (FPGA), or any other kind of processor or processing circuit. Other types of circuits that may be included in the computer system 420 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (communication circuits, etc.) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 420 may also include one or more storage devices that may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 420 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touchscreen, voice recognition device, or any other device which enables a user of the system to input information into and receive information from the computer system 420.

[0075] Some or all of the steps may be performed by a hardware device (or by using a hardware device), such as a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most critical steps may be performed by such a device.

[0076] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is feasible using a non-transient recording medium, which is a digital recording medium, etc., that stores electronically readable control signals and cooperates (or can cooperate) with a programmable computer system to carry out each method. Examples include floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs, or FLASH memory. Thus, the digital recording medium may be computer-readable.

[0077] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so as to carry out any of the methods described herein.

[0078] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to perform one of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.

[0079] Another embodiment includes a computer program stored in a machine-readable carrier for carrying out any of the methods described herein.

[0080] Therefore, in other words, embodiments of the present invention are computer programs having program code for carrying out any of the methods described herein when the computer program is executed on a computer.

[0081] Accordingly, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for carrying out any of the methods described herein when executed by a processor. The data carrier, digital recording medium, or recording medium is typically tangible and / or non-transient. Another embodiment of the present invention is an apparatus as described herein, comprising a processor and a recording medium.

[0082] Therefore, another embodiment of the present invention is a data stream or signal sequence representing a computer program for carrying out any of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0083] Another embodiment includes processing means, for example, a computer or programmable logic device configured or adapted to carry out any of the methods described herein.

[0084] Another embodiment includes a computer having an installed computer program for carrying out any of the methods described herein.

[0085] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for carrying out any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0086] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to carry out any of the methods described herein. Generally, the methods are advantageously carried out by any hardware device.

[0087] As used herein, the term "and / or" includes all possible combinations of one or more of the related items and may be abbreviated as " / ".

[0088] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus.

[0089] Embodiments may be based on the use of machine learning models or machine learning algorithms. Instead of relying on models and inference, machine learning may refer to algorithms and statistical models that a computer system can use to perform a particular task without using explicit instructions. For example, machine learning may use data transformations inferred from the analysis of historical data and / or training data instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or a machine learning algorithm. For a machine learning model to analyze image content, the machine learning model may be trained with training images as input and training content information as output. By training a machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model “learns” to recognize image content so that image content not included in the training data becomes recognizable using the machine learning model. The same principle may be used in the same way for other types of sensor data: by training a machine learning model with training sensor data and a desired output, the machine learning model “learns” the transformation between sensor data and output, which can then be used to provide output based on non-trained sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain feature vectors that can be used as input to a machine learning model.

[0090] A machine learning model may be trained using training input data. The example above uses a training method called "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, each of which may contain multiple input data values ​​and multiple desired output values; that is, each training sample is associated with a desired output value. By specifying both the training samples and the desired output values, the machine learning model "learns" during training which output values ​​to provide based on input samples similar to the provided samples. In addition to supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack corresponding desired output values. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). A classification algorithm may be used if the output is limited to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. A regression algorithm may be used if the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two objects are. In addition to supervised or semi-supervised learning, unsupervised learning may be used to train machine learning models. In unsupervised learning, input data (only) may be provided, and unsupervised learning algorithms may be used to find structure in the input data (for example, by grouping or clustering the input data, or by finding commonalities in the data). Clustering is the process of assigning input data containing multiple input values ​​into multiple subsets (clusters), so that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but are not similar to input values ​​in another cluster.

[0091] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as “software agents”) are trained to take actions in their surroundings. A reward is calculated based on the actions taken. Reinforcement learning is based on training one or more software agents to choose actions that result in software agents that perform better on a given task, with cumulative rewards increasing (as revealed by the increase in rewards).

[0092] Furthermore, several techniques may be applied as part of a machine learning algorithm. For example, feature representation learning may be used. In other words, a machine learning model may be trained at least partially using feature representation learning, and / or a machine learning algorithm may include feature representation learning components. A feature representation learning algorithm, which may be called a representation learning algorithm, may not only store information in its own input but may also transform the information to make it useful, often as a preprocessing step before performing classification or prediction. Feature representation learning may be based, for example, on principal component analysis or cluster analysis.

[0093] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide the identification of input values ​​that raise suspicion by being significantly different from the majority of the input or training data. In other words, a machine learning model may be trained with anomaly detection, at least in part, and / or a machine learning algorithm may include anomaly detection components.

[0094] In some examples, a machine learning algorithm may use a decision tree as its predictive model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by branches of the decision tree, and the output values ​​corresponding to these items may be represented by leaves of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. When discrete values ​​are used, the decision tree may be represented as a classification tree, and when continuous values ​​are used, the decision tree may be represented as a regression tree.

[0095] Correlation rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model may be based on one or more correlation rules. Correlation rules are created by identifying relationships between variables in a large amount of data. A machine learning algorithm may identify and / or utilize one or more correlational rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply knowledge.

[0096] Machine learning algorithms are typically based on machine learning models. In other words, the term “machine learning algorithm” may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term “machine learning model” may refer to a set of data structures and / or rules that represent learned knowledge (for example, based on training performed by a machine learning algorithm). In embodiments, usage of a machine learning algorithm may mean usage of one underlying machine learning model (or multiple underlying machine learning models). Usage of a machine learning model may mean that a machine learning model and / or a set of data structures / rules that are a machine learning model are trained by a machine learning algorithm.

[0097] For example, a machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or brain. An ANN consists of multiple interconnected nodes and multiple junctions, so-called edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (simply) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (nonlinear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weights" of the edges or nodes that provide the input. The weights of nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0098] Alternatively, a machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with a relevant learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing inputs with multiple training input values ​​belonging to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, a machine learning model may be a Bayesian network, which is a stochastic directed acyclic graphical model. A Bayesian network may use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, a machine learning model may be based on a search algorithm and a genetic algorithm, which is a heuristic method that mimics the process of natural selection. [Explanation of symbols]

[0099] 100 Surgical Microscope Systems 110 System 112 One or more interfaces 114 One or more processors 116 One or more storage devices 120 Microscopes 122 Optical imaging sensor 130 displays 130a Eyepiece Display 130b Auxiliary Display 140 Arm 150-158 Branching points of blood vessels 160 Curves of blood vessels 162 A part of a blood vessel 164;166 Branching points of blood vessels 210 Acquire imaging sensor data 220 Determine information regarding areas of interest. 230 Determine anatomical features of interest 240 Detecting the location of anatomical features of interest 250 Trigger the autofocus function 310 Current Focus 320 Focal area 330 Interest Features 340;350 User Interface Elements 400 System 410 Microscope 420 Computer Systems

Claims

1. A system (110; 420) for a microscope (120; 410) of a surgical microscopy system (100; 400), said system (110; 420) comprising one or more processors (114) and one or more storage devices (116), said system comprising: acquiring image sensor data from at least one optical image sensor (122) of the microscope; determining information regarding an area of ​​interest of a user of the surgical microscopy system based on input of the user; determining an anatomical feature of interest within the area of ​​interest; locating the anatomical feature of interest in the imaging sensor data; triggering an autofocus function of the microscope to focus on the location of the anatomical feature of interest; A system that is configured to:

2. the system is configured to track the position of the anatomical feature of interest across multiple frames of the imaging sensor data. The system of claim 1 .

3. the system is configured to trigger the autofocus function when a position of the anatomical feature of interest is shifted relative to a field of view of the imaging sensor data by at least a predetermined time interval. The system of claim 2.

4. the system is configured to detect the location of the anatomical feature of interest in at most every other frame of the imaging sensor data.

4. The system according to claim 2 or 3.

5. the system is configured to locate the area of ​​interest in the imaging sensor data and determine anatomical features of interest within a portion of the imaging sensor data representing the area of ​​interest. The system of claim 1 .

6. the system is configured to perform image segmentation on at least a portion of the imaging sensor data representing the area of ​​interest to determine at least one feature present in the portion of the imaging sensor data representing the area of ​​interest, and to determine an anatomical feature of interest based on the at least one feature present in the portion of the imaging sensor data representing the area of ​​interest. The system of claim 5.

7. the system is configured to perform object detection on at least a portion of the imaging sensor data representing the area of ​​interest to identify at least one feature present in the portion of the imaging sensor data representing the area of ​​interest, and to determine an anatomical feature of interest based on the identified at least one feature present in the portion of the imaging sensor data representing the area of ​​interest.

7. The system according to claim 5 or 6.

8. the system is configured to perform the object detection to identify at least one of a blood vessel, a blood vessel branch, a hemorrhage, and a tumor within at least a portion of the imaging sensor data representing the area of ​​interest. The system of claim 7.

9. the system is configured to perform the object detection to identify at least one of a clip and a stitch as a non-anatomical feature of interest, detect a location of the non-anatomical feature of interest in the imaging sensor data, and perform the autofocus function of the microscope further based on the location of the non-anatomical feature of interest. The system of claim 7.

10. the system is configured to determine the extent of the anatomical feature of interest based on the extent of one or more features located adjacent to a feature on which the anatomical feature of interest is based; The system of claim 5.

11. the system is configured to, if the area of ​​interest indicated by the user relates to two or more features or an area between two or more features, determine an anatomical feature of interest based on the two or more features, and determine a position of the anatomical feature of interest based on positions of the two or more features. The system of claim 5.

12. the system is configured to, when an area of ​​interest indicated by the user relates to an area between the two or more features, select the two or more features from features located within the area of ​​interest and features located outside the area of ​​interest. The system of claim 11.

13. the system is configured to detect a pointer manipulated by the user within the imaging sensor data to determine the area of ​​interest; Alternatively, the system is configured to determine a portion of a surgical site being operated on by the user in the imaging sensor data to determine the area of ​​interest; Alternatively, the system is configured to determine the area of ​​interest using an eye-tracking mechanism; Alternatively, the system is configured to determine the area of ​​interest based on a voice description of anatomical features obtained via a voice command system of the surgical microscopy system; Alternatively, the system is configured to determine the area of ​​interest based on a user input signal obtained via a user interface (130) of the surgical microscopy system; Alternatively, the system is configured to determine the area of ​​interest from a predetermined image area after the user has finished aligning the field of view. The system of claim 1 .

14. the system is configured to perform image segmentation and / or object detection to determine a plurality of features in the imaging sensor data, determine a visual representation of the plurality of features, provide a display signal including the visual representation to a display device (130) of the surgical microscopy system, and obtain input from the user in response to the visual representation of the plurality of features. The system of claim 1 .

15. the imaging sensor data includes a first component having color imaging data and a second component having at least one of hyperspectral imaging data, multispectral imaging data, and fluorescence imaging data, and the system is configured to determine and / or locate the anatomical feature of interest based on at least the second component. The system of claim 1 .

16. the system is configured to generate a digital view based on the imaging sensor data, highlight areas and / or features of interest in the digital view, and provide a display signal including the digital view to a display device (130) of the surgical microscopy system. The system of claim 1 .

17. A microscope (120; 410) having an optical imaging sensor and a system (110; 420) according to claim 1, Surgical microscope system (100; 400).

18. 1. A method for a microscope of a surgical microscopy system, the method comprising: acquiring (210) imaging sensor data from at least one optical imaging sensor of the microscope; determining (220) information regarding an area of ​​interest of a user of the surgical microscopy system based on input of the user; determining anatomical features of interest within the area of ​​interest (230); locating the anatomical feature of interest in the imaging sensor data (240); Triggering an autofocus function of the microscope (250) to focus on the location of the anatomical feature of interest; A method comprising:

19. A computer program comprising:

20. A method for implementing a method according to claim 18, comprising: a program code for executing a method according to claim 18 when the program code is executed on a processor; Computer program.

Images