Systems, methods, and computer program for a surgical imaging system

Abstract

Examples relate to a system (110), a method, and a computer program for a surgical imaging system (100), and to a corresponding surgical imaging system including such a system for surgical imaging. The system (110) is configured to obtain an eye movement tracking sensor signal from an eye movement tracking system (130). The system is configured to determine a gaze of a user of a surgical imaging device (120) of the surgical imaging system to a portion of a field of view of the surgical imaging device based on the eye movement tracking sensor signal. The system is configured to adjust a field of view of the surgical imaging device based on a user's gaze of a portion of the field of view.

Description

Technical Field

[0001] Examples relate to systems, methods, and computer programs for surgical imaging systems, and to corresponding surgical imaging systems including such systems for surgical imaging systems. Background Art

[0002] A surgical imaging system is an imaging system, such as a microscope system, designed for use during a surgical procedure. Such imaging systems typically provide high magnification. Therefore, the field of view of the surgical imaging device of such a surgical imaging system is typically small. During a surgical procedure, the surgeon or an assistant typically wishes to adjust the field of view, for example, to focus on different areas of the surgical site being operated on by the surgeon.

[0003] In some cases, the surgeon may view the surgical site using a head mounted display that uses imaging sensor data generated by a surgical imaging device. For example, an example of such use of a head mounted display is given in EP3904947A1.

[0004] It would be desirable to have an improved concept for a surgical imaging system that provides improved adjustment of the field of view seen by the surgeon. Summary of the invention

[0005] This desire is addressed in the subject-matter of the independent claims.

[0006] Various examples of the present disclosure are based on the discovery that the field of view of a surgical imaging device (e.g., a microscope of a surgical microscope system) can be automatically or semi-automatically adjusted by tracking the gaze of a user of the surgical imaging device. In the proposed concept, the gaze of the user of the surgical imaging device on a portion of the field of view is determined, and the field of view is adjusted based on the determined gaze. Thus, the field of view can be adjusted without the need for the surgeon to manually adjust the field of view.

[0007] Some aspects of the present disclosure relate to a system for a surgical imaging system. The system includes one or more processors and one or more storage devices. The system is configured to obtain an eye tracking sensor signal from an eye tracking system. The system is configured to determine a gaze of a user of a surgical imaging device of the surgical imaging system on a portion of a field of view of the surgical imaging device based on the eye tracking sensor signal. The system is configured to adjust the field of view of the surgical imaging device based on the user's gaze on the portion of the field of view. By determining the field of view based on the user's gaze, the field of view can be adjusted without requiring the surgeon to manually adjust the field of view.

[0008] In some examples, the system can be configured to focus the field of view around a portion of the field of view where gaze can be detected. Thus, the field of view can be adjusted so that the user has an improved field of view of the portion of the field of view that the user has gazed at.

[0009] In some cases, it may not be desirable to continuously adjust the field of view, as the user may prefer to keep the field of view still while the surgeon performs part of the surgical procedure. Instead, the user may trigger the adjustment using an input device. Thus, the system may be configured to obtain an input signal from an input device of the surgical imaging system and trigger the adjustment of the field of view based on a user command to trigger the adjustment of the field of view contained in the input signal. This may avoid inadvertent adjustments to the field of view.

[0010] For example, the system can be configured to obtain an input signal from one of an input modality of a handle of the surgical imaging system, an input modality of a foot pedal of the surgical imaging system, an input modality of the surgical imaging device, a camera-based input device of the surgical imaging system, a depth sensor-based input device of the surgical imaging system, and a voice-based input device of the surgical imaging system. Obviously, the surgical imaging system can include many input devices, which allows providing user commands with little effort and a high degree of flexibility.

[0011] However, in some examples, the system can be configured to continuously adjust the field of view of the surgical imaging device based on the user's gaze. In this case, no additional trigger may be required, which can further reduce the burden on the user. In particular, the system can be configured to continuously and gradually adjust the field of view of the surgical imaging device based on the user's gaze. The gradual adjustment can avoid sudden changes in the field of view that may be disturbing to the user.

[0012] In some examples, adjustment of the field of view can employ image analysis techniques to identify a possible target of the user's gaze. For example, the system can be configured to perform object detection on imaging sensor data representing the field of view of a surgical imaging device to determine one or more regions of interest visible in the field of view. The system can be configured to adjust the field of view to a region of interest that intersects a portion of the field of view where the gaze can be detected. This can improve the selection of an adjusted field of view, for example, because the field of view can be adjusted to provide the user with an improved view of an object of interest.

[0013] In general, the concepts presented can be used with purely optical surgical imaging devices, such as a microscope with an eyepiece that displays the beam path through a camera. However, if the field of view is displayed as a digital view, such as through an eyepiece display, an auxiliary display (mounted on the surgical imaging system), or a head-mounted display, various examples of the present disclosure can provide additional functionality. For example, the system can be configured to generate a digital view based on imaging sensor data and highlight a region of interest that intersects a portion of the field of view where gaze is detected within the field of view before adjusting the field of view. This can provide visual feedback to the user before ultimately adjusting the field of view based on the gaze.

[0014] Surgical imaging systems are typically highly complex and multifunctional systems that provide a number of functions that can be used to adjust the field of view. Thus, adjusting the field of view may include one or more of: providing a control signal to a robotic adjustment system of the surgical imaging system, providing a control signal to a zoom function of the surgical imaging system, and adjusting a crop factor of image processing performed by the surgical imaging system (thereby applying a digital zoom). Each of the above approaches (robotic adjustment system, zoom function, and image processing) has advantages, such as a high degree of freedom (robotic adjustment system) and the ability to adjust the field of view without moving the surgical imaging device (zoom function and image processing).

[0015] After certain adjustments, such as applying optical zoom, changing working distance, or lateral movement of the surgical imaging device, the field of view may be out of focus. In some examples, the system can be configured to trigger an autofocus function of the surgical imaging system after adjusting the field of view of the surgical imaging device. By using the autofocus function, a clear view of the surgical site can be ensured.

[0016] As described above, the proposed concepts can be used with digital or hybrid surgical imaging devices that are capable of providing a digital view of a surgical site. For example, the system can be configured to obtain imaging sensor data showing a field of view of a surgical imaging device from an optical imaging sensor of the surgical imaging device, and generate a digital view based on the imaging sensor data, the digital view representing an adjusted field of view. This can provide additional flexibility, both in terms of the display used (e.g., a digital eyepiece, an auxiliary display, or a head-mounted display), and in terms of the image processing performed to allow an enhanced view of the surgical site to be obtained.

[0017] In general, the user's gaze can be determined by different modalities. For example, the system can be configured to obtain eye tracking sensor signals from an eye tracking system integrated into the eyepiece of the surgical imaging device. This option can be selected if the display used is an eyepiece display, or if the eyepiece provides an optical view of the surgical site (through the surgical imaging device).

[0018] Alternatively or additionally, the system may be configured to obtain eye tracking sensor signals from an eye tracking system integrated into a head-mounted display of the surgical imaging system. This option may be selected if the display used is a head-mounted display.

[0019] One aspect of the present disclosure relates to a surgical imaging system, which includes a surgical imaging device, an eye tracking sensor and the system described above. For example, the surgical imaging system may be a surgical microscope system.

[0020] Some aspects of the present disclosure relate to a method for a surgical imaging system. The method includes obtaining an eye tracking sensor signal of an eye tracking system. The method includes determining a gaze of a user of a surgical imaging device of the surgical imaging system on a portion of a field of view of the surgical imaging device based on the eye tracking sensor signal. The method includes adjusting the field of view of the surgical imaging device based on the gaze of the user on the portion of the field of view.

[0021] One aspect of the present disclosure relates to a computer program having a program code which, when executed on a processor, performs the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Some examples of apparatus and / or methods will be described below, by way of example only, and with reference to the accompanying drawings, in which

[0023] Figure 1a A block diagram illustrating an example of a system for a surgical imaging system;

[0024] Figure 1b A schematic diagram showing an example of a surgical imaging system, particularly a surgical microscope system;

[0025] Figure 2 A flow chart illustrating an example of a method for a surgical imaging system;

[0026] Figure 3 An example of gaze-based field of view adjustment is shown;

[0027] Figure 4 shows an example of a visual indicator used to highlight an area of ​​interest prior to adjusting the field of view; and

[0028] Figure 5 A schematic diagram of an example of a system including a surgical imaging device and a computer system is shown. DETAILED DESCRIPTION

[0029] Various examples will now be described more fully with reference to the accompanying drawings, in which some examples are shown. In the drawings, the thickness of lines, layers and / or regions may be exaggerated for clarity.

[0030] Figure 1a A block diagram of an example of a system 110 for use with the surgical imaging system 100 is shown (in Figure 1b 100). System 110 is a component of surgical imaging system 100 and may be used to control various aspects of surgical imaging system 100. In particular, it may be used to control (and adjust) the field of view provided by surgical imaging device 120 of surgical imaging system 100 in various ways (which will be described in more detail below). In addition, system 110 may be configured to control other aspects of the surgical imaging system, and / or perform sensor data processing (e.g., imaging sensor data processing), and / or provide display signals for various displays in the surgical imaging system.

[0031] In general, the system 110 may be considered a computer system. The system 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the system 110 also includes one or more interfaces 112. The one or more processors 114 are coupled to the one or more storage devices 116 and to the one or more interfaces 112. In general, the functionality of the system 110 may be provided by the one or more processors 114 in conjunction with the one or more interfaces 112 (for exchanging data / information with one or more other components of the surgical imaging system 100 (and external to the surgical imaging system 100), such as the optical imaging sensor of the surgical imaging device 120, the eye tracking system 130 of the surgical imaging system, the input devices 140; 150; 160, the robotic adjustment system 170, and / or the headgear / head-mounted display 180), and in conjunction with the one or more storage devices 116 (for storing information, such as machine-readable instructions of a computer program executed by the one or more processors). In general, the functionality of the one or more processors 114 may be implemented by the one or more processors 114 executing the machine-readable instructions. Thus, any features attributed to the one or more processors 114 may be defined by one or more of the plurality of machine-readable instructions. The system 110 may include the machine-readable instructions, for example, within one or more storage devices 116.

[0032] As described above, system 110 is part of surgical imaging system 100, which includes various components in addition to system 110. For example, surgical imaging system 100 includes surgical imaging device 120, and may include one or more additional components, such as eye tracking system 130, input devices 140, 150, 160, robotic adjustment system 170, and / or head mount / head mounted display 180 as described above. Figure 1b A schematic diagram of an example of such a surgical imaging system 100, in particular a surgical microscope system 100, is shown. In the following, the surgical imaging system 100 may also be referred to as a surgical microscope system 100, which is a surgical imaging system 100 including a (surgical) microscope as a surgical imaging device 120. However, the proposed concept is not limited to such an embodiment. The surgical imaging system 100 can be based on various (single or multiple) surgical imaging devices, such as one or more microscopes, one or more endoscopes, one or more radiation imaging devices, one or more surgical tomography devices (e.g., optical coherence tomography devices), etc. Therefore, the surgical imaging system can alternatively be a surgical endoscope system, a surgical radiation imaging system, or a surgical tomography system. However, the following illustration assumes that the surgical imaging device 120 is a (surgical) microscope and the surgical imaging system 100 is a surgical microscope system 100.

[0033] Thus, the surgical imaging system or surgical microscope system 100 may include a microscope 120. In general, a microscope, such as microscope 120, is an optical instrument suitable for examining objects that are too small to be examined by the human eye (alone). For example, a microscope can provide optical magnification of a sample. The microscope can be a purely optical microscope (wherein the eyepiece displays light propagating from a surgical site observed through the microscope 120), a hybrid (optical and digital) microscope, or a digital microscope (wherein the eyepiece or display displays a digital view of the surgical site). In modern hybrid or digital microscopes, an optical imaging sensor (also) provides optical magnification. In the latter two cases, the microscope 120 therefore includes an optical imaging sensor coupled to the system 110. The microscope 120 may also include one or more optical magnification components for magnifying the view on the sample, such as an objective (i.e., a lens). For example, the surgical imaging device or microscope 120 is also often referred to as the "optical carrier" of the surgical imaging system.

[0034] There are various different types of surgical imaging devices. If the surgical imaging device is used in the medical or biological fields, the object observed by the surgical imaging device can be an organic tissue sample (for example, arranged in a culture dish or present in a part of the patient's body). In the various examples presented here, the surgical imaging device 120 can be a microscope of a surgical microscope system, i.e., a microscope used during a surgical procedure, such as tumor surgery or during tumor surgery. Therefore, the object viewed by the surgical imaging device (displayed in the field of view of the surgical imaging device) and the object displayed in the digital view generated based on the imaging sensor data provided by the (optional) optical imaging sensor can be an organic tissue sample of the patient, and in particular can be a surgical site operated by the surgeon during the surgical procedure. For example, the object to be imaged, i.e., the surgical site, can be a surgical site of the brain during a neurosurgery procedure. However, the proposed concept is also applicable to other types of surgeries, such as cardiac surgery or ophthalmology.

[0035] Generally speaking, a surgical imaging system, such as surgical microscope system 100, is a system that includes a microscope 120 and additional components that operate in conjunction with the microscope (e.g., system 110 (which may be a computer system adapted to control the surgical microscope system and, for example, process imaging sensor data of the microscope)) as well as additional sensors, displays, etc.

[0036] Figure 1b A schematic diagram of an example of a surgical imaging system 100 , and in particular a surgical microscope system 100 , is shown, which includes a system 110 and a microscope 120 . Figure 1b The illustrated surgical operating microscope system 100 includes several optional components, such as a base unit 105 (comprising the system 110) with a (rolling) stand, an eyepiece 125 (e.g., with an eyepiece display) disposed on the microscope 120, an eye tracking system 130, an auxiliary display disposed on the base unit 105, one or more input devices 140; 150; 160, and a robotic adjustment system 170, which is shown as a (robotic or manual) arm 170 for holding the microscope 120 and is coupled to the base unit 105 and the microscope 120. In general, these optional and non-optional components can be coupled to the system 110, and the system 110 can be configured to control and / or interact with the respective components. Figure 1b1 , the connection between the head-mounted device 180 and the system 110 and the connection between the two examples of the eye-tracking system 130 (as part of the head-mounted device 180 or as part of the eyepiece 125) are shown as wired connections. In other words, the head-mounted device 180 and / or the eye-tracking system 130 may be connected to the system 110 using a wired connection, i.e., they may be wire-bound devices. Therefore, the interface 112 may be configured to communicate with the head-mounted device 180 and / or the eye-tracking system 130 using a wired connection. However, the proposed concepts are not limited to these examples. Alternatively, the head-mounted device 180 and / or the eye-tracking system may be a wireless device, i.e., a wireless head-mounted device 180 including the eye-tracking system 130, i.e., they may be connected to the system 110 using a wireless (data) connection. Therefore, the interface 112 may be configured to communicate with the head-mounted device 180 and / or the eye-tracking system 130 using a wireless (data) connection.

[0037] In the proposed concept, the system 110 is particularly used to control the field of view of the surgical imaging device 120. This is achieved by tracking the gaze of a user (e.g., a surgeon or an assistant responsible for controlling the surgical imaging system) and adjusting the field of view accordingly. For example, the system 110 is configured to obtain eye tracking sensor signals from the eye tracking system 130. The system 110 is configured to determine the gaze of the user of the surgical imaging device 120 of the surgical imaging system on a portion of the field of view of the surgical imaging device based on the eye tracking sensor signals. The system 110 is configured to adjust the field of view of the surgical imaging device based on the gaze of the user on the portion of the field of view.

[0038] To perform gaze-based field of view adjustment, the system uses eye tracking sensor signals from eye tracking system 130. In general, eye tracking sensor signals may indicate the gaze of the user, i.e., the direction in which the user is looking. For example, the eye tracking sensor signal may indicate where the user is looking, such as a portion of a display, or a central or peripheral area of ​​the eyepiece. The implementation of the eye tracking sensor signal may depend on the eye tracking sensor used. For example, the system may be configured to obtain eye tracking sensor signals from an eye tracking system integrated into the eyepiece 125 of the surgical imaging device. In this case, the eye tracking sensor signal may indicate whether the user is looking toward the central portion of the field of view provided by the eyepiece, or toward the peripheral portion of the field of view provided by the eyepiece (and its direction, such as to the right, to the left, up, down, to the upper right, to the lower left, etc.). Alternatively, system 110 may be configured to obtain eye tracking sensor signals from an eye tracking sensor coupled to a display (e.g., to the auxiliary display described above) or as part of a head-mounted device / head-mounted display. For example, the system can be configured to obtain eye tracking sensor signals from an eye tracking system integrated into the surgical imaging system headgear 180. In these cases, the eye tracking sensor signals can indicate the portion of the display that the user is looking at (e.g., focusing on).

[0039] The system then uses the eye tracking sensor signal to determine the user's gaze, and in particular the portion of the field of view to which the gaze is directed. For example, the system can be configured to estimate the portion of the field of view to which the gaze is directed from the viewing direction if the eye tracking sensor signal indicates a viewing direction of the user (e.g., toward the center or toward the periphery (and directions thereof)). Alternatively, the system can be configured to convert the portion of the display to a corresponding portion of the field of view if the eye tracking sensor signal indicates a portion of the display (e.g., within a head mounted display or on the auxiliary display described above) to which the user is looking. In both cases, the system determines the portion of the field of view currently provided to the user that the user is looking at. In the present context, the field of view of the surgical imaging device is the field of view provided to the user by the surgical imaging device, e.g., the field of view seen by the user when the user looks through the eyepiece or the user looks at a display of the surgical imaging device. In some examples, when multiple displays (e.g., multiple head mounted displays and / or multiple digital eyepieces) are used, the field of view of different users of the surgical imaging system may be different, for example, by using user-specific image cropping and / or applying user-specific digital zoom.

[0040] In the preceding examples, a portion of the field of view was used without reference to a particular feature of interest. Rather, only a portion of the field of view was referenced. However, in some cases, additional image processing may be used to identify one or more specific regions of interest within the field of view (e.g., anatomical features of interest, or more generally, objects of interest (e.g., surgical markers, etc.) and take them into account when adjusting the field of view of the surgical imaging device. For example, the system may be configured to perform (machine learning-based) object detection (or more generally, image segmentation) on imaging sensor data representing the field of view of the surgical imaging device to determine one or more regions of interest (e.g., anatomical or non-anatomical features or objects of interest) visible in the field of view. For example, the imaging sensor data may be analyzed to determine and distinguish features in the imaging sensor data, such as anatomical features (e.g., vessels, tumors, branches, tissue portions, etc.) or non-anatomical features (e.g., markers, clips, or sutures). To this end, one or both of the following machine learning-based techniques may be used: - Image segmentation and object detection. The system can be configured to perform image segmentation and / or object detection to determine features and thereby regions of interest within the imaging sensor data. In object detection, the location of one or more predefined objects (i.e., the objects for which the corresponding machine learning model is trained) in the imaging sensor data, as well as the classification of the object (if the machine learning model is trained to detect multiple different types of objects) are output by the machine learning model. Generally speaking, the location of one or more predefined objects is provided in the form of a bounding box, i.e., a set of locations forming a rectangular shape around the corresponding detected object. In image segmentation, the location of features (i.e., portions of the imaging sensor data having similar properties, such as portions belonging to the same object) is output by the machine learning model. Typically, the location of features is provided in the form of a pixel mask, i.e., the location of pixels belonging to the feature is output on a per-feature basis.

[0041] For object detection and image segmentation, a trained machine learning model is used to perform the corresponding tasks. For example, in order to train a machine learning model being trained to perform object detection, multiple imaging sensor data or imaging sensor data samples can be provided as training input samples, and a list of corresponding bounding box coordinates can be provided as the expected output of the training, and a training algorithm based on supervised learning is used to perform training using multiple training input samples and corresponding expected outputs. For example, in order to train a machine learning model being trained to perform image segmentation, multiple imaging sensor data or imaging sensor data samples can be provided as training input samples, and corresponding pixel masks can be provided as the expected output of the training, and a training algorithm based on supervised learning is used to perform training using multiple training input samples and corresponding expected outputs. In some examples, the same machine learning model can be used to perform both object detection and image segmentation. In this case, the above two types of expected outputs can be used in parallel during the training process, and the machine learning model is trained to output both bounding boxes and pixel masks. Therefore, the machine learning model can be used to determine one or more potential features of interest, thereby determining one or more regions of interest.

[0042] Now, these one or more regions of interest can be used to support determining the portion of the field of view where the gaze is detected. For example, the system can be configured to associate the user's gaze with one or more (potential) regions of interest and determine the portion of the field of view so that it intersects with one of the one or more regions of interest. In fact, the system can be configured to adjust the field of view to the region of interest that intersects with the portion of the field of view where the gaze is detected.

[0043] The system is then configured to adjust the field of view of the surgical imaging device based on the user's gaze at a portion of the field of view. For example, the system can be configured to adjust the field of view based on the portion of the field of view to which the gaze is directed. Because the user is looking toward this portion of the field of view, the field of view can be adjusted (e.g., moved) so that the portion of the field of view to which the gaze is directed moves toward the center of the field of view. For example, Figure 3 An example of such an adjustment is shown. The system can be configured to center the field of view around the portion of the field of view where gaze is detected. In other words, the field of view can be adjusted so that the portion of the field of view where gaze is detected before the adjustment is located at the center of the field of view (or closer to the center) after the adjustment.

[0044] In general, there are different ways to adjust the field of view. In some cases, the entire surgical imaging device may be moved (e.g., laterally or vertically), which may affect the field of view. Therefore, adjusting the field of view may include providing a control signal to a robotic adjustment system 170 (e.g., a robotic arm) of the surgical imaging system (e.g., via one or more interfaces), wherein the robotic adjustment system 170 is configured to move the surgical imaging device relative to the surgical site based on the control signal. Additionally, or alternatively, an optical and / or digital zoom function of the surgical imaging device (or system) may be used to increase or decrease the magnification, which may also affect the field of view. Therefore, adjusting the field of view may include providing a control signal to a zoom function of the surgical imaging system (e.g., via one or more interfaces 112). Finally, the crop factor of the field of view displayed in the digital view may be adjusted, which may be as a digital zoom and / or as a way to display an off-center portion of the field of view captured by the optical imaging sensor of the surgical imaging device. Therefore, adjusting the field of view may include adjusting a crop factor of image processing performed by the surgical imaging system (e.g., by the system 110), for example, as part of the generation of the digital view. After adjusting the field of view (particularly when a robotic adjustment system is used), the focus of the surgical imaging device can be readjusted to improve image quality after the adjustment of the field of view. For example, the system can be configured to trigger an autofocus function of the surgical imaging system after adjusting the field of view of the surgical imaging device.

[0045] The proposed concept provides a convenient and intuitive way to adjust the field of view to the part of interest to the user. However, some things are only of temporary interest to the surgeon. In addition, during an hour-long surgical procedure, the surgeon may need to occasionally move their eyes away from the main focus of the surgery. In this case, it may not be desirable to adjust the field of view. Therefore, the adjustment of the field of view can be triggered or confirmed by an input device of the surgical imaging system. In various examples, the system can be configured to obtain an input signal from an input device 140, 150, 160 of the surgical imaging system, and trigger the adjustment of the field of view based on a user command for triggering the adjustment of the field of view contained in the input signal. For example, in some examples, the system can be configured to detect that the user's gaze is outside the central part of the field of view, and prompt the user to confirm that the field of view needs to be adjusted, wherein the user command is (e.g., in response to the prompt) after the confirmation. Alternatively, the system can be configured to initiate a determination of the user's gaze based on a user command, and then perform the adjustment after the user triggers the determination of the user's gaze. Since the surgical imaging system is typically a complex system with countless input devices, various of these devices can be used to provide input signals. For example, the system can be configured to obtain an input signal from one of an input modality of a handle 140 of the surgical imaging system (e.g., a button, a knob, a lever, or a touch surface), an input modality of a foot pedal 150 of the surgical imaging system (e.g., a button or a lever), an input modality of the surgical imaging device (e.g., a button, a knob, a touch surface, or a touch screen disposed on the surgical imaging device), a camera-based input device of the surgical imaging system (e.g., imaging sensor data of a user gesture), a depth sensor-based input device of the surgical imaging system (e.g., depth sensor data representing a user gesture), and a voice-based input device 160 of the surgical imaging system (e.g., audio data including a voice version of a user command). The system can be configured to process the corresponding input data to determine (e.g., extract, derive) a user command for triggering an adjustment of the field of view from the input signal.

[0046] To support adjustment of the field of view, the system can provide a visual marker indicating that an adjustment is about to be made and a target (e.g., a central area) of the adjusted field of view (which may typically correspond to a portion of the field of view to which the gaze is directed). As described above, the system can be configured to generate a digital view based on imaging sensor data. The system can be configured to highlight a portion of the field of view to which the gaze is directed in the digital view before adjusting the field of view, and thus highlight the central area of ​​the adjusted field of view. For example, an area of ​​interest that intersects a portion of the field of view can be highlighted. In other words, the system can be configured to highlight an area of ​​interest that intersects a portion of the field of view where the gaze is detected within the field of view before adjusting the field of view. Figure 4An example of such a user using a visual marker to highlight an area of ​​interest is shown. After the highlighted portion of the digital view is displayed, the user can confirm the adjustment through an input signal. In other words, the system can be configured to trigger an adjustment of the field of view after the digital view with the highlighted portion is displayed to the user and the input signal includes a user command indicating confirmation of the adjusted field of view.

[0047] In some other examples, the adjustment can be performed without additional triggering or confirmation by the user. For example, the system can be configured to continuously (and automatically, i.e., without user input) adjust the field of view of the surgical imaging device based on the user's gaze. To avoid sudden and unexpected changes in the field of view, the adjustment can be limited in degree and suddenness. For example, the system can be configured to continuously and gradually adjust the field of view of the surgical imaging device based on the user's gaze, for example, by limiting the maximum change in the field of view per unit time (e.g., in millimeters).

[0048] In various examples, a digital view of the surgical site is created and provided as part of a display signal to a display of a surgical imaging system. To generate the digital view, as previously described, the system can be configured to obtain imaging sensor data showing the field of view of the surgical imaging device from an optical imaging sensor of the surgical imaging device, and to generate a digital view based on the imaging sensor data. In the proposed system, the digital view can represent an adjusted field of view. The digital view can be viewed by a user of the surgical imaging system (e.g., a surgeon). To this end, the display signal can be provided to a display, for example, an auxiliary display arranged at a base unit 105 of a surgical microscope system, an eyepiece display integrated in an eyepiece 125 of a surgical imaging device, or one or both displays of the headgear / head-mounted display 180 described above. Thus, the system can be configured to generate a display signal for a display device of a microscope system, wherein the display signal is based on the digital view. For example, the display signal can be a signal for driving (e.g., controlling) a corresponding display device. For example, the display signal can include video data and / or control instructions for driving the display. For example, the display signal can be provided via one of the one or more interfaces 112 of the system. Thus, the system 110 may include a video interface 112 adapted to provide a display signal to a display 130 of the microscope system 100 .

[0049] In the proposed surgical imaging system, at least one optical imaging sensor may be used to provide the above-mentioned imaging sensor data. Thus, the optical imaging sensor may be part of a surgical imaging device 120 (e.g., a microscope) and may be configured to generate imaging sensor data. For example, at least one optical imaging sensor of the surgical imaging device 120 may include or be an imaging sensor based on an APS (active pixel sensor) or a CCD (charge coupled device). For example, in an APS-based imaging sensor, light is recorded at each pixel using a photodetector and an active amplifier of the pixel. APS-based imaging sensors are typically based on CMOS (complementary metal oxide semiconductor) or S-CMOS (scientific CMOS) technology. In a CCD-based imaging sensor, incident photons are converted into electron charges at the semiconductor-oxide interface and then moved between capacitor bins in the imaging sensor through the circuit of the imaging sensor to perform imaging. The system 110 may be configured to obtain (i.e., receive or read) imaging sensor data from the optical imaging sensor. The imaging sensor data may be obtained by receiving the imaging sensor data from an optical imaging sensor (e.g., via interface 112), reading the imaging sensor data from a memory of the optical imaging sensor (e.g., via interface 112), or reading the imaging sensor data from a storage device 116 of system 110, e.g., after the imaging sensor data has been written to the storage device 116 by the optical imaging sensor or by another system or processor.

[0050] The one or more interfaces 112 of the system 110 may correspond to one or more inputs and / or outputs for receiving and / or sending information within a module, between modules, or between modules of different entities, which information may be a digital (bit) value according to a specified code. For example, the one or more interfaces 112 may include an interface circuit configured to receive and / or send information. The one or more processors 114 of the system 110 may be implemented using one or more processing units, one or more processing devices, any means for processing (e.g., a processor, a computer, or a programmable hardware component (which may work with corresponding adapted software)). In other words, the described functions of the one or more processors 114 may also be implemented in software and then executed on one or more programmable hardware components. Such hardware components may include a general-purpose processor, a digital signal processor (DSP), a microcontroller, etc. The one or more storage devices 116 of the system 110 may include at least one element of a computer-readable storage medium group, such as a magnetic or optical storage medium, such as a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a network memory.

[0051] Further details and aspects of the system and surgical imaging system are described in conjunction with the proposed concept or one or more examples above or below (such as Figures 2 to 5 ) is mentioned in the description related to the present invention. The system and / or surgical microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept, or one or more examples described above or below.

[0052] Figure 2 A flowchart showing an example of a corresponding method for a surgical imaging system, for example, for use with Figure 1a The method includes obtaining 210 an eye tracking sensor signal of an eye tracking system. The method includes determining 220 a gaze of a user of a hand surgical imaging device of the surgical imaging system on a portion of a field of view of the surgical imaging device based on the eye tracking sensor signal. The method includes adjusting 230 the field of view of the surgical imaging device based on the gaze of the user on the portion of the field of view.

[0053] For example, the method can be replaced by Figure 1a to Figure 1b A system for introducing and / or performing surgical imaging in accordance with an embodiment of the present invention. Figure 1a to Figure 1b The introduced features related to the systems and surgical imaging systems may also be included in the corresponding methods.

[0054] Further details and aspects of the method are provided in conjunction with the proposed concept or with one or more examples described above or below (e.g. Figure 1a to Figure 1b , Figures 3 to 5 ) is mentioned in the description related to the invention. The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept, or one or more examples described above or below.

[0055] Two examples of the proposed concept are shown below. Figure 3 An example of adjustment of the field of view based on gaze is shown. Figure 3 , the field of view provided by the surgical imaging device is shown at two points in time - a first field of view 310 before adjustment and a second field of view 320 after adjustment. Overlaid on the first field of view 310, the target 315 of the user / surgeon's gaze is displayed as overlaid on the field of view (visualized by the "eye" symbol). After adjusting the field of view, the adjusted field of view 320 is centered on the previous target 325 of the user's gaze.

[0056] exist Figure 4 An example is shown in where the adjustment of the field of view is visualized. Figure 4 An example of a visual indicator 425 is shown that is used to highlight the area of ​​interest before adjusting the field of view. Figure 3 Similarly, on the left, a first field of view 410 is shown with a user's gaze target 415. On the right, the same field of view 420 is shown with a visual marker 425 highlighting a region of interest that intersects the target 415.

[0057] Various examples of the present disclosure relate to a digital device for controlling visualization of a region of interest (ie, a ROI finder). The proposed ROI finder can be used to change a region of interest displayed in, for example, but not limited to, a digital viewer.

[0058] In other systems, the user is often required to manually and / or deliberately move the microscope to change the ROI that the user or surgeon sees. In contrast, in the proposed concept, the area visualized with a surgical imaging device (e.g., a microscope) via a display (digital viewer, screen, etc.) can automatically change depending on where the user / surgeon is looking in the image (i.e., the current field of view).

[0059] The proposed concept may gain time for the user / surgeon and / or improve patient safety, since the user / surgeon may not need to touch the surgical imaging device / microscope to see the ROI that the user / surgeon wants to see (if the surgeon normally uses a handle to change the ROI). If the surgeon normally uses the interface of the surgical imaging system to change the ROI, the proposed concept may bring an improved concept to the surgeon, since the surgeon may not need to use the interface (or may only use it to confirm or trigger the adjustment of the field of view) to adjust the field of view to see the ROI that the surgeon / user wants to see.

[0060] Further details and aspects of the region of interest (ROI) finder are mentioned in the description related to the proposed concept, or in one or more examples mentioned above or below (e.g., Figure 1a to Figure 2 , Figure 5 ). The ROI finder may include one or more additional optional features corresponding to one or more aspects of the proposed concept, or one or more examples described above or below.

[0061] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0062] Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of the corresponding block or item or feature of the corresponding apparatus.

[0063] Some embodiments relate to a microscope comprising the apparatus of FIGS. 1 to 2. Figure 4 Alternatively, the microscope may be a system as described in one or more of the related descriptions in FIG. Figure 4 Part of or connected to one or more of the related described systems. Figure 5 A schematic diagram of a system 500 configured to perform the methods described herein is shown. System 500 includes a microscope 510 and a computer system 520. Microscope 510 is configured to capture images and is connected to computer system 520. Computer system 520 is configured to perform at least a portion of the methods described herein. Computer system 520 can be configured to execute a machine learning algorithm. Computer system 520 and microscope 510 can be separate entities, but can also be integrated together in a common housing. Computer system 520 can be part of a central processing system of microscope 510 and / or computer system 520 can be part of a subcomponent of microscope 510, such as a sensor, actuator, camera or lighting unit of microscope 510, etc.

[0064] The computer system 520 can be a local computer device (e.g., a personal computer, laptop, tablet, or mobile phone) having one or more processors and one or more storage devices, or it can also be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 520 may include any circuit or combination of circuits. In one embodiment, the computer system 520 may include one or more processors, which may be of any type. As used herein, a processor may refer to any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), such as a computing circuit of a microscope or a microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 520 may be custom circuits, application specific integrated circuits (ASICs), etc., such as, for example, one or more circuits (e.g., communication circuits) for wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems, etc. The computer system 520 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as a main memory in the form of a random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 520.

[0065] Some or all method steps can be performed by (or using) hardware devices, such as, for example, processors, microprocessors, programmable computers or electronic circuits. In some embodiments, such devices can perform one or more of the most important method steps.

[0066] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transient storage medium, such as a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM and EPROM, EEPROM or flash memory, on which are stored electronically readable control signals that cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer readable.

[0067] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0068] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.For example, the program code may be stored on a machine readable carrier.

[0069] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0070] In other words, therefore, one embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0071] Therefore, another embodiment of the invention is a storage medium (or a data carrier, or a computer-readable medium) comprising a computer program stored thereon for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0072] Therefore, another embodiment of the invention is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection, for example via the Internet.

[0073] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0074] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0075] Another embodiment according to the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. For example, the apparatus or system may include a file server for transmitting the computer program to the receiver.

[0076] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0077] Embodiments may be based on the use of machine learning models or machine learning algorithms. Machine learning may refer to algorithms and statistical models that a computer system can use to perform specific tasks without the use of explicit instructions, but rather relying on models and reasoning. For example, in machine learning, data transformations inferred from analysis of historical and / or training data may be used instead of rule-based data transformations. For example, a machine learning model may be used or a machine learning algorithm may be used to analyze the content of an image. In order to analyze the content of an image using a machine learning model, a training image may be used as input and training content information may be used as output to train the machine learning model. By training a machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the image, so that the image content not included in the training data may be recognized using the machine learning model. The same principle may also be used for other categories of sensor data: by training a machine learning model using training sensor data and desired outputs, the machine learning model "learns" the conversion between sensor data and output, which may be used to provide outputs based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine learning model.

[0078] The machine learning model can be trained using training input data. The example described in detail above uses a training method called "supervised learning". In supervised learning, a machine learning model is trained using multiple training samples, each of which may include multiple input data values ​​and multiple expected output values, i.e., each training sample is associated with an expected output value. By specifying the training samples and the expected output values, the machine learning model "learns" which output value to provide based on input samples similar to the samples provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms (such as classification algorithms, regression algorithms, or similarity learning algorithms). When the output is limited to a limited set of values ​​(categorical variables), a classification algorithm can be used, i.e., the input is classified into one of a limited set of values. When the output can have any numerical value (within a range), a regression algorithm can be used. Similarity learning algorithms can be similar to classification and regression algorithms, but are based on learning from examples using a similarity function that measures the similarity or correlation between two objects. In addition to supervised or semi-supervised learning, unsupervised learning can also be used to train machine learning models. In unsupervised learning, (only) input data may be provided, and an unsupervised learning algorithm may be used to find structure in the input data (e.g., find commonalities in the data by grouping or clustering the input data). Clustering is the assignment of input data comprising multiple input values ​​into subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but different from input values ​​included in other clusters.

[0079] Reinforcement learning is the third class of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Based on the actions taken, rewards are calculated. Reinforcement learning is based on training one or more software agents to choose actions that increase the cumulative reward, making the software agent get better at a given task (as evidenced by an increase in reward).

[0080] Additionally, some techniques can be applied to some machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained at least in part using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms, also called representation learning algorithms, can preserve the information in their input but can also transform it in a way that makes it useful, often as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0081] In some examples, anomaly detection (i.e., outlier detection) can be used, the purpose of which is to provide identification of input values ​​that are suspicious by being significantly different from the majority of inputs or training data. In other words, a machine learning model can be trained at least in part using anomaly detection, and / or a machine learning algorithm can include an anomaly detection component.

[0082] In some examples, a machine learning algorithm may use a decision tree as a prediction model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by branches of the decision tree, and output values ​​corresponding to the item may be represented by leaves of the decision tree. Decision trees may support both discrete and continuous values ​​as output values. If discrete values ​​are used, decision trees may be referred to as classification trees, and if continuous values ​​are used, they may be referred to as regression trees.

[0083] Association rules are another technique that can be used with machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. Machine learning algorithms can identify and / or exploit one or more relationship rules that represent knowledge derived from the data. For example, these rules can be used to store, manipulate, or apply knowledge.

[0084] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may generally refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may generally refer to a data structure and / or rule set that represents learned knowledge (e.g., based on training performed by a machine learning algorithm). In an embodiment, using a machine learning algorithm may mean using an underlying machine learning model (or multiple underlying machine learning models). Using a machine learning model may mean that the machine learning model and / or the data structure / rule set that is the machine learning model is trained by a machine learning algorithm.

[0085] For example, a machine learning model can be an artificial neural network (ANN). ANNs are systems inspired by biological neural networks, such as can be found in the retina or the brain. ANNs include a number of interconnected nodes and a number of connections between the nodes, the so-called edges. There are usually three types of nodes, input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node can represent an artificial neuron. Each edge can send information from one node to another node. The output of a node can be defined as a (non-linear) function of its input (e.g., the sum of its inputs). Based on the "weight" of the edge or the node providing the input, the input of the node can be used in the function. The weights of the nodes and / or edges can be adjusted during the learning process. In other words, the training of an artificial neural network can include adjusting the weights of the nodes and / or edges of the artificial neural network, i.e. achieving the desired output for a given input.

[0086] Alternatively, the machine learning model can be a support vector machine, a random forest model or a gradient enhancement model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine can be trained by providing an input with multiple training input values ​​belonging to one of two categories. A support vector machine can be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network can use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics a natural selection process.

[0087] Reference Mark List

[0088] 100 Surgical Imaging System / Surgical Microscope System

[0089] 110 System

[0090] 112 Interface

[0091] 114 Processor

[0092] 116 Storage Devices

[0093] 120 Surgical imaging equipment / microscope

[0094] 125 Eyepiece

[0095] 130 Eye Tracking System

[0096] 140 Handle

[0097] 150 Pedals

[0098] 160 Voice-based input devices

[0099] 170 Robot Arm

[0100] 180 Headset / Head-mounted Display

[0101] 210 Obtain eye tracking sensor signal

[0102] 220 Determine user's gaze

[0103] 230 Adjusting the Field of View

[0104] 310 First Vision

[0105] 315 Target

[0106] 320 Second Vision

[0107] 325 Previous Target

[0108] 410 Vision

[0109] 415 Target

[0110] 420 View

[0111] 425 Visual Markers

[0112] 500 System

[0113] 510 Microscope

[0114] 520 Computer Systems

Claims

1. A system (110; 520) for a surgical imaging system (100; 500), the system comprising one or more processors (114) and one or more storage devices (116), wherein the system is configured to: Obtaining an eye tracking sensor signal from an eye tracking system (130) integrated into an eyepiece (125) of a surgical imaging device, or from an eye tracking system integrated into a head-mounted device (180) of the surgical imaging system; determining a gaze of a user of a surgical imaging device (120; 510) of the surgical imaging system on a portion of a field of view of the surgical imaging device based on the eye tracking sensor signal; and The field of view of the surgical imaging device is adjusted based on the user's gaze on the portion of the field of view. 2 . The system of claim 1 , wherein the system is configured to focus the field of view around the portion of the field of view where the gaze is detected.

3. A system according to any one of claims 1 or 2, wherein the system is configured to obtain an input signal from an input device (140; 150) of the surgical imaging system and trigger adjustment of the field of view based on a user command for triggering adjustment of the field of view contained in the input signal.

4. A system according to claim 3, wherein the system is configured to obtain the input signal from one of an input modality of a handle (140) of the surgical imaging system, an input modality of a foot pedal (150) of the surgical imaging system, an input modality of the surgical imaging device, a camera-based input device of the surgical imaging system, a depth sensor-based input device of the surgical imaging system, and a voice-based input device (160) of the surgical imaging system.

5. The system of any one of claims 1 or 2, wherein the system is configured to continuously adjust the field of view of the surgical imaging device based on the gaze of the user.

6. The system of claim 5, wherein the system is configured to continuously and gradually adjust the field of view of the surgical imaging device based on the gaze of the user.

7. A system according to any one of claims 1 to 6, wherein the system is configured to perform object detection on imaging sensor data representing the field of view of the surgical imaging device to determine one or more regions of interest visible in the field of view and adjust the field of view to the region of interest that intersects the portion of the field of view where the gaze is detected.

8. The system of claim 7, wherein the system is configured to generate a digital view based on the imaging sensor data and to highlight the region of interest that intersects the portion of the field of view within which the gaze is detected before adjusting the field of view.

9. The system of any one of claims 1 to 8, wherein adjusting the field of view comprises one or more of: providing a control signal to a robotic adjustment system (170) of the surgical imaging system, providing a control signal to a zoom function of the surgical imaging system, and adjusting a crop factor of image processing performed by the surgical imaging system.

10. A system according to any one of claims 1 to 9, wherein the system is configured to obtain imaging sensor data showing the field of view of the surgical imaging device from an optical imaging sensor of the surgical imaging device, and to generate a digital view based on the imaging sensor data, wherein the digital view represents an adjusted field of view.

11. A surgical imaging system (100; 500) comprising a surgical imaging device (120; 510), an eye tracking sensor and a system (110; 520) according to any one of claims 1 to 10.

12. The surgical imaging system of claim 11, wherein the surgical imaging system is a surgical microscope system.

13. A method for a surgical imaging system, the method comprising: Obtaining (210) an eye tracking sensor signal from an eye tracking system integrated into an eyepiece (125) of a surgical imaging device, or from an eye tracking system integrated into a head-mounted device (180) of the surgical imaging system; determining (220) a gaze of a user of a surgical imaging device of the surgical imaging system on a portion of a field of view of the surgical imaging device based on the eye tracking sensor signal; and The field of view of the surgical imaging device is adjusted (230) based on the user's gaze at the portion of the field of view.

14. A computer program having a program code for carrying out the method according to claim 13 when the computer program is executed on a processor.

Images