System, method and computer program for a surgical imaging system

Abstract

The embodiments relate to a system (110), a method, and a computer program for a surgical imaging system (100), and a corresponding surgical imaging system including the system for the surgical imaging system. The system (110) is configured to acquire an eye tracking sensor signal from an eye tracking system (130). The system is configured to determine, based on the eye tracking sensor signal, a line of sight of a user of a surgical imaging device (120) of the surgical imaging system relative to a field of view of the surgical imaging device. The system is configured to adjust the field of view of the surgical imaging device based on the user's line of sight relative to the field of view.

Description

[Technical Field]

[0001] The embodiments relate to a system, a method and a computer program for a surgical imaging system and a corresponding surgical imaging system comprising such a system. [Background technology]

[0002] A surgical imaging system is an imaging system, such as a microscope system, designed to be used during a surgical procedure. Such imaging systems often provide high magnification. As a result, the field of view of the surgical imaging device of such a surgical imaging system is often small. During a surgical procedure, a surgeon or their assistant often desires to adjust the field of view, for example, to focus on a different area of ​​the surgical site where the surgeon is operating.

[0003] In some cases, the surgeon may view the surgical site using a head-mounted display, which uses imaging sensor data generated by a surgical imaging device. Examples of such uses of head-mounted displays are shown, for example, in EP 3904947 A1.

[0004] There may be a need for improved concepts for surgical imaging systems that provide improved adjustment of the field of view observed by the surgeon. Summary of the Invention [Means for solving the problem]

[0005] These needs are addressed by the subject matter of the independent claims.

[0006] Various examples of the present disclosure are based on the finding that the field of view of a surgical imaging device, e.g., a microscope, of a surgical microscopy system can be automatically or semi-automatically adjusted by tracking the line of sight of a user of the surgical imaging device. In the proposed concept, the line of sight of the user of the surgical imaging device relative to a field of view is determined, and the field of view is adjusted based on the determined line of sight. Thus, the field of view can be adjusted without the surgeon having to manually adjust the field of view.

[0007] Some aspects of the present disclosure relate to a system for a surgical imaging system. The system includes one or more processors and one or more storage devices. The system is configured to acquire an eye tracking sensor signal from an eye tracking system. The system is configured to determine a line of sight of a user of a surgical imaging device of the surgical imaging system relative to a portion of a field of view of the surgical imaging device based on the eye tracking sensor signal. The system is configured to adjust a field of view of the surgical imaging device based on the user's line of sight relative to the portion of the field of view. Determining the field of view based on the user's line of sight allows the field of view to be adjusted without the surgeon having to manually adjust the field of view.

[0008] In some examples, the system can be configured to center the field of view around a portion of the field of view where the line of sight is detectable, and thus the field of view can be adjusted so that the user has an improved view of the portion of the field of view at which they are fixating.

[0009] In some cases, continuous adjustment of the field of view may be undesirable because the user desires the field of view to remain stationary while the surgeon performs certain portions of a surgical procedure. Instead, the user may trigger the adjustment using an input device. Thus, the system may be configured to receive an input signal from an input device of the surgical imaging system and trigger the adjustment of the field of view based on a user command for triggering the adjustment of the field of view contained in the input signal. This may avoid unintended adjustment of the field of view.

[0010] For example, the system may be configured to obtain an input signal from one of an input modality of a handle of the surgical imaging system, an input modality of a foot pedal of the surgical imaging system, an input modality of a surgical imaging device, a camera-based input device of the surgical imaging system, a depth sensor-based input device of the surgical imaging system, and an audio-based input device of the surgical imaging system. As will be apparent, a surgical imaging system may include many input devices, thereby enabling user commands to be provided with little effort and great flexibility.

[0011] However, in some examples, the system may be configured to continuously adjust the field of view of the surgical imaging device based on the user's line of sight, eliminating the need for an additional trigger and further reducing user discomfort. In particular, the system may be configured to continuously and incrementally adjust the field of view of the surgical imaging device based on the user's line of sight. The incremental adjustment may avoid sudden changes in the field of view that could be disorienting to the user.

[0012] In some examples, adjusting the field of view can employ image analysis techniques to identify likely targets of the user's line of sight. For example, the system can be configured to perform object detection on image sensor data representing the field of view of the surgical imaging device to determine one or more regions of interest that are visible within the field of view. The system can be configured to adjust the field of view to a region of interest that intersects with a portion of the field of view where the line of sight is detectable. This can, for example, adjust the field of view to provide the user with an improved view of the object of interest, thereby improving the selection of the adjusted field of view.

[0013] In general, the proposed concepts can be used with purely optical surgical imaging devices, such as microscopes with an eyepiece that shows the light beam path through the camera. However, various examples of the present disclosure can provide additional functionality when the field of view is shown as a digital view, for example, via an eyepiece display, an auxiliary display (attached to the surgical imaging system), or a head-mounted display. For example, the system may be configured to generate the digital view based on imaging sensor data and highlight regions of interest within the field of view where the line of sight intersects with the portion of the field of view where the line of sight was detected before adjusting the field of view. This can provide visual feedback to the user before the field of view is finally adjusted based on the line of sight.

[0014] Surgical imaging systems are often very complex and versatile systems, offering many functions that can be used to adjust the field of view. Adjusting the field of view may therefore include one or more of providing a control signal to a robotic adjustment system of the surgical imaging system, providing a control signal to a zoom function of the surgical imaging system, and adjusting a crop factor of the image processing performed by the surgical imaging system (thus applying digital zoom). Each of the above-mentioned means (robotic adjustment system, zoom function, and image processing) has advantages such as a high degree of freedom (robotic adjustment system) and adjusting the field of view without requiring movement of the surgical imaging device (zoom function and image processing).

[0015] After certain adjustments, such as applying an optical zoom, changing the working distance, or moving the surgical imaging device laterally, the field of view may be out of focus. In some examples, the system can be configured to trigger an autofocus function of the surgical imaging system after adjusting the field of view of the surgical imaging device. Through the use of the autofocus function, clarity of the field of view on the surgical site can be ensured.

[0016] As outlined above, the proposed concept can be used with digital or hybrid surgical imaging devices capable of presenting a digital view over the surgical site. For example, the system can be configured to obtain imaging sensor data indicative of the field of view of the surgical imaging device from the optical imaging sensor of the surgical imaging device and generate a digital view representing an adjusted field of view based on the imaging sensor data. This can provide additional flexibility with respect to both the display being used (e.g., digital eyepiece, auxiliary display, or head-mounted display) and the image processing being performed to enable an augmented view over the surgical site.

[0017] In general, the user's gaze can be determined via different modalities. For example, the system may be configured to acquire eye tracking sensor signals from an eye tracking system integrated into the eyepiece of the surgical imaging device. This option may be selected if the display being used is an eyepiece display or if the eyepiece (through the surgical imaging device) provides an optical view of the surgical site.

[0018] Alternatively or additionally, the system can be configured to acquire eye tracking sensor signals from an eye tracking system integrated into the headset of the surgical imaging system, this option can be selected when the display being used is a head mounted display.

[0019] One aspect of the present disclosure relates to a surgical imaging system comprising a surgical imaging device, an eye tracking sensor, and the system introduced above. For example, the surgical imaging system may be a surgical microscope system.

[0020] Some aspects of the present disclosure relate to a method for a surgical imaging system, the method including acquiring an eye tracking sensor signal of an eye tracking system, determining a line of sight of a user of a surgical imaging device of the surgical imaging system relative to a field of view portion of the surgical imaging device based on the eye tracking sensor signal, and adjusting a field of view of the surgical imaging device based on the line of sight of the user relative to the field of view portion.

[0021] One aspect of the present disclosure relates to a computer program having a program code for performing the above method when the computer program is run on a processor. [Brief explanation of the drawings]

[0022] Some examples of apparatus and / or methods are now described, by way of example only, and with reference to the accompanying drawings, in which: [Figure 1a] 1 is a block diagram of an example system for a surgical imaging system. [Figure 1b] 1 is a schematic diagram of an example of a surgical imaging system, in particular a surgical microscopy system. [Figure 2] FIG. 1 illustrates a flowchart of an example method for a surgical imaging system. [Figure 3] FIG. 10 illustrates an example of gaze-based adjustment of the field of view. [Figure 4] 10A-10C illustrate examples of visual indicators used to highlight regions of interest before adjusting the field of view. [Figure 5] 1 is a schematic diagram of an example of a system including a surgical imaging device and a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0023] Various examples will now be described more fully with reference to the accompanying drawings, in which the thickness of lines, layers and / or regions may be exaggerated for clarity, and in which several examples are shown.

[0024] 1a illustrates a block diagram of an example system 110 for surgical imaging system 100 (shown in more detail in FIG. 1b). System 110 is a component of surgical imaging system 100 and may be used to control various aspects of surgical imaging system 100. In particular, it may be used to control (and adjust) the field of view provided by surgical imaging device 120 of surgical imaging system 100 through various means, which will be introduced in more detail below. Additionally, system 110 may be configured to control additional aspects of the surgical imaging system, and / or to perform sensor data processing (e.g., image sensor data processing), and / or to provide display signals to various displays of the surgical imaging system.

[0025] Overall, system 110 may be considered a computer system. System 110 includes one or more processors 114 and one or more storage devices 116. Optionally, system 110 further includes one or more interfaces 112. One or more processors 114 are coupled to one or more storage devices 116 and one or more interfaces 112. Overall, the functionality of system 110 may be provided by one or more processors 114 in combination with one or more interfaces 112 (for exchanging data / information with one or more other components of surgical imaging system 100 (and outside of surgical imaging system 100), such as an optical imaging sensor of surgical imaging device 120, an eye tracking system 130, surgical imaging system input devices 140; 150; 160, a robotic coordination system 170, and / or a headset / head-mounted display 180) and one or more storage devices 116 (for storing information, such as machine-readable instructions for a computer program executed by the one or more processors). Overall, the functionality of the one or more processors 114 can be implemented by the one or more processors 114 executing machine-readable instructions. Thus, any feature attributed to the one or more processors 114 can be defined by one or more of the machine-readable instructions. The system 110 can include machine-readable instructions in one or more storage devices 116, for example.

[0026] As outlined above, system 110 is part of a surgical imaging system 100 that includes various components in addition to system 110. For example, surgical imaging system 100 includes surgical imaging device 120 and may include one or more additional components, such as the eye-tracking system 130, input devices 140; 150; 160, robotic coordination system 170, and / or headset / head-mounted display 180, as described above. FIG. 1b shows a schematic diagram of an example of such a surgical imaging system 100, in particular a surgical microscopy system 100. In the following, surgical imaging system 100 may also be referred to as a surgical microscopy system 100, which is a surgical imaging system 100 that includes a (surgical) microscope as surgical imaging device 120. However, the proposed concept is not limited to such an embodiment. Surgical imaging system 100 may be based on various (single or multiple) surgical imaging devices, such as one or more microscopes, one or more endoscopes, one or more radiological imaging devices, one or more surgical tomography devices (such as optical coherence tomography devices), etc. Thus, the surgical imaging system may alternatively be a surgical endoscopy system, a surgical radiography system, or a surgical tomography system, although in the following examples it will be assumed that the surgical imaging device 120 is a (surgical) microscope and the surgical imaging system 100 is a surgical microscope system 100.

[0027] Thus, the surgical imaging system or surgical microscope system 100 can include a microscope 120. Generally, a microscope, such as microscope 120, is an optical instrument suitable for inspecting objects too small to be inspected by the human eye (alone). For example, a microscope can provide optical magnification of a sample. The microscope can be a purely optical microscope (the eyepiece shows an image of light traveling from the surgical site being viewed through the microscope 120), a hybrid (optical and digital) microscope, or a digital microscope (the eyepiece or display shows a digital view of the surgical site). In modern hybrid or digital microscopes, optical magnification is also provided for an optical imaging sensor. Thus, in the latter two cases, the microscope 120 includes an optical imaging sensor coupled to the system 110. The microscope 120 can further include one or more optical magnification components, objective lenses (i.e., lenses), used to expand the field of view on the sample. For example, the surgical imaging device or microscope 120 is often referred to as the “optical carrier” of the surgical imaging system.

[0028] There are various different types of surgical imaging devices. When a surgical imaging device is used in the medical or biological fields, the object viewed through the surgical imaging device may be, for example, a sample of organic tissue located in a Petri dish or present in a portion of a patient's body. In various examples presented herein, the surgical imaging device 120 may be a microscope of a surgical microscope system, i.e., a microscope used during a surgical procedure, such as an oncological surgical procedure, or during tumor surgery. Thus, the object viewed through the surgical imaging device (shown within the field of view of the surgical imaging device) and shown in the digital view generated based on imaging sensor data provided by the (optional) optical imaging sensor may be a sample of the patient's organic tissue, and in particular, a surgical site where a surgeon operates during a surgical procedure. For example, the imaged object, i.e., the surgical site, may be a surgical site in the brain during a neurosurgery procedure. However, the proposed concept is also suitable for other types of surgery, such as cardiac surgery or ophthalmology.

[0029] Generally, a surgical imaging system such as surgical microscopy system 100 is a system that includes a microscope 120, additional components that operate in conjunction with the microscope, such as system 110 (which may be a computer system adapted to control the surgical microscopy system and, for example, process the microscope's imaging sensor data), additional sensors, a display, etc.

[0030] 1b shows a schematic diagram of an example of a surgical imaging system 100, specifically a surgical microscopy system 100 comprising a system 110 and a microscope 120. The surgical microscopy system 100 shown in FIG. 1b includes several optional components, such as a base unit 105 (including system 110) with a (rolling) stand, an eyepiece 125 (e.g., with an eyepiece display) disposed on the microscope 120, an eye-tracking system 130, an auxiliary display disposed on the base unit 105, one or more input devices 140; 150; 160, and a robotic adjustment system 170 shown as an arm 170 (robotic or manual) that holds the microscope 120 in place and is coupled to the base unit 105 and the microscope 120. In general, these optional and non-optional components may be coupled to a system 110 that is configurable to control and / or interact with each component. In FIG. 1b, the connection between the headset 180 and the system 110 and the connection between the two examples of the eye tracking system 130 (either as part of the headset 180 or as part of the eyepiece 125) are shown as wired connections. In other words, the headset 180 and / or the eye tracking system 130 can be connected to the system 110 using a wired connection, i.e., they can be wirebound devices. Accordingly, the interface 112 can be configured to communicate with the headset 180 and / or the eye tracking system 130 using a wired connection. However, the proposed concept is not limited to these examples. Alternatively, the headset 180 and / or the eye tracking system can be a wireless device, i.e., a wireless headset 180 with the eye tracking system 130, i.e., they can be connected to the system 110 using a wireless (data) connection. Accordingly, the interface 112 can be configured to communicate with the headset 180 and / or the eye tracking system 130 using a wireless (data) connection.

[0031] In the proposed concept, the system 110 is used, among other things, to control the field of view of the surgical imaging device 120. This is done by tracking the gaze of a user (e.g., a surgeon or an assistant responsible for controlling the surgical imaging system) and adjusting the field of view accordingly. For example, the system 110 is configured to acquire an eye tracking sensor signal from the eye tracking system 130. The system 110 is configured to determine, based on the eye tracking sensor signal, the line of sight of the user of the surgical imaging device 120 of the surgical imaging system relative to a portion of the field of view of the surgical imaging device. The system 110 is configured to adjust the field of view of the surgical imaging device based on the user's line of sight relative to the portion of the field of view.

[0032] To perform gaze-based adjustments of the field of view, the system uses eye tracking sensor signals from the eye tracking system 130. Overall, the eye tracking sensor signals can indicate the user's gaze, i.e., the direction of the user's field of view. For example, the eye tracking sensor signals can indicate where the user is looking, e.g., at a portion of the display or at a central or peripheral region of the eyepiece. The implementation of the eye tracking sensor signals can depend on the eye tracking sensor being used. For example, the system can be configured to acquire eye tracking sensor signals from an eye tracking system integrated into the eyepiece 125 of the surgical imaging device. In this case, the eye tracking sensor signals can indicate whether the user is observing a central portion of the field of view provided by the eyepiece or a peripheral portion of the field of view provided by the eyepiece (and its direction, e.g., right, left, up, down, top right, bottom left, etc.). Alternatively, the system 110 can be configured to acquire eye tracking sensor signals from an eye tracking sensor that is coupled to a display, e.g., the auxiliary display mentioned above, or that is part of a headset / head-mounted display. For example, the system can be configured to acquire eye tracking sensor signals from an eye tracking system integrated into the headset 180 of the surgical imaging system. In these cases, the eye tracking sensor signal can indicate which part of the display the user is looking at (e.g., focusing on).

[0033] The system then uses the eye tracking sensor signal to determine the user's gaze, and in particular, the portion of the field of view to which the gaze is directed. For example, if the eye tracking sensor signal indicates the direction of the user's field of view (e.g., a direction toward the center or a direction toward the periphery (and peripheral direction)), the system may be configured to estimate the portion of the field of view to which the gaze is directed from the direction of the field of view. Alternatively, if the eye tracking sensor signal indicates a portion of a display at which the user is looking (e.g., in a head-mounted display or on an auxiliary display as described above), the system may be configured to convert the portion of the display into a corresponding portion of the field of view. In both cases, the system determines the portion of the field of view currently provided to the user at which the user is looking. In the present context, the field of view of the surgical imaging device is the field of view provided to the user by the surgical imaging device, e.g., the field of view the user observes when looking through the eyepieces or when viewing the display of the surgical imaging device. In some examples, when multiple displays (e.g., multiple head-mounted displays and / or multiple digital eyepieces) are used, the field of view may differ for different users of the surgical imaging system, for example, through the use of user-specific image cropping and / or the application of user-specific digital zoom.

[0034] In previous examples, the field of view portion was used without reference to a particular feature of interest; instead, only the field of view portion was referenced. However, in some cases, additional image processing can be used to identify one or more particular regions of interest (e.g., anatomical features of interest, or more generally, objects of interest (such as surgical markers)) within the field of view and take these into account when adjusting the field of view of the surgical imaging device. For example, the system can be configured to perform (machine learning-based) object detection (or more generally, image segmentation) on the imaging sensor data representing the field of view of the surgical imaging device to determine one or more regions of interest (e.g., anatomical or non-anatomical features or objects of interest) that are visible within the field of view. For example, the imaging sensor data can be analyzed to determine and distinguish features within the imaging sensor data, such as anatomical features (e.g., blood vessels, tumors, branches, portions of tissue, etc.) or non-anatomical features (e.g., markers, clips, or stitches). To this end, one or both of the following machine learning-based techniques can be used: image segmentation and object detection. The system can be configured to perform image segmentation and / or object detection to determine features, and therefore regions of interest, in the imaging sensor data. In object detection, the locations of one or more predefined objects (i.e., the objects for which the respective machine learning models are trained) in the imaging sensor data are output by the machine learning model, along with a classification of the objects (if the machine learning models are trained to detect multiple different types of objects). Typically, the locations of the one or more predefined objects are provided as a bounding box, i.e., a set of locations forming a rectangle surrounding each object being detected. In image segmentation, the locations of features (i.e., portions with similar attributes, e.g., portions of the imaging sensor data that belong to the same object) are output by the machine learning model. Typically, the locations of the features are provided as a pixel mask, i.e., the locations of pixels belonging to the feature are output for each feature.

[0035] For both object detection and image segmentation, machine learning models trained to perform the respective tasks are used. For example, to train a machine learning model trained to perform object detection, a plurality or samples of imaging sensor data can be provided as training input samples, a corresponding list of bounding box coordinates can be provided as a desired output of the training, and a supervised learning-based training algorithm is used to perform the training using the plurality of training input samples and the corresponding desired output. For example, to train a machine learning model trained to perform image segmentation, a plurality or samples of imaging sensor data can be provided as training input samples, and a corresponding pixel mask can be provided as a desired output of the training, and a supervised learning-based training algorithm is used to perform the training using the plurality of training input samples and the corresponding desired output. In some examples, the same machine learning model can be used to perform both object detection and image segmentation. In this case, the two types of desired output described above are available in parallel during training, and the machine learning model is trained to output both a bounding box and a pixel mask. Thus, the machine learning model can be used to determine one or more potential features of interest, and thus one or more regions of interest.

[0036] These one or more regions of interest may then be used to support the determination of a portion of the field of view in which the gaze is detected. For example, the system may be configured to correlate the user's gaze with one or more (potential) regions of interest and determine a portion of the field of view that intersects with one of the one or more regions of interest. In effect, the system may be configured to adjust the field of view to the region of interest that intersects the portion of the field of view in which the gaze is detected.

[0037] In this case, the system is configured to adjust the field of view of the surgical imaging device based on the user's line of sight relative to the field of view portion. For example, the system can be configured to adjust the field of view based on the field of view portion to which the line of sight is directed. As the user observes the field of view portion, the field of view can be adjusted (e.g., moved) so that the field of view portion to which the line of sight is directed moves toward the center of the field of view. An example of such an adjustment is shown, for example, in FIG. 3. The system can be configured to center the field of view around the field of view portion to which the line of sight was detected. In other words, the field of view can be adjusted so that the field of view portion to which the line of sight was detected before the adjustment is at the center (or closer to the center) of the field of view after the adjustment.

[0038] Generally, various means exist for adjusting the field of view. In some cases, the entire surgical imaging device can be moved (e.g., laterally or vertically), thereby affecting the field of view. Adjusting the field of view, therefore, can include providing a control signal (e.g., via one or more interfaces) to a robotic adjustment system 170 (e.g., a robotic arm) of the surgical imaging system, which is configured to move the surgical imaging device relative to the surgical site based on the control signal. Additionally or alternatively, an optical and / or digital zoom function of the surgical imaging device (or system) can be used to increase or decrease magnification, which also affects the field of view. Adjusting the field of view, therefore, can include providing a control signal to a zoom function of the surgical imaging system (e.g., via one or more interfaces 112). Finally, the crop factor of the field of view shown in the digital view can be adjusted, which can function as a digital zoom and / or a means of showing off-center portions of the field of view captured by the optical imaging sensor of the surgical imaging device. Thus, adjusting the field of view may include, for example, adjusting a crop factor of image processing performed by the surgical imaging system (e.g., by system 110) as part of generating the digital view. After the field of view is adjusted (particularly if a robotic adjustment system is used), the focus of the surgical imaging device can be readjusted to improve image quality after adjusting the field of view. For example, the system can be configured to trigger an autofocus function of the surgical imaging system after adjusting the field of view of the surgical imaging device.

[0039] The proposed concept provides a convenient and intuitive means for adjusting the field of view to a portion of the user's interest. However, some things are only of temporary interest to the surgeon. Furthermore, during a surgical procedure lasting an hour or more, the surgeon may need to occasionally look away from the primary focus of the surgical procedure. Adjusting the field of view in such situations may be undesirable. Therefore, the field of view adjustment can be triggered or confirmed via an input device of the surgical imaging system. In various examples, the system can be configured to receive an input signal from the input device 140, 150, 160 of the surgical imaging system and trigger the field of view adjustment based on a user command for triggering the field of view adjustment contained in the input signal. For example, in some examples, the system can be configured to detect that the user's gaze is outside the central portion of the field of view and output a prompt to the user to confirm their desire to adjust the field of view, with the confirmation occurring after the user command (e.g., in response to the prompt). Alternatively, the system may be configured to initiate a determination of the user's gaze based on the user command and then perform the adjustment after the user-triggered determination of the user's gaze. Because surgical imaging systems are often complex systems with a myriad of input devices, a variety of these devices can be used to provide input signals. For example, the system can be configured to obtain input signals from one of an input modality (e.g., a button, knob, control stick, or touch surface) of the handle 140 of the surgical imaging system, an input modality (e.g., a button or control stick) of the foot pedal 150 of the surgical imaging system, an input modality (e.g., a button, knob, touch surface, or touch screen disposed on the surgical imaging device), a camera-based input device of the surgical imaging system (e.g., image sensor data of user gestures), a depth sensor-based input device of the surgical imaging system (e.g., depth sensor data representative of user gestures), and a voice-based input device 160 of the surgical imaging system (e.g., audio data including spoken versions of user commands).The system can be configured to process each input data to determine (eg, extract, derive) a user command from the input signal to trigger an adjustment of the field of view.

[0040] To support the adjustment of the field of view, the system can provide a visual marker indicating that an adjustment is about to be made and a target (e.g., a central region) of the adjusted field of view (which may typically correspond to the portion of the field of view toward which the gaze is directed). As outlined above, the system can be configured to generate a digital view based on imaging sensor data. Before adjusting the field of view, the system can be configured to highlight in the digital view the portion of the field of view toward which the gaze is directed, and thus the central region of the adjusted field of view. For example, a region of interest that intersects with the portion of the field of view can be highlighted. In other words, before adjusting the field of view, the system can be configured to highlight a region of interest that intersects with the portion of the field of view in which the gaze is detected. An example of such a user visual marker for highlighting a region of interest is shown in FIG. 4. After the highlighted portion of the digital view is shown, the user can confirm the adjustment via an input signal. In other words, the system can be configured to trigger the adjustment of the field of view after the digital view with the highlighted portion is shown to the user and a user command indicating confirmation of the adjusted field of view is given as an input signal.

[0041] In some other examples, the adjustment can be performed without additional triggering or confirmation by the user. For example, the system may be configured to continuously (and automatically, i.e., without user input) adjust the field of view of the surgical imaging device based on the user's line of sight. To avoid sudden, unintended changes in the field of view, the adjustment can be limited in range and abruptness. For example, the system can be configured to continuously and incrementally adjust the field of view of the surgical imaging device based on the user's line of sight, e.g., by limiting the maximum amount of change in the field of view (e.g., in millimeters) per unit time.

[0042] In various examples, a digital view of the surgical site is created and provided to the display of the surgical imaging system as part of a display signal. To generate the digital view, as described above, the system can be configured to obtain image sensor data indicative of the field of view of the surgical imaging device from an optical imaging sensor of the surgical imaging device and generate the digital view based on the image sensor data. In the proposed system, the digital view may represent an adjusted field of view. The digital view may be viewed by a user of the surgical imaging system, e.g., a surgeon. To this end, the display signal can be provided to a display, e.g., an auxiliary display located in the base unit 105 of the surgical microscope system, an eyepiece display integrated within the eyepiece 125 of the surgical imaging device, or one or two displays of the headset / head-mounted display 180 described above. Thus, the system can be configured to generate display signals based on the digital view for display devices of the microscope system. For example, the display signal can be a signal for driving (e.g., controlling) the respective display device. For example, the display signal can include video data and / or control instructions for driving the display. For example, the display signal can be provided via one of the one or more interfaces 112 of the system. Accordingly, the system 110 may include a video interface 112 suitable for providing a display signal to a display of the microscope system 100 .

[0043] In the proposed surgical imaging system, at least one optical imaging sensor can be used to provide the image sensor data described above. Accordingly, the optical imaging sensor, which can be part of the surgical imaging device 120 (e.g., a microscope), can be configured to generate the image sensor data. For example, the at least one optical imaging sensor of the surgical imaging device 120 can include or be an active pixel sensor (APS) or charge-coupled device (CCD)-based imaging sensor. For example, in an APS-based imaging sensor, light is recorded at each pixel using a pixel photodetector and an active amplifier. APS-based imaging sensors are often based on complementary metal-oxide-semiconductor (CMOS) or scientific CMOS (S-CMOS) technology. In a CCD-based imaging sensor, imaging is performed by converting incident photons into electronic charges at a semiconductor-oxide interface, which are then transferred between capacitive bins within the imaging sensor by the imaging sensor's circuitry. System 110 can be configured to acquire (i.e., receive or read) imaging sensor data from the optical imaging sensor. The imaging sensor data can be acquired by receiving the imaging sensor data from the optical imaging sensor (e.g., via interface 112), by reading the imaging sensor data from a memory of the optical imaging sensor (e.g., via interface 112), or by reading the imaging sensor data from storage device 116 of system 110, for example, after the imaging sensor data has been written to storage device 116 by the optical imaging sensor or by another system or processor.

[0044] The one or more interfaces 112 of the system 110 can correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values, according to a specified code within a module, between modules, or between modules of different entities. For example, the one or more interfaces 112 can comprise interface circuits configured to receive and / or transmit information. The one or more processors 114 of the system 110 can be implemented using any means for processing, such as one or more processing units, one or more processing devices, processors, computers, or programmable hardware components operable with appropriately adapted software. In other words, the described functionality of the one or more processors 114 can be implemented as software running on one or more programmable hardware components in this case. Such hardware components can comprise general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. The one or more storage devices 116 of the system 110 may comprise at least one element of a group of computer-readable storage media, such as a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or network storage.

[0045] Further details and aspects of the system and surgical imaging system are mentioned in relation to the proposed concept or one or more examples above or below (e.g., FIGS. 2-5). The system and / or surgical microscopy system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above or below.

[0046] 2 illustrates a flowchart of an example of a corresponding method for a surgical imaging system, such as the surgical imaging system 100 illustrated in connection with FIG. 1a and / or FIG. 1b. The method includes acquiring 210 an eye tracking sensor signal of an eye tracking system. The method includes determining 220 a line of sight of a user of a surgical imaging device of the surgical imaging system relative to a field of view portion of the surgical imaging device based on the eye tracking sensor signal. The method includes adjusting 230 a field of view of the surgical imaging device based on the user's line of sight relative to the field of view portion.

[0047] For example, the method can be performed by a system and / or surgical imaging system introduced in connection with one of Figures 1a-1b. Features introduced in connection with the system and surgical imaging system of Figures 1a-1b can also be included in the corresponding method.

[0048] Further details and aspects of the method are mentioned in relation to the proposed concept or one or more examples above or below (e.g., Figures 1a-1b, 3-5). The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above or below.

[0049] Below, two examples of the proposed concept are presented. Figure 3 shows an example of gaze-based adjustment of the field of view. In Figure 3, the field of view provided by the surgical imaging device is shown at two points in time: a first field of view 310 before adjustment and a second field of view 320 after adjustment. Overlaid on the first field of view 310 is a target 315 of the user / surgeon's gaze (visualized by an "eye" symbol) superimposed on the first field of view. After adjustment of the field of view, the adjusted field of view 320 is centered on the previous target 325 of the user's gaze.

[0050] An example of how the field of view adjustment is visualized is shown in Figure 4. Figure 4 shows an example of a visual indicator 425 used to highlight an area of ​​interest before adjusting the field of view. Similar to Figure 3, on the left side, a first field of view 410 is shown with a target 415 for the user's gaze. On the right side, the same field of view 420 is shown with a visual marker 425 highlighting the area of ​​interest that intersects with the target 415.

[0051] Various examples of the present disclosure relate to a digital device for controlling the visualization of a region of interest (i.e., a ROI finder). The proposed ROI finder can be used, for example, but not limited to, to change the region of interest displayed in a digital viewer.

[0052] In other systems, users often have to manually and / or intentionally move the microscope to change the ROI the user or surgeon sees. In contrast, in the proposed concept, the area visualized by the surgical imaging device (e.g., microscope) via a display (digital viewer, screen, etc.) can be automatically changed according to where the user / surgeon is looking in the image (i.e., current field of view).

[0053] The proposed concept may result in a gain in user / surgeon time and / or increased patient safety, as the user / surgeon does not need to touch the surgical imaging device / microscope to observe the ROI they wish to observe (when surgeons normally use a handle to change the ROI).The proposed concept may result in an improved concept for surgeons, as the surgeon does not need to use the mouthpiece of the surgical imaging system to adjust the field of view (or only use the mouthpiece to confirm or trigger the adjustment of the field of view) to observe the ROI they wish to observe, as they normally do when using the mouthpiece of the surgical imaging system to change the ROI.

[0054] Further details and aspects of the ROI finder are mentioned in relation to the proposed concept or one or more examples above or below (e.g., FIGS. 1a-2, 5). The ROI finder may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0055] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0056] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0057] Some embodiments relate to a microscope including a system such as that described in connection with one or more of FIGS. 1-4. Alternatively, the microscope may be part of a system such as that described in connection with one or more of FIGS. 1-4 or may be connected to a system such as that described in connection with one or more of FIGS. 1-4. FIG. 5 shows a schematic diagram of a system 500 configured to perform the methods described herein. The system 500 includes a microscope 510 and a computer system 520. The microscope 510 is configured to capture images and is connected to the computer system 520. The computer system 520 is configured to perform at least a portion of the methods described herein. The computer system 520 may be configured to execute a machine learning algorithm. The computer system 520 and the microscope 510 may be separate entities or may be integrated within a common housing. The computer system 520 may be part of a central processing system of the microscope 510 and / or part of a subsidiary component of the microscope 510, such as a sensor, actor, camera, or lighting unit of the microscope 510.

[0058] The computer system 520 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed across various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 520 may include any circuit or combination of circuits. In one embodiment, the computer system 520 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuitry that may be included in computer system 520 may be custom circuitry, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 520 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.Computer system 520 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touch screen, voice recognition device, or any other device that allows a user of the system to input information to and receive information from computer system 520.

[0059] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry. In some embodiments, any one or more of the critical steps may be performed by such an apparatus.

[0060] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed by a non-transitory storage medium, such as a digital storage medium, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Therefore, the digital storage medium may be computer-readable.

[0061] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

[0062] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code that is operative to perform any of the methods when the computer program product is run on a computer, and that may be stored, for example, on a machine-readable carrier.

[0063] Further embodiments comprise the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0064] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein when the computer program runs on a computer.

[0065] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a recording medium.

[0066] A further embodiment of the present invention is, therefore, a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0067] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0068] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0069] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0070] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

[0071] Embodiments may be based on the use of machine learning models or algorithms. Instead of relying on models and inference, machine learning may refer to algorithms and statistical models that a computer system may use to perform a particular task without explicit instructions. For example, machine learning may use data transformations inferred from an analysis of past data and / or training data instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or algorithm. For a machine learning model to analyze image content, the machine learning model may be trained using training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize image content, such that image content not included in the training data can be recognized using the machine learning model. The same principle may be used for other types of sensor data as well: by training the machine learning model with training sensor data and a desired output, the machine learning model "learns" a transformation between sensor data and output, which can be used to provide an output based on the non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata and / or image data) may be pre-processed to obtain feature vectors that are used as input to machine learning models.

[0072] A machine learning model may be trained using training input data. The above example uses a training method called "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, where each sample may include multiple input data values ​​and multiple desired output values, i.e., each training sample is associated with a desired output value. By specifying both the training samples and the desired output values, the machine learning model "learns" during training which output value to provide based on input samples similar to the provided sample. In addition to supervised learning, semi-supervised learning may also be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). A classification algorithm may be used when the output is restricted to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. A regression algorithm may be used when the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two objects are. In addition to supervised or semi-supervised learning, unsupervised learning may also be used to train machine learning models. In unsupervised learning, input data may be provided (only), and unsupervised learning algorithms may be used to find structure in the input data (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data containing multiple input values ​​into multiple subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but dissimilar to input values ​​contained in another cluster.

[0073] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as "software agents") are trained to take actions in their surroundings. Based on the actions taken, rewards are calculated. Reinforcement learning is based on training one or more software agents to select actions that increase cumulative rewards (as manifested by increasing rewards), resulting in the software agents becoming better at a given task.

[0074] Furthermore, some techniques may be applied to parts of the machine learning algorithm. For example, feature representation learning may be used. In other words, a machine learning model may be trained at least in part using feature representation learning, and / or a machine learning algorithm may include a feature representation learning component. A feature representation learning algorithm, which may be referred to as a representation learning algorithm, may not only preserve information in its input, but may also transform the information to make it useful, often as a preprocessing step before performing classification or prediction. Feature representation learning may be based on, for example, principal component analysis or cluster analysis.

[0075] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide identification of input values ​​that raise suspicion by differing significantly from the majority of the input or training data. In other words, a machine learning model may be trained at least in part with anomaly detection and / or a machine learning algorithm may include an anomaly detection component.

[0076] In some examples, a machine learning algorithm may use a decision tree as a predictive model. In other words, the machine learning model may be based on a decision tree. In a decision tree, an observation about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to the item may be represented by a leaf of the decision tree. The decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0077] Association rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. A machine learning algorithm can identify and / or utilize one or more association rules that represent knowledge derived from the data. These rules can be used, for example, to store, manipulate, or apply the knowledge.

[0078] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a data structure and / or a set of rules that represent learned knowledge (e.g., based on training performed by a machine learning algorithm). In embodiments, the use of a machine learning algorithm may refer to the use of an underlying machine learning model (or underlying machine learning models). The use of a machine learning model may refer to the machine learning model and / or the set of data structures / rules that are the machine learning model being trained by a machine learning algorithm.

[0079] For example, the machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or brain. An ANN includes multiple interconnected nodes and multiple junctions, or edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (nonlinear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weights" of the edges or nodes that provide the input. The weights of the nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0080] Alternatively, the machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing input with multiple training input values ​​that belong to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model may be based on a genetic algorithm, a search algorithm and a heuristic method that mimics the process of natural selection. [Explanation of symbols]

[0081] 100 Surgical Imaging System / Surgical Microscope System 110 System 112 Interface 114 processors 116 Storage Devices 120 Surgical imaging devices / microscopes 125 eyepiece 130 Eye Tracking System 140 Handle 150 Foot Pedal 160 Voice-based Input Devices 170 Robot Arm 180 Headset / Head Mounted Display 210 Acquire eye tracking sensor signal 220 Determine the user's gaze 230 Adjusting the Field of View 310 First View 315 Target 320 Second Vision 325 and earlier targets 410 field of view 415 Target 420 field of view 425 Visual Markers 500 Systems 510 Microscope 520 Computer Systems

Claims

1. A system (110; 520) for a surgical imaging system (100; 500), said system comprising one or more processors (114) and one or more storage devices (116), said system comprising: acquiring an eye tracking sensor signal from an eye tracking system (130) integrated into an eyepiece (125) of a surgical imaging device or from an eye tracking system integrated into a headset (180) of said surgical imaging system; determining a line of sight of a user of a surgical imaging device (120; 510) of the surgical imaging system relative to a field of view portion of the surgical imaging device based on the eye tracking sensor signal; adjusting the field of view of the surgical imaging device based on the user's line of sight relative to the field of view portion; The system is configured as follows:

2. the system is configured to center the field of view around a portion of the field of view where the line of sight is detected. The system of claim 1 .

3. the system is configured to receive an input signal from an input device (140; 150) of the surgical imaging system and trigger an adjustment of the field of view based on a user command for triggering an adjustment of the field of view contained in the input signal.

3. The system according to claim 1 or 2.

4. the system is configured to obtain the input signal from one of an input modality of a handle (140) of the surgical imaging system, an input modality of a foot pedal (150) of the surgical imaging system, an input modality of the surgical imaging device, a camera-based input device of the surgical imaging system, a depth sensor-based input device of the surgical imaging system, and an audio-based input device (160) of the surgical imaging system; The system of claim 3.

5. the system is configured to continuously adjust a field of view of the surgical imaging device based on the user's line of sight.

3. The system according to claim 1 or 2.

6. the system is configured to continuously and incrementally adjust a field of view of the surgical imaging device based on a line of sight of the user. The system of claim 5.

7. the system is configured to perform object detection on image sensor data representing a field of view of the surgical imaging device to determine one or more regions of interest visible within the field of view, and adjust the field of view to a region of interest that intersects a portion of the field of view where a line of sight is detected. A system according to any one of claims 1 to 6.

8. the system is configured to generate a digital view based on the imaging sensor data and, before adjusting the field of view, highlight a region of interest within the field of view that intersects with a portion of the field of view where the line of sight is detected. The system of claim 7.

9. adjusting the field of view includes one or more of providing a control signal to a robotic adjustment system (170) of the surgical imaging system, providing a control signal to a zoom function of the surgical imaging system, and adjusting a crop factor of image processing performed by the surgical imaging system; A system according to any one of claims 1 to 8.

10. the system is configured to obtain image sensor data indicative of a field of view of the surgical imaging device from an optical imaging sensor of the surgical imaging device, and to generate a digital view representing an adjusted field of view based on the image sensor data. A system according to any one of claims 1 to 9.

11. A surgical imaging system (100; 500) comprising: The surgical imaging system (100; 500) comprises a surgical imaging device (120; 510), an eye tracking sensor and a system (110; 520) according to any one of claims 1 to 10. Surgical imaging systems (100; 500).

12. the surgical imaging system is a surgical microscope system. The surgical imaging system of claim 11.

13. 1. A method for a surgical imaging system, the method comprising: acquiring (210) an eye tracking sensor signal from an eye tracking system integrated into an eyepiece (125) of a surgical imaging device or from an eye tracking system integrated into a headset (180) of said surgical imaging system; determining (220) a line of sight of a user of a surgical imaging device of the surgical imaging system relative to a field of view portion of the surgical imaging device based on the eye tracking sensor signal; adjusting (230) the field of view of the surgical imaging device based on the user's line of sight relative to the field of view portion; A method comprising:

14. A computer program comprising: The computer program has a program code for performing the method of claim 13 when the computer program is run on a processor. Computer program.

Images