Systems, methods, and computer programs for a surgical imaging system and a surgical imaging system

Abstract

Examples relate to a system (110), a method and a computer program for a surgical imaging system (100), and to a surgical imaging system (100) comprising such a system (110). The system (110) is configured to obtain an image data stream of an imaging device (120, 150) of the surgical imaging system; process the image data stream to track a position of a portion of a surgical tool; determine, based on a change in the position of the portion of the surgical tool, a position of a digital pointing indicator relative to a field of view of the surgical imaging device (120) of the surgical imaging system showing a surgical site; generate an overlay comprising the digital pointing indicator to be overlaid on the field of view of the surgical imaging device; provide the overlay with the digital pointing indicator for a display device (130a, 130b) of the surgical imaging system; obtain a voice command of a user of the surgical imaging system, the voice command comprising an instruction for performing a spatial positioning function; and trigger, based on the voice command, the execution of the spatial positioning function in accordance with the position of the digital pointing indicator relative to the field of view of the surgical imaging device.

Description

Technical Field

[0001] Examples relate to systems, methods, and computer programs for use in surgical imaging systems, and to surgical imaging systems that include such systems. Background Technology

[0002] Currently, surgical microscope systems require surgeons to manually control focusing and other functions. For this, surgeons typically use additional tools such as foot pedals, touchscreens, or even mouth controllers. This process demands significant attention and coordination, which reduces the surgeon's ability to focus on the surgical procedure itself. The process can be considered cumbersome and potentially disruptive to the surgical workflow, as the surgeon must frequently navigate the entire image and make iterative adjustments to achieve optimal focusing. Additionally, selecting specific areas for manipulation (such as optical coherence tomography (OCT) scans) involves the use of the same or similar manual controls. Surgeons use manual controls to position the cursor and confirm actions, which can add complexity and reduce efficiency during procedures.

[0003] It may be desirable to provide an improved concept for controlling surgical imaging systems. Summary of the Invention

[0004] This expectation is addressed through the subject matter of the independent claims.

[0005] The various examples disclosed herein are based on the finding that during surgery, the surgeon must be close to the surgical site. When operating on the surgical site, the surgeon is equipped with surgical instruments such as scalpels, clamps, or suction devices. Such instruments (especially if they are elongated) are not only used to perform surgical procedures but can also be directed to different features within the surgical site. In the proposed concept, this mechanism is used to control a cursor superimposed on a digital view of the surgical site, allowing the surgeon to visually place the cursor within the surgical site at the location where spatial positioning functions (such as focusing operations or OCT scans) will be performed. Using voice commands, the spatial positioning function can then be triggered at the location corresponding to the cursor. Thus, spatial positioning functions can be triggered without the need for foot pedals, touchscreens, or mouth controllers, without distracting the surgeon from the surgical procedure.

[0006] Some aspects of this disclosure relate to a system for a surgical imaging system. The system includes one or more processors and one or more storage devices. The system is configured to acquire an image data stream from an imaging device of the surgical imaging system (e.g., from the imaging device). The system is configured to process the image data stream to track the position of a portion of a surgical instrument. The system is configured to determine the position of a digital pointing indicator relative to the field of view of the surgical imaging device of the surgical imaging system, showing the surgical site, based on changes in the position of that portion of the surgical instrument. The system is configured to generate an overlay map including the digital pointing indicator to be overlaid on the field of view of the surgical imaging device. The system is configured to provide the overlay map with the digital pointing indicator to a display device of the surgical imaging system. The system is configured to acquire (e.g., from) a voice command from a user of the surgical imaging system, wherein the voice command includes instructions for performing a spatial positioning function. The system is configured to trigger the execution of the spatial positioning function based on the voice command, according to the position of the digital pointing indicator relative to the field of view of the surgical imaging device. Thus, the spatial positioning function can be triggered without the need for a foot pedal, touchscreen, or mouth controller, without distracting the surgeon from the surgical procedure.

[0007] Generally, the proposed concept applies to both fully digital surgical imaging devices (e.g., surgical microscopes or surgical endoscopes) and hybrid surgical imaging devices. In a fully digital surgical imaging device, or in a hybrid surgical imaging device operating in a fully digital mode, image data from the surgical imaging device is used to generate a view of the surgical site. Therefore, the system can be configured to: acquire image data from the surgical imaging device; and generate a view of the surgical site with an overlay image, and provide this view of the surgical site with the overlay image to a display device of the surgical imaging system. In this way, the overlay image can provide a digital view of the surgical site, which offers additional degrees of freedom regarding the display device being used compared to a hybrid microscope where the overlay image is superimposed on an optical view of the surgical site.

[0008] In many cases, such as in fully digital or hybrid surgical imaging systems, image data from the surgical imaging device is used to generate, or can be used to generate, a view of the surgical site. This image data can also be used to track surgical tools. Therefore, the system can be configured to obtain image data streams from the surgical imaging device of the surgical imaging system. In this way, additional hardware (e.g., a camera) may not be required to track surgical tools.

[0009] Alternatively or additionally, the system can be configured to acquire image data streams from the panoramic camera of the surgical imaging system. Therefore, a larger field of view can be tracked at the expense of an additional camera.

[0010] During surgical procedures, various spatial positioning operations are performed by surgical imaging systems. For example, a surgical imaging system may perform localized illumination of a specific area of ​​the surgical site, focusing on a specific point within the surgical site, capturing an image of the region of interest within the surgical site, setting digital markers for the field of view based on commands, or performing a scan using a measuring tool such as an OCT scanner. Therefore, spatial positioning functions can be one of the following: focusing functions, functions for setting markers, functions for illuminating a portion of the surgical site / field of view, functions for changing the field of view, or functions for performing measurements using a measuring device. The proposed concept is applicable to such spatial positioning functions.

[0011] While some spatial positioning functions (such as focusing, setting markers, setting a new center point in the field of view, or performing measurements) are performed at a specific point in the surgical site / field of view, others (such as illuminating a portion of the surgical site or defining the extent of a new field of view) are performed over an area of ​​the surgical site / field of view. In the proposed concept, an area can be selected by setting two points (defining two corners of the area) and using the orientation of the field of view to infer the remaining corners of that area. For example, the system can be configured to determine two positions of a digital pointing indicator relative to the field of view of the surgical imaging device of the surgical imaging system if a voice command instructs that a spatial positioning function be performed in a certain area. The system can be configured to determine the area based on the two positions and the orientation of the field of view of the surgical imaging device. The system can be configured to trigger the execution of a spatial positioning function in that area. In this way, the surgeon can intuitively trigger area-based spatial positioning functions.

[0012] While surgeons can select locations by explicitly pointing within the field of view, this can be less than ideal when additional surgical personnel or tools obstruct access to parts of the surgical site. Therefore, the digital pointing indicator can be moved by the movement of that part of the surgical tool, but not necessarily to the location indicating that part of the surgical tool. In other words, the system can be configured to determine the position of the digital pointing indicator based on the relative movement of that part of the surgical tool. Specifically, the digital pointing indicator may be offset from that part of the surgical tool, making it more akin to using a mouse or touchpad to control the digital pointing indicator than to using a tablet computer. For example, the system can be configured to determine the position of the digital pointing indicator such that it is offset from that part of the surgical tool, as perceived by a user viewing a digital pointing indicator superimposed on a view of the surgical site obtained by the surgical imaging device. This reduces the likelihood of the surgeon being constrained by objects obstructing access to the entire surgical site, and smaller movements may be sufficient to move the digital pointing indicator to the desired position, thus improving the ergonomics of targeting.

[0013] During surgical procedures, many (if not all) spatial positioning functions are performed toward the center of the field of view of the surgical imaging apparatus. Therefore, to avoid having to move the digital pointing indicator across the entire field of view, the digital pointing indicator can be constrained to the central portion of the field of view. In other words, the system can be configured to determine the position of the digital pointing indicator such that it is constrained to the central portion of the field of view of the surgical imaging apparatus. For example, the central portion of the field of view may cover at least 30% (or at least 40%, or at least 50% and / or up to 90% (and / or up to 80%, up to 70%)) of the area of ​​the field of view of the surgical imaging apparatus. This facilitates the placement of the digital pointing indicator.

[0014] Specifically, the system can be configured to determine the position of the digital pointing indicator such that the digital pointing indicator is constrained to a rectangular portion of the field of view, which is located at the center of the field of view of the surgical imaging device. The rectangular portion of the field of view is easily understood by the surgeon because in many cases, at least when the user is using an auxiliary screen, the surgeon is also provided with a rectangular view of the surgical site.

[0015] In many cases, multiple surgical instruments may be (partially) visible simultaneously in the field of view. To determine which of these surgical instruments should be used to move the digital pointing indicator, the surgeon can identify the surgical instrument to be used by moving (e.g., shaking) the surgical instrument after activating the digital pointing indicator. In other words, the system can be configured to determine, based on the movement of the surgical instruments after activating the digital pointing indicator, that a surgical instrument is being used to control the digital pointing indicator if multiple surgical instruments are visible in the field of view of the surgical imaging device. For example, the system can be configured to provide visual or audio feedback after determining that a surgical instrument is being used to control the digital pointing indicator. In this way, the proposed concept can be used even in "busy" surgical sites.

[0016] More generally, providing feedback on the functions being performed increases surgeons' confidence in using the system, making them more likely to continue using it. For example, the system can be configured to include visual indicators representing voice commands in the overlay after processing them. In this way, surgeons can be confident that the voice commands have been recorded and correctly interpreted.

[0017] In a surgical setting, many different types of surgical tools are available. Most of these tools (such as scalpels, retractors, clamps, and suction tubes) are intuitively designed for pointing at objects. For example, a system can be configured to determine the position of one of the following: the end of a surgical tool (e.g., a retractor or clamp), the blade of a scalpel (e.g.), or the opening of a tube (e.g., a surgical aspirator or suction device).

[0018] Another aspect of this disclosure relates to a surgical imaging system that includes such a system. For example, the surgical imaging system may be a surgical imaging system used in ophthalmic surgery.

[0019] Another aspect of this disclosure relates to a method for a surgical imaging system. The method includes acquiring an image data stream from an imaging apparatus of the surgical imaging system. The method includes processing the image data stream to track the position of a portion of a surgical instrument. The method includes determining, based on changes in the position of that portion of the surgical instrument, the position of a digital pointing indicator relative to the field of view of the surgical imaging apparatus of the surgical imaging system, indicating a surgical site. The method includes generating an overlay image including the digital pointing indicator to be overlaid on the field of view of the surgical imaging apparatus. The method includes providing the overlay image with the digital pointing indicator to a display device of the surgical imaging system. The method includes acquiring a voice command from a user of the surgical imaging system. The voice command includes instructions for performing a spatial positioning function. The method includes triggering the execution of the spatial positioning function based on the voice command, according to the position of the digital pointing indicator relative to the field of view of the surgical imaging apparatus.

[0020] One aspect of this disclosure relates to a computer program having program code for performing the methods described above when the program is executed on a processor. Attached Figure Description

[0021] The following description will use only examples of devices and / or methods, with reference to the accompanying drawings, in which:

[0022] Figure 1a A block diagram of an example system for surgical imaging is shown;

[0023] Figure 1b A schematic diagram illustrating an example of a surgical imaging system is shown;

[0024] Figure 1c and Figure 1d An example of an overlay diagram is shown, which includes a digital pointing indicator superimposed on a view of the surgical site;

[0025] Figure 2 A flowchart illustrating an example method for a surgical imaging system is shown; and

[0026] Figure 3 A schematic diagram of an example system including an imaging device and a computer system is shown. Detailed Implementation

[0027] Various examples will now be described more fully with reference to the accompanying drawings, some of which illustrate certain aspects. For clarity, the thickness of lines, layers, and / or regions may be exaggerated in the drawings.

[0028] Figure 1a A surgical imaging system 100 is shown. Figure 1b A schematic diagram of an example of system 110 (shown). System 110 is a component of surgical imaging system 100 and can be used to control various aspects of surgical imaging system 100. Specifically, the system can be used to acquire and / or process imaging data by at least one optical imaging sensor of imaging device 120 of surgical imaging system through various means, which will be described in more detail below. In addition, system 110 can be configured, for example, to control additional aspects of surgical imaging system 100 to provide display signals to one or more display devices 130a, 130b of surgical imaging system, and / or control and / or perform automatic operations of surgical imaging system 100 (such as spatial positioning functions provided by surgical imaging system 100).

[0029] Generally, system 110 may be a computer system. System 110 includes one or more processors 114 and one or more storage devices 116. Optionally, system 110 further includes one or more interfaces 112. The one or more processors 114 are coupled to one or more storage devices 116 and one or more interfaces 112. Generally, the functionality of system 110 may be provided by one or more processors 114 in conjunction with one or more interfaces 112 (which are used to exchange data / information with one or more other components of surgical imaging system 100 and located outside surgical imaging system 100, such as optical imaging sensors of imaging device 120, one or more display devices 130a, 130b, microphone 140, panoramic camera 150 and / or OCT scanner 160) and in conjunction with one or more storage devices 116 (which are used to store information, such as machine-readable instructions of computer programs being executed by one or more processors). Generally, the functionality of one or more processors 114 may be implemented by one or more processors 114 executing machine-readable instructions. Therefore, any feature attributable to one or more processors 114 may be defined by one or more machine-readable instructions. System 110 may include, for example, machine-readable instructions located within one or more storage devices 116.

[0030] As outlined above, system 110 is part of surgical imaging system 100, which includes various components in addition to system 110. For example, surgical imaging system 100 includes imaging device 120 and may include one or more additional components, such as one or more display devices 130a, 130b, such as eyepiece display 130b and / or auxiliary display 130a. Figure 1bA schematic diagram of an example of such a surgical imaging system 100, particularly an ophthalmic surgical microscope system 100, is shown. The surgical microscope system is the surgical imaging system 100, which includes a (surgical) microscope as an imaging device 120. However, the proposed concept is not limited to such embodiments. The surgical imaging system 100 can be based on various (single or multiple) imaging devices, such as one or more microscopes, one or more endoscopes, and / or one or more exoscopes (sometimes also referred to as external telescopes). Exoscopes are camera-based imaging systems, and particularly camera-based 3D imaging systems, suitable for providing images of surgical sites with high magnification and large depth of field. Compared to microscopes that can be used via eyepiece 130a, exoscopes are used only via display modalities (such as monitors or head-mounted displays). Therefore, the surgical imaging system 100 may alternatively be a surgical endoscope system or a surgical exoscope system. The following illustrations assume that the surgical imaging device 120 is a surgical microscope or a surgical exoscope, and that the surgical imaging system 100 is a surgical microscope system 100 or a surgical exoscope system 100.

[0031] Therefore, the surgical imaging system or surgical microscope system 100 may include an imaging device (such as microscope 120). Generally, a microscope (such as microscope 120) is an optical instrument suitable for examining objects too small to be examined (alone) with the naked eye. For example, a microscope can provide optical magnification of a sample (such as a surgical site). In this concept, optical magnification is also provided for at least one optical imaging sensor. Therefore, microscope 120 includes an optical imaging sensor coupled to system 110. Microscope 120 may further include one or more optical magnification components, such as objectives (i.e., lenses), for magnifying a view of the sample. For example, the surgical imaging device or microscope 120 is generally referred to as the "optics carrier" of an imaging system.

[0032] Various types of surgical imaging devices exist. If the imaging device is used in the medical or biological field, the object viewed through the imaging device can be, for example, a sample of organic tissue arranged in a petri dish or present in a part of a patient's body. In the various examples presented here, the imaging device 120 can be a microscope of a surgical microscope system, i.e., a microscope used during surgical procedures such as ophthalmological procedures (i.e., eye surgery). Therefore, the sample viewed through the surgical imaging device can be a sample of the patient's organic tissue, and specifically, a surgical site (e.g., the eye) operated on by the surgeon during a surgical procedure. However, the proposed concept is also applicable to other types of surgery (such as neurosurgery or cardiac surgery).

[0033] Figure 1b A schematic diagram of an example of a surgical imaging system 100, particularly a surgical microscope system 100, including system 110 and surgical imaging device 120 is shown. Figure 1b The surgical microscope system 100 shown includes several optional components, such as a base unit 105 (including system 110) with a (rolling) support, an eyepiece display 130b disposed at the imaging device 120, an auxiliary display 130a disposed at the base unit, and an arm that holds the microscope 120 in place and is connected to the base unit and the microscope 120. Figure 1b In this context, the surgical imaging system 100 further includes a microphone 140, a panoramic camera 150, and an OCT scanner 160.

[0034] In the proposed concept, system 110 is used to control the spatial positioning function of surgical imaging system 100 by tracking the position of a portion (e.g., tip, blade, or tube opening) of a surgical tool (such as a scalpel, clamp, or suction device). Specifically, system 110 is configured to acquire and process image data streams (e.g., video streams or image sequences) from imaging devices 120, 150 of the surgical imaging system. For example, system 110 may be configured to receive or read image data from the respective imaging devices 120, 150 of the surgical imaging system. For example, image data streams may be acquired from surgical imaging device 120 (such as a surgical endoscope or surgical microscope) or from the panoramic camera 150 of the surgical imaging system. In this context, the panoramic camera 150 of the surgical imaging system is a separate wide-field-of-view camera that provides a broader anatomical context around the surgical site. Unlike the main surgical imaging device 120, which provides a high-magnification, detailed view through an eyepiece, the panoramic camera 150 displays a lower-magnification, wider field of view of the entire surgical area to the surgical team.

[0035] Regardless of the source of the image data stream, it is processed to track the location of that part of the surgical tool 20. Figure 1c and Figure 1d An example of an overlay diagram is shown, which includes a digital pointing indicator 30 superimposed on a view of the surgical site (i.e., the eye). Figure 1c and Figure 1d The image data stream of the surgical imaging device 120 is shown as an example image frame in which the end of the surgical instrument 20 is visible. Similar image frames can be obtained from an optional panoramic camera 150, although at a lower magnification. In both types of image frames, the surgical instrument 20 is clearly visible. This means that the location of the end, blade, or tube opening can be visually determined. The system 110 is configured to process the image data stream to determine and track the location of this part of the surgical instrument (e.g., relative to the surgical site, relative to the field of view of the surgical imaging device 120, or relative to the field of view shown in the image data stream (which could be the field of view of the panoramic camera)). This can be accomplished using various mechanisms.

[0036] The first mechanism is based on the use of machine learning. In other words, machine learning models and / or algorithms can be used to determine the position of a part of an tool (such as its end). For example, assuming a limited number of surgical tools (scalpels, clamps, scissors, retractors, hemostatic forceps, surgical drills, surgical saws, suction devices, etc.) are used during surgery, a machine learning model can be trained to predict the position of that part of the surgical tool. Specifically, pose estimation can be used for this purpose. In other words, the system can be configured to process image data using a machine learning model to estimate the pose of the surgical tool and determine the position of that part of the surgical tool based on the estimated pose of the surgical tool.

[0037] Pose estimation is a computer vision technique that determines the position and orientation of an object in 2D or 3D space based on image data. In surgical applications, pose estimation involves detecting and tracking the spatial coordinates, rotation angles, and overall configuration of surgical instruments, anatomical structures, or other relevant objects within the field of view. For surgical imaging applications, pose estimation using machine learning typically involves deep neural networks. These models process microscope image data by identifying key features and landmarks of the target object. The network can be designed to output direct pose parameters (position and orientation) or intermediate representations (such as key points that can be used to derive the pose).

[0038] Training this machine learning model requires a large dataset of labeled images showing the object of interest from various angles, locations, and lighting conditions. The training data further includes ground-based pose information for each image. This can be obtained through manual annotation, synthetic data generation, or by using other sensing systems as a reference. The model learns to associate visual features with corresponding pose parameters through supervised learning, thereby minimizing the difference between the predicted pose and the actual pose.

[0039] To determine the position of a specific part of a surgical tool (such as a scalpel blade), the system combines pose estimation results with known object dimensions and imaging device parameters (such as zoom level and working distance). Using the overall pose as a starting point, geometric transformations can be used to calculate the position of any point on the object. The system can calculate the position of specific features (such as the tip or edge of a scalpel blade) as the pixel coordinates of that part of the surgical tool relative to the image data stream, or, if the working distance and zoom factor of the corresponding imaging device are known, the position of that part relative to the surgical site.

[0040] A second technique for determining the location of a portion of a surgical tool uses a 3D model of the surgical tool to determine its pose (i.e., position and orientation). Specifically, the system can be configured to determine the location of that portion of the surgical tool by fitting a projection of a 3D model of the surgical tool onto the surgical tool as shown in image data. The projection of the 3D model of the surgical tool onto the image data can be accomplished through a process known as model-based tracking. By creating a virtual projection of the tool's 3D model and overlaying that virtual projection onto a 2D microscope image, optimization algorithms can be used to adjust the position and orientation of the projected model until the projected model best matches the actual tool visible in the image. This fitting process typically involves reducing or minimizing the difference between the edges of the projected model and the corresponding edges detected in the image data. The algorithm iteratively refines the tool's pose parameters (position and orientation) until the projected model is closely aligned with an image of the real tool. Once the projection is accurately fitted onto the surgical tool in the image data, the location of that portion of the tool (such as a surgical scalpel blade) can be determined.

[0041] In the proposed concept, the position of this part of the surgical tool is not only determined once, but also tracked. Tracking the position of this part of the surgical tool means determining its position multiple times to establish relative changes and movements. In other words, the position of the surgical tool is determined multiple times, and the relative changes in position are determined. In other words, system 110 can be configured to determine the relative movement of this part of the surgical tool.

[0042] In many cases, multiple surgical tools may be (partially) visible simultaneously in the field of view. To determine which of these surgical tools should be used to move the digital pointing indicator, the surgeon may identify the surgical tool to be used by moving (e.g., shaking) the surgical tool after activating the digital pointing indicator. For example, system 110 may be configured to determine that a surgical tool is being used to control the digital pointing indicator based on the movement (e.g., shaking movement) of the surgical tool after the digital pointing indicator is activated, if multiple surgical tools are visible in the field of view of the surgical imaging device. For example, system 110 may be configured to use the determined position of the surgical tool to detect the movement of the surgical tool and ignore the positions of other surgical tools in order to determine the position of the digital pointing indicator. To let the surgeon know that the surgical tool has been successfully identified, system 110 may be configured to provide visual or audio feedback (e.g., confirmation of sound or the outline around the surgical tool as part of an overlay) after determining that the surgical tool is being used to control the digital pointing indicator.

[0043] The system uses changes in the position of this part of the surgical tool to determine the position of the digital pointing indicator 30, for example, as Figure 1c and Figure 1dAs shown. In other words, system 110 is configured to determine the position of the digital pointing indicator 30 based on changes in the position of this part of the surgical tool. Specifically, the digital pointing indicator 30 (i.e., the cursor or pointer, such as...) Figure 1c and Figure 1d The position of the crosshairs shown (which will be superimposed on the field of view of the surgical imaging device 120) is determined relative to the field of view 10 of the surgical imaging device 120, which displays the surgical site. This does not (necessarily) mean that the pointing indicator is placed in the field of view to indicate the position of the end of the surgical instrument. Rather, in the proposed concept, the digital pointing indicator 30 moves based on the relative movement of this part of the surgical instrument (e.g., similar to a mouse / touchpad cursor). In other words, the system 110 can be configured to determine the position of the digital pointing indicator based on the relative movement of this part of the surgical instrument. For example, when the digital pointing indicator is activated, the system 110 can be configured to move the digital pointing indicator 30 relative to the field of view 10 of the surgical imaging device 120 according to the movement of this part of the surgical instrument. Specifically, in order to avoid obstructing the surgeon's view of the surgical site and surgical instruments, system 110 may be configured to determine the position of the digital pointing indicator such that the digital pointing indicator 30 (generally) is offset from that part of the surgical instrument, as perceived by the user when viewing the digital pointing indicator superimposed on the view of the surgical site obtained by the surgical imaging device.

[0044] In most cases, spatial positioning functions (i.e., functions to be performed at a specific location, such as focusing, setting markers, changing the field of view, or using measuring devices) are used. Figure 1b The OCT scanner 160 shown (performing measurement functions) is more commonly used at the center of the field of view rather than at the periphery, because surgeons typically set the field of view so that the relevant part of the surgical site is centered. To facilitate the placement of the digital pointing indicator, it can be restricted to the center of the field of view. For example, as... Figure 1d As shown, system 110 can be configured to determine the position of digital pointing indicator 30 such that the digital pointing indicator is constrained to a central portion 40 of the field of view (e.g., a rectangular portion 40 constrained to the center of the field of view). For example, this central portion may cover at least 30% (or at least 40%, or at least 50% and / or up to 90% (and / or up to 80%, up to 70%)) of the area of ​​the field of view of the surgical imaging device.

[0045] In the proposed concept, a digital pointing indicator 30 is used to select the location to trigger the spatial positioning function. For this purpose, as... Figure 1c and Figure 1dAs shown, a digital pointing indicator 30 is shown overlaid on the field of view 10 of a surgical imaging device 120, which the surgeon uses to perform surgical procedures. This is accomplished using an overlay image to be superimposed on the field of view of the surgical imaging device 120. In this context, an overlay image is a layer of digital information shown on top of a digital or optical view of the surgical site. The system 110 is configured to generate an overlay image that includes a digital pointing indicator to be superimposed on the field of view of the surgical imaging device. For example, in the overlay image, the digital pointing indicator 30 may be shown as a dot, a circle, a mouse cursor, etc.

[0046] System 110 is configured as a display device 130a, 130b of a surgical imaging system to provide an overlay image with a digital pointing indicator 30. The purpose is to show the digital pointing indicator 30 overlaid on a view of the surgical site, as provided by surgical imaging device 120. To achieve this, two scenarios can be distinguished. In the first scenario, if the view of the surgical site is a (non-digital) optical view, the overlay image can be superimposed on the view by projecting the overlay image onto a translucent mirror or displaying the overlay image on a translucent display, wherein the mirror or display is arranged in the optical path between the eyepiece and objective lens of the surgical imaging device. In this case, the display device can be a projector or a translucent display. In many cases, surgical imaging device 120 can be a hybrid or fully digital surgical imaging device. Therefore, in the second scenario, the overlay image can be superimposed on a digital view of the surgical site, as provided by the optical imaging sensor of surgical imaging device 120. Therefore, system 110 can be configured to acquire image data (e.g., image data stream) from surgical imaging device 120 and generate a (digital) view of the surgical site with the overlay image superimposed based on the image data. System 110 can be configured to provide a view of a surgical site with an overlay to the display devices 130a, 130b of a surgical imaging system.

[0047] A digital pointing indicator 30 is shown superimposed on the field of view of the surgical imaging device 120 to enable the surgeon to select a location within the surgical site where a spatial positioning function is to be performed. While performing the location selection by moving the digital pointing indicator relative to the field of view of the surgical imaging device 120, a voice command is used to trigger the spatial positioning function at the location where the digital pointing indicator is shown. Therefore, the system 110 is configured (e.g., via microphone 140) to receive (e.g., from) a voice command from a user of the surgical imaging system, wherein the voice command includes instructions for performing the spatial positioning function. For example, the system 110 may be configured to perform speech recognition or speech-to-text algorithms or machine learning models on the voice command and use the results of the speech recognition or speech-to-text algorithms or machine learning models to determine to trigger the spatial positioning function and / or which spatial positioning function to trigger. Feedback that the voice command has been successfully recognized may be provided to the user / surgeon. Therefore, the system may be configured to include a visual indicator (e.g., an icon or text indicating a spatial positioning function) representing the voice command in the overlay after processing the voice command.

[0048] Some spatial positioning functions can be applied to a region of a surgical site, rather than a single location within the surgical site. For example, a region can be selected by determining two positions of a digital pointing indicator corresponding to two corners of the region. For instance, system 110 can be configured to determine two positions of the digital pointing indicator relative to the field of view of the surgical imaging apparatus of the surgical imaging system if a voice command (e.g., a spatial positioning function associated with the voice command) instructs that a spatial positioning function be performed at a certain region. These two positions can be considered as the diagonals of the rectangle defining the region. Therefore, system 110 can be configured to determine the region based on the two positions and based on the orientation of the field of view of the surgical imaging apparatus.

[0049] System 110 is configured to trigger a spatial positioning function based on voice commands, according to one or more locations (i.e., regions) of the digital pointing indicator relative to the field of view of the surgical imaging apparatus (i.e., the location of the surgical site where the digital pointing indicator is shown in the field of view of the surgical imaging apparatus). For example, system 110 may include a data storage device for at least one trigger word or sentence associated with at least one spatial positioning function, and trigger the spatial positioning function when the corresponding trigger word or sentence is detected.

[0050] The following sections provide some specific examples of the proposed concept. The proposed concept introduces cursor-linked guidance and intelligent control of surgical imaging devices (e.g., microscopes) by providing a cursor-linked control system integrated into a surgical imaging system (e.g., a surgical microscope system). This system enables surgeons to intuitively interact with the functions of the imaging device through the movement of their surgical tools and voice commands.

[0051] The system tracks the position and movement of surgical instruments within the field of view of the imaging device. This allows the cursor (a digital pointing indicator) to move accordingly. The surgeon can then use voice commands to perform functions, such as focusing by saying "Focus here" or initiating an optical coherence tomography (OCT) scan directly at the cursor's location by saying "Scan here." (Example of measuring device 160, such as...) Figure 1b As shown, it is used for spatial positioning (scanning at surgical sites).

[0052] The proposed concept eliminates the need for cumbersome manual controls such as foot pedals, touchscreens, or mouth controllers. By combining tool tracking and voice recognition, the system significantly simplifies workflow, reduces cognitive load, and improves accuracy. Unlike other methods, this system allows for seamless, hands-free operation. This increases efficiency and allows surgeons to focus entirely on the procedure.

[0053] The proposed concept integrates a voice-command-enabled cursor-linked control system into a surgical microscope. This innovative integration enables intuitive interaction by tracking the movement of surgical tools within the imaging device's field of view and linking these movements to a cursor on the screen. Surgeons can use voice commands to guide the surgical imaging system (e.g., the surgical imaging device) to perform specific functions, such as focusing or scanning, at the cursor's location. By eliminating the need for manual navigation via foot pedals, touchscreens, or mouth controllers, this system improves workflow efficiency, simplifies operation, and enhances accuracy.

[0054] The proposed concept is based on tool tracking and cursor integration: a tracking mechanism (such as optical, infrared, or other motion tracking technologies) is used to detect the position and movement of surgical tools in real time. These detected movements are linked to a cursor visible within the imaging device's field of view, allowing the cursor to move along with the tool.

[0055] Additionally, the voice command functionality is integrated through a voice recognition function that accepts commands such as "Focus here" to adjust the microscope's focus to the cursor position or "Scan here" to initiate an optical coherence tomography (OCT) scan at a specified point (i.e., in a spatially localized manner). Additional commands enable other functions, such as brightness adjustment or image capture. For example, the system can use a noise filtering system to process voice commands to improve reliability in the operating room environment.

[0056] The proposed concept also enables hands-free operation by combining tool tracking and voice commands to provide hands-free and intuitive operation, minimizing physical interaction with manual controls. Its intuitive design allows surgeons to focus on surgical procedures while reducing cognitive and physical stress.

[0057] Integration of this system with existing microscope systems is possible, as it can be retrofitted to existing surgical microscopes or excimers or integrated into new designs. The system can connect to current OCT modules and focusing adjustment mechanisms, thereby utilizing existing optical and imaging capabilities.

[0058] The functional workflow begins with initialization. For example, when the microscope detects the surgeon's tool within its field of view, the cursor activation system can be activated. The tracking module then begins monitoring the tool's position and displays the cursor at the corresponding location. During tool tracking and cursor movement, the cursor dynamically follows the surgeon's movement of the tool in real time. The cursor can be precisely calibrated to align with the tip (or other part) of the surgical tool for accurate positioning.

[0059] Surgeons can issue commands such as "Focus here" or "Scan here" as voice commands. These commands are processed by a speech recognition function, which triggers the execution of the requested function at the cursor position.

[0060] In various examples, feedback and confirmation can be provided. The system can provide immediate feedback through visual or auditory cues to confirm the successful execution of commands. For example, the "Focus" command produces sharp, clear images, while the "OCT Scan" command displays scan results immediately (e.g., in real time).

[0061] In the proposed concept, the surgical imaging system displays a cursor that moves precisely in sync with the surgeon's tool within the field of view, as cursor movement is linked to the tool. Focusing adjustments or OCT scans can be activated via voice commands such as "Focus here" or "Scan here." The proposed workflow achieves hands-free operation by eliminating the need for manual controls such as foot pedals, touchscreens, or mouth controllers. Immediate (e.g., real-time) feedback (e.g., visual or auditory feedback) can be used to confirm actions linked to cursor-based operations. The proposed system integrates tool tracking and voice recognition with existing microscope components, such as OCT modules or focusing mechanisms, thus providing efficient system integration.

[0062] In the proposed concept, system 110 can be used to generate and provide an overlay image to be superimposed on a digital or optical view of a surgical site. For example, if a digital view is used, the digital view is generated using image data from the surgical imaging device 120 (e.g., by performing image processing on image data from the optical imaging sensor of the surgical imaging device). The aforementioned overlay image is then superimposed on the digital view of the surgical site. Alternatively, if the overlay image is superimposed on an optical view of the surgical site, only the overlay image can be provided to a display device. In both cases, i.e., in the case of a digital view with an overlay image or only an overlay image, the corresponding visual data (digital view with an overlay image or only an overlay image) is provided to a display device, such as an eyepiece display 130b, a stereoscopic head-mounted display, or a monitor 130a. This visual data can be created and provided as part of a display signal to the display device of the surgical imaging system. In other words, system 110 can be configured to output a display signal to the display devices 130a, 130b of a surgical or scientific imaging system, wherein the display signal includes the overlay image and optionally includes the digital view of the surgical site. For example, the display signal may be a signal used to drive (e.g., control) a corresponding display device. For example, the display signal may include video data and / or control commands for driving the display. For example, the display signal may be provided via one of one or more interfaces 112 of the system. Thus, the system 110 may include a video interface 112 adapted to provide display signals to the display devices 130a, 130b of the microscope system 100.

[0063] In the proposed surgical imaging system, the optical imaging sensor of the surgical imaging device 120 (or panoramic camera 150) is used to provide the aforementioned image data. Therefore, the optical imaging sensor (which may be part of the imaging device 120 (e.g., a microscope) and / or the panoramic camera 150) can be configured to generate image data. For example, the optical imaging sensor of the imaging device 120 or the panoramic camera 150 may include or be an APS (Active Pixel Sensor) based imaging sensor or a CCD (Charge-Coupled Device) based imaging sensor. For example, in an APS-based imaging sensor, light at each pixel is recorded using a photodetector and an active amplifier of the pixel. APS-based imaging sensors are typically based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology. In a CCD-based imaging sensor, incident photons are converted into electronic charges at the semiconductor-oxide interface, and these electronic charges then move between capacitor cells within the imaging sensor via the sensor's circuitry to perform imaging.

[0064] One or more interfaces 112 of system 110 may correspond to one or more input and / or output terminals for receiving and / or transmitting information within a module, between modules, or between modules of different entities. This information may be a digital (bit) value based on a specified code. For example, one or more interfaces 112 may include interface circuitry configured to receive and / or transmit information. One or more processors 114 of system 110 may be implemented using one or more processing units, one or more processing devices, or any component for processing, such as a processor, computer, or programmable hardware component that can operate with appropriately adapted software. In other words, the described functionality of one or more processors 114 may also be implemented in software, which then executes on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. One or more storage devices 116 of system 110 may include at least one element from the group of computer-readable storage media, such as magnetic or optical storage media, such as hard disk drives, flash memory, floppy disks, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or network storage devices.

[0065] In conjunction with the proposed ideas or with the preceding or following text (e.g., Figure 2 or Figure 3 The one or more examples described herein refer to further details and aspects of system 110 and surgical imaging system 100. System 110 and surgical imaging system 100 may include one or more additional optional features corresponding to one or more aspects of the proposed concept or to one or more examples described above or below.

[0066] Figure 2 It shows a surgical imaging system (such as Figure 1b A flowchart illustrating an example of a corresponding method for a surgical imaging system 100. Combined with... Figures 1a to 1d The features discussed in surgical imaging systems 100 and 110 may also be included in Figure 2The corresponding method includes obtaining image data streams from imaging devices 120 and 150 of a surgical imaging system. The method includes processing the image data streams to track the position of a portion of a surgical tool 20. The method includes determining, based on changes in the position of that portion of the surgical tool, the position of a digital pointing indicator 30 relative to the field of view 10 of the surgical imaging device 120 of the surgical imaging system, showing the surgical site. The method includes generating an overlay image, including the digital pointing indicator to be overlaid on the field of view of the surgical imaging device. The method includes providing an overlay image with the digital pointing indicator to display devices 130a and 130b of the surgical imaging system. The method includes obtaining a voice command from a user of the surgical imaging system. The voice command includes instructions for performing spatial positioning functions. The method includes triggering, based on the voice command, the performance of spatial positioning functions according to the position of the digital pointing indicator relative to the field of view of the surgical imaging device.

[0067] In conjunction with the proposed idea or the preceding or following text (e.g., Figures 1a to 1d or Figure 3 The examples described herein refer to further details and aspects of the method. The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0068] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as “ / ”.

[0069] Although some aspects are described in the context of the device, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of the corresponding device.

[0070] Some implementations involve a surgical imaging device, including, for example, combining Figures 1a to 2 One or more of the systems described herein. Alternatively, the surgical imaging device may be as combined with Figures 1a to 2 One or more of the systems described herein are part of or connected to that system. Figure 3A schematic diagram of a system 300 configured to perform the methods described herein is shown. System 300 includes a surgical imaging device 310 and a computer system 320. The surgical imaging device 310 is configured to capture images and is connected to the computer system 320. The computer system 320 is configured to perform at least a portion of the methods described herein. The computer system 320 may be configured to execute machine learning algorithms. The computer system 320 and the surgical imaging device 310 may be separate entities, but may also be integrated together in a common housing. The computer system 320 may be part of the central processing system of the surgical imaging device 310, and / or the computer system 320 may be part of sub-components of the surgical imaging device 310, such as sensors, actuators, cameras, or illumination units of the surgical imaging device 310.

[0071] Computer system 320 may be a local computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed across various locations (e.g., distributed across local clients and / or one or more remote server farms and / or data centers)). Computer system 320 may include any circuitry or combination of circuitry. In one embodiment, computer system 320 may include one or more processors, which may be of any type. As used herein, a processor may refer to any type of computing circuitry, such as, but not limited to, microprocessors, microcontrollers, complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, graphics processors, digital signal processors (DSPs), multi-core processors, field-programmable gate arrays (FPGAs), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 320 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radio components, and similar electronic systems. Computer system 320 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for disposing of removable media (such as optical discs (CDs), flash memory cards, digital video discs (DVDs), etc.). Computer system 320 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information into and receive information from computer system 320.

[0072] Some or all of the method steps may be performed by (or using) hardware devices (such as processors, microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more of the most important method steps may be performed by such devices.

[0073] Depending on certain specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be executed using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) that stores electronically readable control signals thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to cause the corresponding method to be executed. Therefore, the digital storage medium can be computer-readable.

[0074] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to enable the execution of one of the methods described herein.

[0075] Generally, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0076] Other implementations include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0077] Therefore, in other words, an embodiment of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is run on a computer.

[0078] Therefore, a further embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon for performing one of the methods described herein when executed by a processor. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. A further embodiment of the invention is an apparatus as described herein, including a processor and a storage medium.

[0079] Therefore, a further embodiment of the invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0080] Further embodiments include a processing element (e.g., a computer or programmable logic device) configured or adapted to perform one of the methods described herein.

[0081] A further embodiment includes a computer on which a computer program is installed for performing one of the methods described herein.

[0082] A further embodiment of the invention includes an apparatus or system configured to transmit, for example, electronically or optically to a receiver, a computer program for performing one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transmitting the computer program to the receiver.

[0083] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0084] Implementation can be based on the use of machine learning models or machine learning algorithms. Machine learning can refer to algorithms and statistical models that a computer system can use to perform specific tasks without explicit instructions (but relying on models and inference). For example, in machine learning, data transformations inferred from the analysis of historical and / or training data can be used instead of rule-based data transformations. For example, machine learning models or algorithms can be used to analyze the content of images. To enable a machine learning model to analyze the content of an image, it can be trained using training images as input and training content information as output. By training a machine learning model using a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model “learns” to recognize the content of images, and thus can be used to identify the content of images not included in the training data. The same principle can also be used for other kinds of sensor data: by training a machine learning model using training sensor data and a desired output, the machine learning model “learns” a transformation between the sensor data and the output, which can be used to provide output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) can be preprocessed to obtain feature vectors, which are used as input to a machine learning model.

[0085] Machine learning models can be trained using training input data. The example specified above uses a training method called "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, where each sample can include multiple input data values ​​and multiple expected output values; that is, each training sample is associated with an expected output value. By specifying both the training samples and the expected output values, the machine learning model "learns" which output value to provide based on input samples similar to those provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms (e.g., classification algorithms, regression algorithms, or similarity learning algorithms). Classification algorithms are used when the output is restricted to a finite set of values ​​(categorical variables), i.e., the input is classified as one of the values ​​in the finite set. Regression algorithms are used when the output can have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but they are based on learning from examples using a similarity function that measures the similarity or correlation between two objects. In addition to supervised or semi-supervised learning, unsupervised learning can also be used to train machine learning models. In unsupervised learning, input data may be provided (only), and unsupervised learning algorithms can be used to find structure within the input data (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering assigns input data, which includes multiple input values, to subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, while being dissimilar to input values ​​included in other clusters.

[0086] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software agents (called "software agents") are trained to take actions in an environment. Rewards are calculated based on the actions taken. Reinforcement learning is based on training one or more software agents to select actions that increase the cumulative reward, thereby causing the software agents to improve on the tasks they are given (as demonstrated by increasing rewards).

[0087] Furthermore, several techniques can be applied to some of these machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained, at least in part, using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms (which may be called representation learning algorithms) can preserve information from their inputs but can also transform them in a way that makes them useful, often as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0088] In some examples, anomaly detection (i.e., outlier detection) can be used to provide identification of input values ​​that raise suspicion by being significantly different from the majority of the input or training data. In other words, machine learning models may be trained using anomaly detection in at least part, and / or machine learning algorithms may include anomaly detection components.

[0089] In some examples, machine learning algorithms can use decision trees as predictive models. In other words, machine learning models can be based on decision trees. In a decision tree, observations about an item (e.g., a set of input values) are represented by branches of the decision tree, and the output value corresponding to that item is represented by the leaves. Decision trees can support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree can be represented as a classification tree; if continuous values ​​are used, the decision tree can be represented as a regression tree.

[0090] Association rules are a further technique that can be used in machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. Machine learning algorithms can identify and / or utilize one or more relational rules that represent knowledge derived from the data. These rules can be used, for example, to store, manipulate, or apply knowledge.

[0091] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" can refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" can refer to a set of data structures and / or rules representing the learned knowledge (e.g., based on training performed by a machine learning algorithm). In implementations, the use of a machine learning algorithm may mean the use of an underlying machine learning model (or multiple underlying machine learning models). The use of a machine learning model may mean that the machine learning model and / or the set of data structures / rules serving as the machine learning model are trained by a machine learning algorithm.

[0092] For example, a machine learning model can be an artificial neural network (ANN). An ANN is a system inspired by biological neural networks (such as those found in the retina or brain). An ANN consists of multiple interconnected nodes and multiple connections (so-called edges) between the nodes. There are typically three types of nodes: input nodes that receive input values; hidden nodes that are connected to other nodes (only); and output nodes that provide output values. Each node can represent an artificial neuron. Each edge transfers information from one node to another. The output of a node can be defined as a (non-linear) function of its inputs (e.g., the sum of its inputs). The inputs of a node can be used as a function based on the "weights" of the edges or nodes that provide the input. The weights of nodes and / or edges can be adjusted during the learning process. In other words, training an artificial neural network can involve adjusting the weights of the nodes and / or edges of the artificial neural network, i.e., to achieve a desired output for a given input.

[0093] Alternatively, the machine learning model can be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with associated learning algorithms that can be used to analyze data (e.g., in classification or regression analysis). An support vector machine can be trained by providing inputs with multiple training input values ​​belonging to one of two categories. An support vector machine can be trained to assign new input values ​​to one of the two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graph model. A Bayesian network can use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the process of natural selection.

[0094] List of reference numerals

[0095] 10: Field of view

[0096] 110: System

[0097] 112: Interface

[0098] 114: Processor

[0099] 116: Storage device

[0100] 120: Surgical Imaging Device

[0101] 130a: Auxiliary display

[0102] 130b: Eyepiece display

[0103] 140: Microphone

[0104] 150: Panoramic Camera

[0105] 160: OCT scanner

[0106] 20: Surgical tools

[0107] 30: Digital pointing indicator

[0108] 40: (Central part of the field of view)

[0109] 100: Surgical Imaging System

[0110] 105: Base Unit

[0111] 210: Obtain image data stream

[0112] 220: Processing image data streams to track the location of a portion of a surgical tool.

[0113] 230: Determine the position of the digital pointer indicator

[0114] 240: Generate an overlay image including a digital pointing indicator.

[0115] 250: Provides an overlay image with a digital pointing indicator for a display device.

[0116] 260: Obtain voice commands

[0117] 270: Trigger the execution of spatial positioning function

[0118] 300: System

[0119] 310: Imaging device

[0120] 320: Computer System

Claims

1. A system (110) for a surgical imaging system (100), the system comprising one or more processors and one or more storage devices, wherein the system is configured to: Obtain the image data stream of the imaging device (120, 150) of the surgical imaging system; The image data stream is processed to track the position of a portion of the surgical tool (20); The position of the digital pointing indicator (30) relative to the surgical imaging device (120) of the surgical imaging system, showing the field of view (10) of the surgical site, is determined based on the change in the position of the part of the surgical tool. Generate an overlay image, the overlay image including the digital pointing indicator to be overlaid on the field of view of the surgical imaging device; Provide the overlay image with the digital pointing indicator to the display device (130a, 130b) of the surgical imaging system; The system obtains voice commands from the user of the surgical imaging system, the voice commands including instructions for performing spatial positioning functions; as well as The spatial positioning function is triggered based on the voice command, according to the position of the digital pointing indicator relative to the field of view of the surgical imaging device.

2. The system of claim 1, wherein the system is configured to: acquire image data of the surgical imaging device (120); generate a view of the surgical site overlaid with the overlay image; and provide the view of the surgical site having the overlay image to the display device (130a, 130b) of the surgical imaging system.

3. The system according to any one of claims 1 or 2, wherein the system is configured to obtain the image data stream from the surgical imaging device (120) of the surgical imaging system.

4. The system according to claim 1 or 2, wherein the system is configured to acquire the image data stream from the panoramic camera (150) of the surgical imaging system.

5. The system according to any one of claims 1 to 4, wherein the spatial positioning function is one of the following: a focusing function, a function for setting a marker, a function for changing the field of view, or a function for performing a measurement using a measuring device (160).

6. The system according to any one of claims 1 to 5, wherein the system is configured to determine two positions of the digital pointing indicator relative to the field of view of the surgical imaging device of the surgical imaging system if the voice command instructs to perform the spatial positioning function in a region, determine the region based on the two positions and based on the orientation of the field of view of the surgical imaging device, and trigger the performance of the spatial positioning function in the region.

7. The system according to any one of claims 1 to 6, wherein the system is configured to determine the position of the digital pointing indicator such that the digital pointing indicator (30) is offset from the portion of the surgical tool (120), as perceived by a user viewing the digital pointing indicator superimposed on the view of the surgical site obtained by the surgical imaging device.

8. The system according to any one of claims 1 to 7, wherein the system is configured to determine the position of the digital pointing indicator such that the digital pointing indicator is constrained to the central portion (40) of the field of view of the surgical imaging device.

9. The system of claim 8, wherein the system is configured to determine the position of the digital pointing indicator such that the digital pointing indicator is constrained to a rectangular portion (40) of the field of view, the rectangular portion being located at the center of the field of view of the surgical imaging device.

10. The system according to any one of claims 8 or 9, wherein the central portion of the field of view covers at least 30% and / or at most 90% of the area of ​​the field of view of the surgical imaging device.

11. The system according to any one of claims 1 to 10, wherein the system is configured to determine the position of the digital pointing indicator based on the relative movement of the portions of the surgical tool.

12. The system according to any one of claims 1 to 11, wherein the system is configured to determine, based on the movement of the surgical tool after activation of the digital pointing indicator, that the surgical tool is being used to control the digital pointing indicator if multiple surgical tools are visible in the field of view of the surgical imaging device. And / or wherein the system is configured to include a visual indicator representing the voice command in the overlay after processing the voice command. And / or the system is configured to determine the position of one of the ends of the surgical instrument, the blade, or the tube opening.

13. A surgical imaging system (100) comprising the system according to any one of claims 1 to 12.

14. A method for use in a surgical imaging system (100), the method comprising: Obtain the image data stream of the imaging device (120, 150) of the surgical imaging system; The image data stream is processed to track the position of a portion of the surgical tool (20); The position of the digital pointing indicator (30) relative to the surgical imaging device (120) of the surgical imaging system, showing the field of view (10) of the surgical site, is determined based on the change in the position of the part of the surgical tool. Generate an overlay image, the overlay image including the digital pointing indicator to be overlaid on the field of view of the surgical imaging device; Provide the overlay image with the digital pointing indicator to the display device (130a, 130b) of the surgical imaging system; The system obtains voice commands from the user of the surgical imaging system, the voice commands including instructions for performing spatial positioning functions; as well as The spatial positioning function is triggered based on the voice command, according to the position of the digital pointing indicator relative to the field of view of the surgical imaging device.

15. A computer program having program code for executing the method of claim 14 when the program is executed on a processor.

Images