Surgical support system and data visualization method for surgical procedures

Abstract

The present disclosure relates to a surgical assistance system (1) having a navigated surgical robot (2) for use in a surgical procedure on a patient, the system comprising: a robot base (4) as a local attachment point for the robot (2); a movable robot arm (8) connected to the robot base (4) and including at least one robot arm segment (10, 12); a visualization system (18) connected to the robot arm (8) and configured to generate and provide at least one real-time in-vivo image (42, 46); a navigation system (26, 30) configured to identify at least one position of the visualization system (18) and thus the at least one real-time in-vivo image (42, 46) with spatial reference to the patient; and a control device (24) configured to assign and store positions in a patient coordinate system (58) to the real-time in-vivo image (42, 46). Furthermore, the present disclosure relates to a data visualization method and a computer-readable storage medium according to the independent claims.

Description

[Technical Field]

[0001] The present disclosure relates to a surgical assistance system, particularly a neurosurgical assistance system, that includes a surgical robot with navigation capabilities for use in surgical procedures on a patient. The robot has a robot base as the robot's local attachment point, in a sense as a local fixed coordinate system. A movable robot arm including at least one robot arm segment is connected to the robot base.

[0002] Furthermore, the assistance system comprises a visualization system with one or more cameras connected to, in particular mounted on, the robotic arm, in particular on the distal end of the robotic arm, the visualization system being configured to generate at least one real-time body image, preferably an internal image, of the patient and to provide it preferably in digital form.

[0003] By using the navigation system of the assistance system, at least the position information, i.e., the position and orientation, of the visualization system and thus of at least one real-time body image can be determined with spatial reference to the patient (with respect to the patient's coordinate system). Typically, manually operated surgical instruments or surgical instruments operated by a navigated surgical robot are also provided, in which case their position relative to the patient can also be preferably tracked by the navigation system.

[0004] Furthermore, a control device is provided, which is at least configured to assign and store a position in a patient's coordinate system to at least one real-time in-vivo image, and which is preferably configured to control the subsystems of the above-mentioned assistance system, i.e. at least the robotic arm, the visualization system, and the navigation system, and to control their interaction. [Background technology]

[0005] In the fields of medicine and medical technology, automation through the integration of digitally controllable technical devices is becoming increasingly important. Robots are increasingly being used in surgical operations, particularly to assist in precise and minimally invasive procedures. In this context, robots are not only intended as standalone robots that perform surgery alone, but are increasingly being used as collaborative robots (cobots), i.e., robots that assist or assist medical personnel, especially surgeons, in the surgical field, interacting directly with them.

[0006] Typically, such robotic-assisted surgery, especially neurosurgery on the brain, involves the use of various visualization and measurement systems for visual, physiological, and functional assessment of the surgical site, as well as for decision-making aids. Classical data collected pre- and intraoperatively include, for example, pre-operative MRI and CT images of the patient, surgical instrument position data provided by navigation systems, images from multimodal digital microscopes, data from the robot for precise positioning of surgical instruments and visualization systems, electrophysiological monitoring data, histological samples, intraoperative ultrasound images, etc.

[0007] Existing surgical support systems, especially their navigation systems, can only abstract a limited amount of the above data. Preoperative patient scans are used as a common reference. State-of-the-art navigation systems can be used to locate and navigate digital microscopes, thereby integrating and simultaneously displaying the positions of surgical instruments and real-time images (live images) acquired by the microscope during surgery.

[0008] On the other hand, current robotic visualization systems allow the position of the robot arm to be linked to the image of a microscope permanently attached to the robot's end effector, allowing these systems to accurately reproduce the conditions under which the microscope image was recorded.

[0009] An important feature of this robot visualization system is also called a "waypoint." A waypoint defines and stores a specific pose of at least the robot arm and each of its segments relative to one another, for example, in the form of a set of joint angles. A stored waypoint can be recalled by a user to accurately return the robot arm to the stored pose and accurately reproduce the visualization conditions of the associated camera represented by the waypoint.

[0010] Additionally, it is possible to rotate or translate the visualization system around a memorized point on the patient, known as a "lock target." However, existing waypoint features between visualization and surgical navigation systems are not currently integrated, resulting in inconsistencies in current image and instrument navigation. As a result, existing waypoint features cannot compensate for unwanted patient movement or position changes. This complicates the surgeon's task, requiring them to track data from the visualization and navigation systems using separate displays on different systems.

[0011] This is further complicated by the fact that additional external data related to the surgery must be tracked by additional systems. These data are, for example, recorded pre- or intraoperatively, cannot be visualized in real time, require external pre-processing, or reside outside the surgical field. Typical examples of this are histological data (location-dependent or location-independent) or electrophysiological data (quality of local (electrochemical) signaling). These are not integrated into either the navigation or robotic positioning systems, significantly complicating their use and interpretation.

[0012] Furthermore, this data cannot be easily presented to the surgeon because, as mentioned above, multiple different systems and screens must be used. This is a major drawback of existing surgical support systems. Furthermore, it is known that data related to surgery can be organized and stored chronologically. However, the surgeon can only review this data in a post-operative environment based on the time it was recorded. Summary of the Invention

[0013] In contrast, an object of the present disclosure is to avoid or at least reduce the drawbacks of the prior art, and in particular to provide a surgical support system, a data visualization (computer-implemented) method including the surgical support system, and a computer-readable storage medium that enable a user to interpret surgery-related data during surgery in a very simple, intuitive, and safe manner. In particular, one of the challenges is to process, integrate, and display different modalities of medical information so that they can be clearly displayed on a single display device as needed, and to display integrated information related to a specific tissue region of a patient, as well as that exact tissue region.

[0014] The problem according to the present invention is solved by the features of claim 1 for the surgical support system, by the features of claim 7 for the data visualization (computer-implemented) method, and by the features of claim 15 for the computer-readable storage medium. The basic idea of ​​the present invention is to construct a surgical support system that enables the spatial arrangement or allocation of surgery-related data of any data modality and is configured to display this data in a spatially integrated manner.

[0015] In the present disclosure, data modality refers to the respective data acquisition method. Data modality may be, for example, imaging-based or functional-based, patient-related or assistance system-related, and / or real-time, pre-operative, or post-operative. This spatial location of data is preferably enabled with a spatial reference to the patient, particularly directly to the patient's anatomy, and more preferably with reference to the patient coordinate system. This collaborative location and display according to the present disclosure allows for easier, more intuitive, and safer interpretation of all surgically relevant data by the user during surgery, regardless of data modality.

[0016] In other words, a surgical assistance system is provided that is configured to integrate or aggregate surgery-related data of different data modalities recorded intraoperatively, preoperatively, and / or postoperatively. A surgical navigation system of the assistance system is provided for spatially locating and visualizing the heterogeneous data based on the different data modalities. The data are related to each other by their relative positions with respect to the patient, in particular by the patient's coordinate system.

[0017] Specifically, the surgical support system, particularly the neurosurgery support system, includes a navigated surgical robot used to perform surgery on a patient, the robot having a robot base as a local attachment point for the robot, and a movable robot arm connected to the robot base and including at least one robot arm segment.

[0018] A visualization system (or visualization device) is connected to, and in particular mounted on, the robotic arm, in particular the distal end of the robotic arm, and is configured to generate at least one real-time internal image of the patient and thereafter make it available in digital or computer-readable format.

[0019] Furthermore, the assistance system has a navigation system and is configured to determine at least position information, i.e., position and orientation, of the visualization system with spatial reference to the patient, and thus to determine position information of at least one real-time intra-body image (e.g., via conversion from the visualization system to an image using depth information).

[0020] To this end, the navigation system comprises in particular a 3D camera and a rigid body (with markers) tracked by the camera, at least one of which is rigidly coupled to the visualization system, the position of which can in particular be determined by triangulation of the 3D image of this rigid body, preferably performed in a control unit of the assistance system.

[0021] Alternatively or additionally, the navigation system may be equipped with a detection device for detecting the joint angles of the robot arm (i.e., the configuration of the robot arm segments), thereby allowing the position information of the visualization system to be determined via the control device if the kinematics of the robot arm are known. Preferably, the assistance system also comprises at least one surgical instrument operated manually or by a navigated surgical robot, the position information of which, in particular the position information of the end effector of the instrument, is tracked by the navigation system in relation to the patient.

[0022] The control device of the assistance system is specially configured to assign at least one position, in particular position information in the patient's coordinate system, to at least one real-time in-vivo image, preferably storing the position information in a data information system of the control device, the at least one real-time in-vivo image having a first data modality resulting from its acquisition method.

[0023] According to the invention, the control device is configured to import other surgery-related data, preferably of the patient and / or of the assistance system, acquired in a second data modality different from that of the visualization system via the data interface into the data information system, which data of the second data modality can be recorded preoperatively and / or in real time (and / or postoperatively, for example to control the surgery).

[0024] Thus, at least one real-time internal image of the patient and other surgery-related data of any data modality are stored in the data information system. Furthermore, the control device is configured to assign a position in the patient's coordinate system to the captured data and store it in the data information system. Figuratively speaking, the data having the second data modality are spatially assigned to the (virtual) patient. This means that all data related to the surgery are assigned a spatial position in the patient's (centered) coordinate system.

[0025] Thus, the data information system forms the basis for a joint spatial representation of both images and other data, regardless of the data modality, so that even data that is not spatially located on the patient, such as operational data from a robot, or non-real-time patient data, such as pre-operative scans or functional measurement data, is assigned at least a spatial location on the patient.

[0026] Finally, the control device is configured to generate a view of the at least one real-time in-vivo image together with the captured data and output this view via a display device, in particular a surgical monitor, of the assistance system. Preferably, the control device is configured to generate a view of the at least one real-time in-vivo image together with only that of the captured data that is spatially located within the at least one real-time in-vivo image based on its position.

[0027] Thus, in particular, the central coordinate system creates a central reference point where all medical data from different modalities can be integrated, allowing, for example, central processing of different information, synergistic views (overlay, augmented reality, etc.), links or correlations, and central output of the patient's important medical information to the surgeon.

[0028] This provides a solution for displaying surgery-related data that is not patient (internal) recording or image data in the same field of view as live images, recordings, or other image data based on its data modality, which is displayed in the best reference frame, i.e., the patient's body. Thus, output or display of surgery-related data with different data modalities can be displayed on a single co-display device.

[0029] The benefit to the surgeon is that he no longer needs to look at multiple displays, but can see all procedure-related data on a single display. Both the output of data on a single display and the spatial representation of all procedure-related data on the patient allow for simpler, more intuitive, and safer interpretation of procedure-related data during surgery. This leads to safer surgery and shorter surgery times.

[0030] Thus, a surgical assistance system has been created that avoids or at least mitigates the above-mentioned shortcomings of the prior art. In this disclosure, the term "end effector" refers to an instrument, tool, or similar medical device that can be used to perform surgery on a patient. In particular, instruments, medical devices such as endoscopes or suction tubes, optical instruments with visualization axes, pointers with distal ends for surgical navigation, etc. may be considered end effectors.

[0031] Here, the term "robot arm segment" refers to a robot part of a robot arm, in particular mounted between a bearing or joint, or in the case of an end robot arm segment, in particular connected in series between the end effector and the previous robot arm segment (or correspondingly to the robot base, in the case of only one robot arm segment).

[0032] The term "position" refers to a geometric location in three-dimensional space, in particular defined by coordinates in a Cartesian coordinate system. In particular, a position can be defined by three coordinates X, Y, Z. On the other hand, the term "attitude" refers to a configuration in space (e.g., at a position). It can also be said that an attitude defines a configuration in three-dimensional space, each indicating a direction and a rotation. In particular, an attitude can be defined by three angles.

[0033] The term "position information" encompasses both position and orientation. In particular, position information can be defined by six coordinates: three position coordinates X, Y, Z and three angular coordinates representing the orientation. Advantageous embodiments are set out in the dependent claims and are particularly described below.

[0034] According to a preferred embodiment, the visualization system comprises an overview camera with a field of view configured to image the intervention area and its surroundings, and / or a microscopic camera (particularly with magnification and zoom capabilities) with a field of view configured to image the surface of the intervention site or tissue, and / or an endoscopic camera with a field of view configured to image the subsurface images of the body of the intervention area, in particular.

[0035] According to a preferred embodiment, the assistance system is configured to register the patient via the navigation system, thereby configuring the navigation system to navigate the instrument and / or the visualization system. According to a preferred embodiment, any of the above-mentioned components or cameras of the visualization system are calibrated with reference to the patient's coordinate system. In particular each specific parameter (of the visualization device) such as focal length, zoom, distortion coefficients etc. are preferably stored in the data information system together with each image.

[0036] The data recorded using the second data modality are preferably image data or scan data of the patient acquired preoperatively, in particular CT scans and / or MRI scans, and / or histological images and / or 3D images. Alternatively or additionally, patient measurement data, in particular electrophysiological data, and / or surgical robot operation data, in particular the set of joint angles of the robot arm, or information about the currently used end effector, its nominal data, or operation parameters such as torque, speed, etc. are recorded.

[0037] For capturing electrophysiological measurement data, the support system according to a preferred embodiment has a serial interface. For capturing histological measurement data, a DICOM interface is preferably provided. To avoid overloading the view with data, in a further refinement, the control device is configured to continuously determine the coordinates of the field of view of at least one real-time in-vivo image in the patient's coordinate system and compare them with the positions assigned to the captured data (second data modality). The control device is then configured to output only the captured data whose positions lie within the coordinates of the field of view. In particular, the surgeon can use input operations to display or hide specific modalities, such as histological data.

[0038] In a preferred further refinement, the control device may be configured to permanently store the real-time recorded data set of the assistance system in a data information system so that it can be recalled at a later time. In the present disclosure, the term "recall" is understood to mean restoring the state of the assistance system, in particular the pose or position of the robot arm, that was stored together with the data set.

[0039] Preferably, the dataset includes at least one real-time navigated pose or position of the robot, preferably a set of its joint angles, and / or at least one real-time in-vivo image and its position information, and / or a real-time navigated position of the instrument, preferably in the patient's coordinate system. If the dataset includes only the poses or positions of the robot and the instrument, and the real-time in-vivo image and its position information, it is a monomodal dataset, since it includes only at least one image in the first data modality related to the patient.

[0040] When this data set is recalled, the stored pose or position of the robot and / or instruments is reset and / or at least one real-time intra-body image is again output / displayed. In this way, so-called "waypoints" of the assistance system can be recreated and utilized at any point during the procedure. According to a further refinement, the dataset is complemented by a selection of captured data recorded preoperatively and / or intraoperatively with a second data modality. In this case, the dataset is multimodal since it includes data about the patient in both the first and second data modality. The selection may include all captured data of the second data modality whose positions are within the field of view, or only a subset thereof.

[0041] A particular data set or "waypoint" can be recalled by approaching a navigated pose or position of the robot stored with that data set and triggering output of the remaining data in the data set by reaching that pose or position, or by selecting and / or activating the data set and triggering approach of the robot pose or position and output of the remaining data in the data set.

[0042] In order to be able to define said selection or subset and / or to avoid overloading said view with an excess of displayed data, the user interface of the assistance system is configured to allow for optional selection or deselection of elements of the dataset: in the case of selection, the element is output and / or assigned together with the dataset to be stored, and in the case of deselection, the element is not output and / or not assigned to the dataset to be stored.

[0043] Preferably, the control device is configured to set the focal position of the visualization system on the patient and to control the robotic arm to maintain the focal position in any orientation of the visualization system. To improve clarity of view and facilitate interpretation of data, according to a further embodiment, the control device is configured to output or display at least the real-time in-vivo images and / or the data collected in the second data modality in a hierarchical order depending on those data modalities and / or the size of the field of view and / or the viewing angle of the field of view and / or depending on the recording time.

[0044] According to a preferred further refinement, the control device, in particular the data information system, is configured to store the data sets in their chronological order and to recall them, thereby creating a history of the surgery and concretely outputting intermediate results of the surgery including data related to the surgery.

[0045] According to a preferred further refinement, the control device, in particular the data information system, is configured to identify or record deltas (differences) between stored data sets, in particular to perform measurements. Measurements can be performed in particular between images and / or captured data of the same data modality and the same position information or position, but at different recording times, in order to identify intraoperative progression. In particular, from a number of stored data sets, the distances or recording time differences between the data sets can be calculated automatically. This information can be useful for the surgeon to perform measurements on the patient, even if the patient's anatomy changes during surgery.

[0046] According to a preferred further refinement, the control device is configured to mark regions and / or coordinates in the real-time in-vivo images and / or in data recorded in a second data modality different from the visualization system, assign functions and / or properties and / or parameters to the marked regions and / or coordinates in the sense of functional mapping, and store the markings and assignments in the data information system. For example, histological examination results on a patient's CT scan can be optionally marked as tumor tissue or healthy tissue and displayed in different colors. Thus, a user can perform a real-time interpretation of the functional data and display it directly and intuitively on the patient's anatomical structure.

[0047] According to a preferred further refinement, the control device is configured to recreate the operating configuration on the digital twin of the assistance system in response to at least one stored data set. Since the poses or positions of the robot arm and the visualization system are stored in the respective data sets, this information can be used to recreate a particular recording device both in the real world and in the digital, in this respect virtual, twin.

[0048] The problem of the present disclosure is solved with respect to a data visualization method for a surgical assistance system according to the present disclosure, in that the method comprises the following steps: navigating, by the control device, a visualization system connected to and especially mounted on a robotic arm, in particular a distal end of the robotic arm, with spatial reference to a patient, by the visualization system, generating and providing, in particular in digital form, at least one real-time in-vivo image of the patient by the visualization system, determining, by the navigation system, position information, in particular the position and orientation, of the visualization system and thus of the at least one real-time in-vivo image with spatial reference to the patient, by the navigation system, assigning, by the control device, position information in the patient's coordinate system to the real-time in-vivo image and storing the position information and the image, importing, by the data interface, data recorded using a second data modality different from the visualization system into the data and information system, assigning positions in the patient's coordinate system to the imported data, storing, by the control device, the positions and the imported data in the data and information system, generating, via the control device, a view of the real-time in-vivo image together with a view of the imported data spatially arranged within the real-time in-vivo image based on their positions, and outputting the view on a display device, preferably a single display device, in particular a surgical monitor.

[0049] With regard to a computer-readable storage medium or a computer program, the problem is solved by a storage medium and a computer program, respectively, comprising instructions that, when executed by a computer, cause the steps of the method according to the present disclosure to be performed. Any disclosure relating to the surgical assistance system of the present disclosure also applies to the method of the present disclosure, and vice versa. [Brief explanation of the drawings]

[0050] The present invention will now be explained in more detail with reference to preferred embodiments and by means of figures. [Figure 1] 1 shows a diagram of a surgical assistance system according to a first preferred embodiment; [Figure 2]1 illustrates a multimodal data set of an assistance system according to a preferred embodiment. [Figure 3] 1 illustrates a method for data visualization of a multimodal dataset according to a preferred embodiment. [Figure 4] 1 shows a multimodal dataset according to a further preferred embodiment; [Figure 5] 1 shows a multimodal dataset according to a further preferred embodiment; [Figure 6] 1 shows a multimodal dataset according to a further preferred embodiment; [Figure 7] 1 shows a multimodal dataset according to a further preferred embodiment; [Figure 8] 1 shows a multimodal dataset according to a further preferred embodiment. The figure is schematic in nature and is intended to aid in understanding the invention. Identical elements are provided with identical reference numerals. Features of different embodiments may be interchanged. DETAILED DESCRIPTION OF THE INVENTION

[0051] FIG. 1 shows, in a schematic side view, a surgical assistance system 1 according to a preferred embodiment. The assistance system 1 has a surgical robot 2, which in the illustrated embodiment includes a locally fixed robot base 4. Alternatively, it may be implemented mobile to utilize the robot 2 in different locations within a hospital operating room as needed. In either case, the robot base 4 forms a local reference point to which is connected an articulated robot arm 8, which includes multiple robot arm segments 10, 12, each connected to one another via joints 14. In this manner, the robot arm segments 10, 12 are movable relative to one another, and the robot arm 8 can be controlled as a whole.

[0052] The visualization system 18 of the assistance system 1 is mounted on the distal end 16 of the robotic arm 8. In the embodiment shown, this comprises a surgical microscope 20 and an overview camera 22 as robotic end effectors. In addition to the embodiment shown, at least one medical instrument may be provided on the end segment 12 of the robotic arm 8 as a rigid or movable end effector. In the embodiment shown, the assistance system 1 comprises a manually operated instrument 32 that is tracked by the navigation system of the assistance system 1.

[0053] The position (position and orientation) of the visualization system 18, and thus the respective positions of the surgical microscope 20 and overview camera 22, can be controlled and coordinated by the robot arm 8. To this end, a specially configured controller 24, which is provided on the robot base 4 and will be described later, serves as a central controller. In order to move the robot arm 8 according to a user's request, the assistance system 1 has an input device (not shown), which is configured as, for example, a joystick or a 3D space mouse, and is configured to manually receive, via tactile manipulation, both translational control command inputs for three orthogonal axes and rotational control command inputs about these three axes, and transfer them to the control device 24 as computer-readable digital control signals and control commands, respectively, so that the control device 24 can actively control the robot arm 8 according to the control commands.

[0054] Furthermore, the assistance system 1 comprises a navigation system consisting of several components, including a navigation camera 26, which in the illustrated embodiment is implemented as an infrared stereo camera, a "tracker" 28 that is tracked by the navigation camera 26 and is fixedly connected to the patient, and a "tracker" 30 that is fixedly connected to the visualization system 18 and thus to the surgical microscope 20, the overview camera 22 and to the end segment 12 of the robotic arm 8. The cameras 20, 22 are optically calibrated and their relative positions to the tracker 30 are known and fixed, so that if the position information of the tracker 30 is known, the position information of the cameras 20, 22 can also be determined via the navigation system.

[0055] Additionally, the navigation system includes a "tracker" 34 that is tracked by the navigation camera 26 and fixedly connected to the manually operated instrument 32. The navigation camera 26 is signally connected to the controller 24 and transmits information about the tracked trackers 28, 30, 34 to the controller, which determines all necessary position information therefrom. Additionally, the navigation system includes angle sensors (not shown) at the joints 14, which record the joint angles 36 / joint configurations of the articulated robotic arm 8 in real time and transmit them to the controller 24 for navigation.

[0056] The patient, shown in Figure 1 for clarity with only a body-mounted tracker 28, is registered with reference to a CT or MRI scan 38, and the registration transformation 40 is stored in the controller 24. Thus, the navigation system is set up with reference to the patient's coordinate system, and the instruments 32 and visualization system 18 and their cameras 20, 22 can be navigated with reference to the patient's coordinate system.

[0057] During surgery, the surgical microscope 20 and the overview camera 22 acquire real-time internal images 42, 46 having respective fields of view 44, 48 and make them available in digital form to the control device 24. The data modality of the two images 42, 46 is identical and, according to the present disclosure, is referred to as a "first data modality." On the other hand, the patient scan 38 is recorded preoperatively, not in real time. Furthermore, the patient scan 38 has a different data modality from that of the visualization system 18, based on the fact that the data is not optically acquired, as is the case with the images 42 and 46. According to the present disclosure, any data modality different from the first data modality is referred to as a "second data modality." It may also be referred to as an "additional / supplemental / further data modality."

[0058] The multimodal data set of the assistance system 1 according to the present invention can alternatively be called "multimodal waypoints (MMW)" and, according to Fig. 1, includes position information 50 of the robot arm 8 determined from the tracking or joint angles 36 of the tracker 30 and position information of the surgical microscope 20 and overview camera 22 determined therefrom. Furthermore, it also includes position information of the instrument 32 determined from the tracking of the tracker 34 and position information of the patient determined from the tracking of the tracker 28.

[0059] The real-time data of the "first data modality" includes real-time in-vivo images 42, 46, their position information relative to the patient (tracker 28), and their fields of view 44, 48. The dataset of the assistance system 1 includes a patient scan 38 as pre-operatively recorded data having a "second data modality" different from the visualization system 18.

[0060] 2 shows in simplified form a multimodal dataset 60 of an assistance system according to a further preferred embodiment. The visualization system of the assistance system based on the dataset according to FIG. 2 is supplemented by a navigated endoscopic camera 52, in comparison with the assistance system according to FIG. 1. The dataset according to FIG. 2 therefore comprises additional real-time images of the body of the body in a "first data modality" and real-time images of the endoscope. Furthermore, the dataset is supplemented by histological image data 54 and electrophysiological measurement data 56, which represent external, non-real-time data in a "second data modality" different from the visualization system 18, which are not detected by the navigation system at the time of their generation and are therefore initially unnavigable.

[0061] As will be disclosed, these "second data modality" data, in this case histological image data 54 and electrophysiological measurement data 56, are also assigned respective locations in the patient's coordinate system 58, symbolized by the double arrow pointing to the coordinate system 58. Thus, a multimodal dataset (or MMW waypoint), as will be disclosed, consists of a set of heterogeneous surgical data organized in time and space according to a fixed frame of reference: the patient.

[0062] In order to visualize not only the images generated in real time and visualized live by the visualization system 18 (the "first data modality"), but also data that is not generated in real time, is not generally part of the image, and is therefore not initially navigable (the "second data modality"), specific location information or coordinates in the patient's anatomical structure, i.e., the patient's coordinate system 58, are assigned using the data visualization method of the present disclosure.

[0063] Figure 3 shows a data visualization method according to a preferred embodiment. Based on the dataset 60 according to Figure 2, the complete sequence is shown, starting from step 100 "Navigating the visualization system 18" and ending with step 800 "Outputting a view of the dataset 60". The starting point is the enrolled patient.

[0064] In a first step 100, the visualization system 18 is navigated to the surgical area with spatial reference to the patient. This can be done, for example, by interaction of the input devices and the control device described above. In step 200, internal overview images, surgical microscope images, and endoscopic images are generated continuously and in real time via the visualization system 18 and made digitally available to the control device 24. In step 300, the navigation system determines the location of the visualization system 18, and thus all of the real-time internal images, with spatial reference to the patient.

[0065] In step 400, the real-time in-vivo images are assigned their respective position information to the patient's coordinate system 58 by the control device and stored in the control device's data information system. It is assumed that in the further course of the surgery, data from a second data modality, for example digitally obtained results / images of a histological examination of a tissue sample previously taken from the patient, becomes available. Following this, in step 500, this data is first imported into the data information system via a data interface.

[0066] As described above, in step 600, this data is assigned a location in the patient coordinate system 58 and stored in the data information system. This can be done by the surgeon assigning them to a location in the real-time image, and thus on the patient, via a user interface, such as a touch screen or user interface on the input device or manually operated instrument 32. Alternatively, the data can first be centered or otherwise positioned in a predetermined manner within the real-time image and then moved by the surgeon.

[0067] However, if the data is recorded directly in the surgical field, such as electrophysiological measurement data in the surgical field of the brain, the surgeon can use a navigated instrument to specify, for example, the location where the data should be recorded, and then, via the recorded position information, the location of the data in the coordinate system 58 is directly identified and stored in the data information system.

[0068] Next, in step 700, a view of the real-time in-vivo image is generated in conjunction with a view of the captured data spatially positioned within the real-time in-vivo image based on the pre-assigned positions, and then in step 800, this view is output to a single display device, in particular a single surgical monitor. The multimodal datasets or waypoints thus generated and stored can of course be recalled and explored by the surgeon at any time from among the many stored multimodal datasets, and upon subsequent recall of the data, all data stored in the multimodal datasets are visualized jointly on the patient's body, regardless of data modality.

[0069] If the visualized data consists of stored images or is otherwise suitable for this purpose, the data can be visualized hierarchically from coarse (c) to fine (f), or vice versa. This is shown for data set 62 in Figure 4, where an image 46 from overview camera 22 with a wide field of view 48 is visualized on the left, an image 42 from surgical microscope 20 with a very small field of view 44 is visualized in the center, and a histological image 54 with the smallest field of view is visualized on the right.

[0070] 5 illustrates another possibility provided by the multimodal datasets disclosed herein, where the above-mentioned real-time images 42, 46 of the surgical microscope 20 and overview camera 20, as well as a real-time external image 62 of the surgeon, a stored histological image 54, and the location of a dataset or waypoint 64 are visualized within the context of a patient scan 38.

[0071] Thus, the surgeon can directly observe the location of the dataset or waypoint 64 on the patient scan 38 and, if desired, can select particular anatomical location information on the patient scan 38 to recall particular other datasets or waypoints that correspond to that anatomical location.

[0072] Alternatively, the surgeon may recall other data sets or waypoints by setting or recreating the recording conditions of the stored data sets or waypoints. In this case, the robot 2 and visualization system 18 are configured to correspond to the joint angles 36 or stored position information of the tracker 30 stored in the recalled data set or waypoint. This allows for accurate reproduction of the visualization conditions. This feature is particularly useful for comparing specific anatomical locations during the course of surgery.

[0073] As described above, the surgeon can recall any dataset or waypoint and, if desired, select which data from the dataset to display. Selected data is displayed, while unselected data is not. Thus, depending on the indication, the surgeon is free to select from all available images and other captured or functional data stored in the dataset or waypoint.

[0074] FIG. 6 shows an example of a multimodal dataset 66 in which only the real-time image 42 of the surgical microscope 20, the instrument 32 positioned therein, and the preoperative patient scan 38 in which the position of the instrument 32 is visualized are selected for display / output.

[0075] According to Figure 7, a number of multimodal data sets or waypoints 68, 70, 72, 74, 76 are generated and stored during surgery. The data sets or waypoints 68, 70, 72, 74, 76 can be used to create a complete picture of the surgery and clearly document it. They can be stored in a data information system according to the time they were recorded, i.e., according to Figure 7, as a function of time t, and displayed according to the spatial hierarchy inherent in the data.

[0076] This information is particularly useful for creating a comprehensive and meaningful surgical record. In contrast to image data from conventional assistance systems, the datasets or waypoints 68, 70, 72, 74, 76 of the present disclosure are organized both temporally and spatially. Furthermore, the datasets or waypoints 68, 70, 72, 74, 76 can naturally also be organized internally and hierarchically, as previously described with reference to FIG. 4.

[0077] FIG. 8 illustrates the use of multimodal datasets or waypoints 78 for functional mapping. Surgeons can use datasets or waypoints 78 for live interpretation of functional data, which can be intuitively displayed on the patient's anatomy. As an example, FIG. 8 illustrates the results of a histological examination directly mapped to the patient's anatomy. In practice, the mapping can be done using, for example, color markings. For example, areas marked with a T indicate tumor tissue, and areas marked with a G indicate healthy tissue. This functionally mapped multimodal dataset or waypoints 78 contributes to optimal tumor resection. [Explanation of symbols]

[0078] 1. Surgical support system 2. Robot 4. Robot Base 8. Robotic Arm 10, 12 robot arm segments 14 Joints 16 Robot arm end 18 Visualization System 20 Surgical Microscope 22 Overview Camera 24 Control device 26 Navigation Camera 28 Patient Tracker 30 Visualization System Tracker 32 Surgical instruments 34 Surgical Instrument Tracker 36 Patient Scans 40 Registration Conversion 42 Images of a surgical microscope 44 Field of view of a surgical microscope 46 Overview Camera Images 48 Overview Camera Field of View 50 Robot arm position information 52 Endoscopy 54 Histological Data 56 Electrophysiological Data 58 Patient Coordinate System 60 multimodal datasets 62 Surgeon's Camera Images 64-78 Multimodal Datasets 100 Visualization System Navigation Process 200 Data creation and provision process of the first data modality 300 Location information identification process in visualization system 400 Location information allocation process 500 Data capture process for the second data modality 600 Location allocation process 700 View Generation Process 800 view output process

Claims

1. A surgical assistance system (1) including a surgical robot (2) with navigation capabilities for use in a surgical operation on a patient, a robot base (4) as a local attachment point for the robot (2) and a mobile robot arm (8) connected to said robot base (4) and including at least one robot arm segment (10, 12); a visualization system (18) connected to said robotic arm (8) and configured to generate and provide at least one real-time in-vivo image (42, 46); a navigation system (26, 30) configured to determine, with spatial reference to the patient, at least one position of the visualization system (18) and thus of the at least one real-time in-vivo image (42, 46); a control device (24) configured to assign and store position information in the patient's coordinate system (58) to the real-time in-vivo images (42, 46), The control device (24) - inputting data (38, 54, 56) recorded in a second data modality different from said visualization system into a data information system of said control device (24) via a data interface; - assigning at least a position, in particular position information, to the acquired data (38, 54, 56) in the patient's coordinate system (58) and storing it together with said position in the data information system; - a surgical support system (1), characterized in that it is configured to generate a view of the real-time in-vivo image (42, 46) together with a view of the captured data (38, 54, 56) spatially arranged within the real-time in-vivo image (42, 46) based on its position, and to output said view via a display device, in particular a surgical monitor.

2. A surgical support system (1) according to claim 1, A surgical support system (1), characterized in that the data (38, 54, 56) recorded in the second data modality is image data (38, 54) and / or measurement data (56) of the patient, and / or operation data of the surgical robot (2).

3. A surgical support system (1) according to any of the preceding claims, 1. The surgical assistance system (1), wherein the control device (24) is configured to permanently store real-time recorded datasets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) in the data and information system and / or to retrieve permanently stored, previously real-time recorded datasets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) from the data and information system.

4. A surgical support system (1) according to claim 3, The real-time recorded dataset (60, 64, 66, 68, 70, 72, 74, 76, 78) comprises at least one real-time navigated position of the robot (2), and / or at least one real-time intracorporeal image (42, 46) and its position, and / or preferably a real-time navigated position of an instrument (32), each defined in the patient coordinate system (58).

5. A surgical support system (1) according to claim 3 or 4, A surgical support system (1) characterized by a user interface configured to selectively select elements of said data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) for output or deselect them from said output.

6. A surgical assistance system (1) according to one of the preceding claims, 1. The surgical assistance system of claim 1, wherein the control device is configured to output at least the in-vivo images of the visualization system and / or the data recorded in the second data modality in an ordered manner depending on the data modality and / or the size of the field of view and / or the viewing angle of the field of view and / or the recording time.

7. A method for data visualization during surgery, in particular neurosurgery, on a patient, in particular for a surgical assistance system (1) according to any of the preceding claims, comprising: (100) navigating, by a control device (24), a visualization system (18) connected to a robotic arm (8) with spatial reference to the patient; (200) producing and providing at least one real-time in-vivo image (42, 46), in particular in digital form, by means of said visualization system (18); (300) determining, by the navigation system (26, 30), the position information of the visualization system (18) and thus of the at least one real-time in-vivo image (42, 46) with spatial reference to the patient; (400) assigning, by a control device (24), position information in the patient's coordinate system (58) to the real-time in-vivo images (42, 46) and storing the position information and the images (42, 46) in a data information system of the control device (24), (500) importing data (54, 56) recorded in a second data modality different from that of the visualization system (18) into the data information system by means of a data interface of the control device (24); (600) assigning, by said control device (24), a position in said patient's coordinate system (58) to said captured data (54, 56) and storing said position and said captured data (54, 56); (700) generating, via the control device (24), views of the real-time in-vivo image (42, 46) together with views of the captured data (54, 56) spatially positioned within the real-time in-vivo image (42, 46) based on their positions; (800) outputting said view via a single, preferably exactly single, display device, in particular a surgical monitor.

8. The data visualization method according to claim 7, - permanently storing in said data information system, by means of an operating interface and said control device (24), the real-time recorded data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of said assistance system (1); and / or - retrieving, by means of an operating interface and the control device (24), from the data and information system, permanently stored and previously recorded real-time data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1), said data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) comprising at least one real-time navigated position of the robot (2), preferably its joint angles (36), and / or said at least one real-time intra-body image (42, 46) and its position information, and / or said real-time navigated position of the instrument (32), in each case defined in the coordinate system (58) of the patient.

9. The data visualization method according to claim 8, The method further comprises: - a step of retrieving the permanently stored and previously real-time recorded data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) from the data information system, which is triggered when the assistance system (1), in particular the robot arm (8), via the control device (24), moves to the position of the stored data sets (60, 64, 66, 68, 70, 72, 74, 76, 78).

10. 10. The data visualization method according to claim 8, - a data visualization method characterized by a step of assigning, by means of a user interface and said control device (24), a selection of data recorded preoperatively and / or intraoperatively with a second data modality to said stored data sets (60, 64, 66, 68, 70, 72, 74, 76, 78).

11. The data visualization method according to any one of claims 8 to 10, - a data visualization method characterized by the step of displaying the location of said stored data sets (64) in said patient scan (38) or image.

12. The data visualization method according to any one of claims 8 to 11, - A data visualization method characterized by the step of creating a history of a data set (68, 70, 72, 74, 76, 78).

13. The method according to any one of claims 8 to 12, A data visualization method further characterized by a step of measuring differences between images and / or scans of the same data modality and the same positional information, but recorded at different times.

14. The method according to any one of claims 7 to 13, The method further comprises: - marking regions and / or coordinates in the real-time in-vivo images and / or in data recorded in the second data modality different from the visualization system; - assigning functions and / or properties and / or parameters (G, T); - storing said markings and assignments in said data information system.

15. 1. A computer-readable storage medium, comprising: A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the data visualization method according to any one of claims 7 to 14.

Images