Surgical assistance system, and data visualisation method for a surgical procedure

Abstract

The present invention relates to a surgical assistance system (1) comprising a navigated surgical robot (2) for use in a surgical procedure on a patient, said system comprising: a robot base (4) acting as a local connection point of the robot (2), and a movable robotic arm (8) that is connected to the robot base (4) and has at least one robotic arm segment (10, 12); a visualisation system (18) which is connected to the robotic arm (8) and is designed to create and provide at least one up-to-date intracorporeal image (42, 46); a navigation system (26, 30) which is designed to determine, with spatial reference to the patient, at least one position of the visualisation system (18) and thus a position of the at least one up-to-date intracorporeal image (42, 46); and a control unit (24) which is designed to assign a position in a coordinate system (58) of the patient to the up-to-date intracorporeal image (42, 46) and to store said position. The present invention also relates to a data visualisation method and to a computer-readable storage medium according to the dependent claims.

Description

[0001] Surgical assistance system and method for data visualization for a surgical procedure

[0002] Description

[0003] Technical area

[0004] The present disclosure relates to a surgical assistance system, in particular a neurosurgical assistance system, comprising a navigated surgical robot for use in a surgical procedure on a patient. The robot has a robot base as the robot's local connection point and, to a certain extent, as a local, stationary coordinate system. A movable robot arm with at least one robot arm segment is connected to the robot base. Furthermore, the assistance system has a visualization system with one or more cameras, which is connected, in particular mounted, to the robot arm, in particular to a terminal side of the robot arm. The visualization system is adapted to create at least one up-to-date, corporal, preferably intracorporeal, image of the patient and to provide it, preferably digitally.A navigation system of the assistance system can determine at least the location, i.e., a position and orientation, of the visualization system, and thus the location of at least one up-to-date corporal image, with spatial reference to the patient. Typically, a surgical instrument that is manually guided or guided by a navigated surgical robot is also provided. If this is the case, its position relative to the patient can preferably also be tracked by the navigation system. Furthermore, a control unit is provided. This is at least adapted to assign and store a position in a patient coordinate system to the at least one up-to-date intracorporeal image.Preferably, it is also adapted to control the aforementioned subsystems of the assistance system—that is, at least to control the robot arm, the visualization system, and the navigation system—and to control their interaction. Technical background.

[0005] In the field of medicine and medical technology, automation with the associated integration of digitally controllable technical devices is becoming increasingly important. Robots are increasingly being used in surgical procedures, particularly to support precise, minimally invasive procedures. In this case, the robot is not only intended as a standalone robot that solely performs the operation, but is increasingly being used as a collaborative robot (cobot), i.e., as an assisting or supporting robot directly in the surgical field, interacting with medical professionals, especially the surgeon.

[0006] For such robot-assisted surgical procedures, especially neurosurgical interventions on the brain, various visualization and measurement systems are typically used for visual, physiological, and functional assessment of the surgical site and as decision support. Typical data collected before and during the procedure include, for example, preoperative MRI and CT scans of the patient, position data of the surgical instruments provided by the navigation system, images from a multimodal digital microscope, data from the robot for precise positioning of the surgical instruments and the visualization system, electrophysiological monitoring data, histological samples, intra-operative ultrasound images, and the like.

[0007] Existing surgical assistance systems, especially their navigation systems, can only summarize a limited amount of some of the data listed above. They use preoperative patient scans as a common reference. The most advanced navigation systems can be used to localize and navigate the digital microscope, so that the position of the surgical instruments and the real-time images (live images) acquired by the microscope during the operation are merged and displayed together. On the other hand, current robotic visualization systems allow the position of a robotic arm to be linked to the images of a microscope that is permanently connected to the robot's end effector. These systems thus make it possible to accurately reproduce the acquisition conditions of microscopic images. This important feature of robotic visualization systems is often referred to as a "waypoint."The waypoint is defined and stored as at least a specific position of the robot arm and its robot arm segments relative to each other, for example, in the form of a set of joint angles. Saved waypoints can be recalled by the user, whereby the robot arm is precisely adjusted to the saved position and the visualization condition of the respective camera represented by the waypoint is precisely reproduced. Furthermore, rotation or translation of the visualization system around a saved point on the patient, the so-called "locked target," is possible. However, the integration of the existing waypoint functionality for the visualization system and the surgical navigation system is not currently available, so navigation of the current image and the instrument is inconsistent.As a result, the existing waypoint functionality is not able to compensate for unwanted movement or position changes of the patient.

[0008] For the surgeon, this presents the challenge that the data from the visualization system and the navigation system must be monitored on different systems, each with separate displays. Further complicating matters is the fact that additional external, operation-relevant data that cannot be visualized in real-time—for example, because it was recorded preoperatively, or during the operation and subsequently processed externally—or that is located outside the surgical area, for example, must be monitored on additional systems. Typical examples of this are histological data (location-specific or non-location-specific) or electrophysiological data (quality of local (electrochemical) signal transmission). These are integrated neither into the navigation system nor into a robot positioning system, which significantly complicates their availability and interpretation.Furthermore, this data cannot be easily presented to the surgeon because, as mentioned, several different systems and screens must be used. This is a major disadvantage of existing surgical assistance systems. It is also known to organize and store surgically relevant data along a timeline. However, the surgeon can only examine this data in a postoperative setting, based on the time it was collected.

[0009] Summary of the present disclosure

[0010] In contrast, the object of the present disclosure is to avoid or at least mitigate the disadvantages of the prior art and, in particular, to provide a surgical assistance system, a (computer-implemented) method for data visualization using a surgical assistance system, and a computer-readable storage medium that allows a user to interpret operation-relevant data during the operation in a particularly simple, intuitive, and secure manner. One object can be seen, in particular, in processing, consolidating, and presenting various modalities of medical information in such a way that they can be clearly displayed on a single display as needed, wherein, in particular, for a specific tissue area of ​​the patient, the correspondingly relevant consolidated information for precisely this tissue area is also displayed.

[0011] The objects are achieved according to the invention with regard to a surgical assistance system by the features of claim 1, with regard to a (computer-implemented) method for data visualization by the features of claim 7 and with regard to a computer-readable storage medium by the features of claim 15.

[0012] A basic idea of ​​the invention is to create a surgical assistance system that is adapted to enable the spatial arrangement or assignment of operation-relevant data of any data modality and to spatially display this data together. The data modality, as defined in the disclosure, describes a respective method for data collection. The data modality can, for example, be imaging or functional, on the patient or on the assistance system and / or up-to-date or preoperative or postoperative. The spatial arrangement of this data is preferably enabled with a spatial reference to the patient, in particular directly to the patient's anatomy, preferably with reference to a coordinate system of the patient. This joint arrangement and display according to the disclosure allows for simpler, more intuitive, and reliable interpretation of all operation-relevant data by a user during the operation, regardless of the data modality.

[0013] In other words, a surgical assistance system is provided that is adapted to combine or aggregate operation-relevant data from different data modalities, which are acquired during, before, and / or after an operation. A surgical navigation system of the assistance system is provided to spatially localize and spatially visualize the data, which is heterogeneous due to their different data modalities. The data are linked by their relative position to the patient, in particular to the patient's coordinate system.

[0014] Specifically, a surgical assistance system, in particular a neurosurgical assistance system, comprises a navigated surgical robot for use in a surgical procedure on a patient. The robot has a robot base as the local connection point of the robot and a movable robot arm connected to the robot base, with at least one robot arm segment. A visualization system (or a visualization unit) is connected, in particular mounted, to the robot arm, in particular to a terminal side of the robot arm. This visualization system is adapted to create at least one up-to-date intracorporeal image of the patient and subsequently provide it in digital or computer-readable form.The assistance system also has a navigation system that is adapted to determine at least one location, i.e., a position and orientation, of the visualization system with spatial reference to the patient, and thus (for example, via a transformation from visualization system to image, e.g., using depth information) the location of at least one current, intracorporeal image. For this purpose, the navigation system can, in particular, have a 3D camera and rigid bodies (with markers) tracked by the camera, at least one of which is permanently coupled to the visualization system. The location of the visualization system can thus be determined, in particular, via triangulation of 3D images of this rigid body, which preferably takes place in a control unit of the assistance system.Alternatively or additionally, the navigation system can have a detection unit for detecting joint angles of the robot arm (i.e., the configuration of the robot arm segments), wherein the position of the visualization system can in turn be determined via the control unit, given otherwise known kinematics of the robot arm. Preferably, the assistance system also has at least one surgical instrument that is guided manually or by the navigated surgical robot, the position of which instrument, in particular a position of an end effector of the instrument, is tracked with respect to the patient by the navigation system. A control unit of the assistance system is specifically adapted to assign at least one position, in particular a position in a coordinate system of the patient, to the at least one up-to-date intracorporeal image and to store this position, preferably in a data information system of the control unit.The at least one up-to-date, intracorporeal image has a first data modality resulting from the acquisition method. According to the invention, the control unit is adapted to import further operation-relevant data, preferably of the patient and / or the assistance system, which is acquired using a second data modality different from the visualization system, into the data information system via a data interface. The data of the second data modality can be acquired preoperatively and / or up-to-date (and / or even postoperatively, for example, to monitor the operation). Thus, the at least one up-to-date, intracorporeal image of the patient and other operation-relevant data of any data modality are located in the data information system. The control unit is further adapted to assign a position in the patient's coordinate system to the imported data and to store it in the data information system.Figuratively speaking, the data is spatially attached to the (virtual) patient using a second data modality. Thus, all surgery-relevant data is assigned a spatial position in the patient's (central) coordinate system. The data information system thus forms a basis for the joint, spatial representation of both the image(s) and the other data, regardless of their data modality. Even data that is not spatially located on the patient, such as robot operating data, or non-current patient data, such as preoperative scans or functional measurement data, is thus assigned at least one spatial position on the patient. Finally, the control unit is adapted to generate a view of the at least one current, intracorporeal image together with the imported data and to output this view via a display device of the assistance system, in particular an OR monitor.Preferably, the control unit is adapted to generate this view of the at least one current intracorporeal image only with those of the imported data which, due to their position, are spatially arranged within the at least one current intracorporeal image.

[0015] In particular, the one central coordinate system creates a central reference into which all medical data from different modalities can be integrated, for example to process different information centrally and, for example, to achieve synergistic views (such as overlays, augmented reality and the like), links or correlations and to provide a surgeon with the important medical information of the patient centrally.

[0016] In this way, a solution has been found to place those operation-relevant data which, due to their data modality, are not (intracorporeal) recordings or image data of the patient, into the same field of view of the surgeon as live images, recordings, or other image data. They are displayed in / on the best possible reference system for this purpose, namely the patient's body. This means that the operation-relevant data, which differ in their data modality, can be output or displayed on a single, common display device. The effect for the surgeon is that they no longer have to look at several displays, but only at a single one, in order to see all operation-relevant data.Both the output of data on a single display and the spatial representation of all surgically relevant data on the patient enable simpler, more intuitive, and safer interpretation of surgically relevant data during the operation. This leads to a safer operation and a reduction in procedure time.

[0017] Thus, a surgical assistance system has been created that avoids or at least mitigates the above-mentioned disadvantages of the state of the art.

[0018] In the present disclosure, the term "end effector" refers to a device, instrument, or similar medical device that can be used to perform a procedure on a patient. In particular, end effectors can be considered to include: an instrument, a medical device such as an endoscope or a suction tube, an optical device with a visualization axis, a pointer with a distal tip for surgical navigation, and others.

[0019] The term “robot arm segment” here means in particular a robot part of the robot arm mounted between bearings or joints or, in the case of the terminal robot arm segment, in particular a robot part connected in series between the end effector and the preceding robot arm segment (in the case of only one robot arm segment, the robot base).

[0020] The term "position" refers to a geometric position in three-dimensional space, which is specified in particular using coordinates of a Cartesian coordinate system. In particular, the position can be specified by the three coordinates X, Y, and Z.

[0021] The term "orientation," in turn, indicates an alignment (e.g., position) in space. One could also say that orientation indicates an orientation with a direction or rotational indication in three-dimensional space. In particular, orientation can be specified using three angles.

[0022] The term "position" encompasses both a position and an orientation. In particular, the position can be specified using six coordinates: three position coordinates X, Y, and Z, as well as three angular coordinates for the orientation. Advantageous embodiments are claimed in the subclaims and are explained in particular below.

[0023] According to a preferred embodiment, the visualization system can comprise an overview camera with a field of view that is provided and designed to capture the intervention area and its surroundings, and / or a microscope camera (in particular with magnification with a zoom function) with a field of view that is provided and designed to capture a surface of the intervention or a tissue, and / or an endoscope camera with a field of view that is provided and designed to capture intracorporeally, in particular below a surface of the intervention area.

[0024] According to a preferred embodiment, the assistance system can be adapted to register the patient via the navigation system, so that the navigation system is configured to navigate the instruments and / or the visualization system.

[0025] Any of the above-mentioned components or cameras of the visualization system may, according to a preferred embodiment, be calibrated with respect to the patient's coordinate system.

[0026] Respective intrinsic parameters (of the visualization units), such as in particular a focal length, a zoom, distortion coefficients and the like, can preferably be stored together with the respective images in the data information system.

[0027] The data acquired with the second data modality is preferably preoperatively acquired image data or scans of the patient, in particular a CT scan and / or an MRI scan and / or a histological scan and / or a 3D scan. Alternatively or additionally, patient measurement data, in particular electrophysiological data, and / or operating data of the surgical robot, such as in particular a set of joint angles of the robot arm or information about the currently used end effector, its nominal data, or operating parameters such as torque, speed, or the like, can be acquired.

[0028] According to a preferred embodiment, the assistance system can have a serial interface for importing electrophysiological measurement data. A DICOM interface can preferably be provided for importing histological measurement data.

[0029] To avoid overloading the view with data, the control unit is adapted in a further development to continuously determine the coordinates of the field of view of at least one current intracorporeal image in the patient's coordinate system and compare them with the positions assigned to the imported data (second data modality). The control unit is then adapted to output only those imported data whose positions are located within the coordinates of the field of view. In particular, the surgeon can display and hide certain modalities via an input, for example, displaying and hiding histological data.

[0030] In a preferred development, the control unit can be adapted to permanently store a currently acquired data set of the assistance system in the data information system and to be able to retrieve it later. The term "retrieving" is to be understood in the context of the disclosure as meaning that the state of the assistance system stored with the data set, in particular a position or attitude of the robot arm, is restored.

[0031] This data set preferably comprises at least one current, navigated posture or position of the robot, preferably its set of joint angles, and / or the at least one current, intracorporeal image and its location, and / or a current, navigated position of the instrument(s), preferably in each case in the patient's coordinate system. If the data set only contains the posture or position of the robot and the instrument and the current, intracorporeal image and its location, then it is a monomodal data set, since with regard to the patient it only contains the at least one image in the first data modality. When this data set is called up, the saved posture or position of the robot and / or the instrument(s) is thus set again, and / or the at least one current, intracorporeal image is output / displayed again.In this way, so-called “waypoints” of the assistance system can be reproduced and accessed at any time during the procedure.

[0032] According to a further development, the dataset is supplemented with a selection of data acquired and imported with the second data modality before and / or during the procedure. In this case, it is a multimodal dataset because it contains data relating to the patient in both the first and second data modalities. The selection can include all imported data of the second data modality whose positions lie within the field of view, or it can contain only a subset of them.

[0033] A specific data set or “waypoint” can be called up by either moving to the navigated position or attitude of the robot saved with this data set, whereby reaching this position or attitude triggers the output of the remaining data of the data set, or by selecting and / or activating the data set, which triggers moving to the position or attitude of the robot and the output of the remaining data of the data set.

[0034] In order to be able to define the aforementioned selection or subset and / or to avoid overloading the aforementioned view with too much displayed data, a user interface of the assistance system is adapted to selectively select or deselect elements of the data set. When selected, the elements are output and / or assigned to the data set to be saved; when deselected, they are not output and / or not assigned to the data set to be saved. Preferably, the control unit is adapted to set a focal point of the visualization system on the patient and to control the robot arm such that the position of the focal point is maintained in any orientation of the visualization system.

[0035] In order to improve the clarity of the view and to facilitate the interpretation of the data, the control unit is adapted according to a further embodiment to output or display at least the current intracorporeal image and / or the data acquired in the second data modality in a hierarchically ordered manner depending on their data modality and / or a size of a field of view and / or a viewing angle of a field of view and / or depending on a time of acquisition.

[0036] According to a preferred development, the control unit, in particular the data information system, is adapted to store and retrieve the data records in their chronological order, which makes it possible to create a history of the intervention and to output targeted intermediate states of the intervention, including the data relevant to the operation.

[0037] According to a preferred development, the control unit, in particular the data information system, is adapted to determine or record a delta (difference) between stored data sets, in particular to perform a measurement. The measurement can be performed in particular between recordings and / or imported data of the same data modality and the same location or position, but with different acquisition times, in order to determine progress during the procedure. In particular, intervals between the data sets or differences in acquisition time can be automatically calculated from the set of stored data sets. This information can be useful for the surgeon to perform measurements on the patient, even if the patient's anatomy has changed during the operation.

[0038] According to a preferred development, the control unit is adapted to mark a region and / or a coordinate within the current intracorporeal image and / or in the data acquired with the second data modality that differs from the visualization system, to assign a function and / or a property and / or a parameter—in the sense of functional mapping—to the marked region and / or the marked coordinate, and to store the marking and assignment in the data information system. For example, results from histological examinations on a patient's CT scan can be marked either as tumor tissue or as healthy tissue and displayed in different colors. The user can thus perform a timely interpretation of functional data and display it directly and intuitively on the patient's anatomy.

[0039] According to a preferred development, the control unit is adapted to reproduce operating settings on a digital twin of the assistance system depending on the at least one stored data set. Since the position or configuration of the robot arm and the visualization system is stored in the respective data set, this information can be used to reproduce a specific recording device both in the real world and on the digital and thus virtual twin.

[0040] The objects of the present disclosure are achieved with regard to a method for data visualization for a surgical assistance system according to the present disclosure in that the method comprises the steps of: navigating a visualization system that is connected, in particular mounted, to a robot arm, in particular on a terminal side of the robot arm, with a spatial reference to the patient, by means of a control unit; creating and providing, in particular digitally, at least one up-to-date, intracorporeal image of the patient, by means of the visualization system; determining a location, in particular a position and an orientation, of the visualization system, and thus a location of the at least one up-to-date, intracorporeal image, with a spatial reference to the patient, by means of a navigation system;Assigning a position in a coordinate system of the patient to the current, intracorporeal image, and storing the position and image, by means of the control unit; importing data, which is acquired using a second data modality different from the visualization system, into a data information system by means of a data interface; assigning a position in the coordinate system of the patient to the imported data; storing the position and the imported data, by means of the control unit in the data information system; generating a view of the current, intracorporeal image together with those of the imported data which, due to their position, are spatially arranged within the current, intracorporeal image, via the control unit; and outputting the view via one, preferably only one, display device, in particular an operating room monitor.

[0041] With regard to a computer-readable storage medium or a computer program, the objects are achieved in that the storage medium or computer program comprises instructions which, when executed by the computer, cause the computer to carry out the steps of the method according to the present disclosure.

[0042] Any disclosure related to the surgical assistance system according to the present disclosure applies to the method according to the present disclosure, and vice versa.

[0043] Short description of the characters

[0044] The invention is explained in more detail below using preferred embodiments with the aid of figures. They show:

[0045] Fig. 1 is a view of a surgical assistance system according to a first preferred embodiment;

[0046] Fig. 2 shows a multimodal data set of an assistance system according to a preferred embodiment,

[0047] Fig. 3 shows a method for data visualization of a multimodal data set according to a preferred embodiment, Fig. 4 shows a multimodal data set according to a further preferred embodiment,

[0048] Fig. 5 shows a multimodal data set according to another preferred embodiment,

[0049] Fig. 6 shows a multimodal data set according to another preferred embodiment,

[0050] Fig. 7 shows a multimodal data set according to another preferred embodiment, and

[0051] Fig. 8 shows a multimodal data set according to another preferred embodiment.

[0052] The figures are schematic in nature and are intended to aid understanding of the invention. Identical elements are provided with the same reference numerals. Features of different embodiments may be interchangeable.

[0053] Detailed description of preferred embodiments

[0054] Figure 1 shows a surgical assistance system 1 according to a preferred embodiment in a schematic side view. The assistance system 1 has a surgical robot 2 with a robot base 4, which is locally fixed in the embodiment shown. Alternatively, it can be designed to be mobile, for example, to enable the robot 2 to be deployed as needed at different locations in an operating room in a hospital. The robot base 4 always forms a local reference point to which a multi-segmented robot arm 8 with several robot arm segments 10, 12 is attached, which are connected to one another via joints 14. In this way, the robot arm segments 10, 12 can be actively moved relative to one another, and the robot arm 8 can be controlled as a whole. A visualization system 18 of the assistance system 1 is attached to an end side 16 of the robot arm 8.In the embodiment shown, this comprises a surgical microscope 20 and an overview camera 22 as end effectors of the robot. In addition to the embodiment shown, at least one medical instrument can be provided rigidly or movably as an end effector on the end segment 12 of the robot arm 8. In the embodiment shown, the assistance system 1 has a manually guided instrument 32 tracked by a navigation system of the assistance system 1.

[0055] The position (position and orientation) of the visualization system 18, and thus the respective position of the surgical microscope 20 and the overview camera 22, can be controlled and adjusted by the robot arm 8. A specially adapted control unit 24, which is provided on the robot base 4 and will be explained again below, serves as the central control unit.

[0056] To move the robot arm 8 according to a user request, the assistance system 1 has an input device (not shown). This can be designed, for example, as a joystick or a 3D space mouse, which is adapted to manually receive both translational control command inputs for three mutually perpendicular axes and rotational control command inputs about these three axes via haptic operation and to forward them to the control unit 24 as computer-readable, digital control signals or control commands, so that the control unit 24 can actively control the robot arm 8 according to the control commands.

[0057] Furthermore, the assistance system 1 has a navigation system comprising several components. This includes a navigation camera 26, which in the embodiment shown is embodied as an infrared stereo camera, a "tracker" 28, which is tracked by the navigation camera 26 and is firmly connected to the patient, and a "tracker" 30, which is firmly connected to the visualization system 18—and thus firmly connected to the surgical microscope 20, the overview camera 22, and the end segment 12 of the robot arm 8. The cameras 20, 22 are optically calibrated, and their relative position to the tracker 30 is known and fixed, so that, knowing the position of the tracker 30, the position of the cameras 20, 22 can also be determined via the navigation system. Furthermore, the navigation system includes a "tracker" 34, which is tracked by the navigation camera 26 and firmly connected to the manually guided instrument 32.The navigation camera 26 is signal-connected to the control unit 24 and transmits the information to the tracked trackers 28, 30, and 34, which then determine all necessary positions. The navigation system also includes angle sensors (not shown) on the joints 14, via which the joint angles 36 / joint configuration of the multi-jointed robot arm 8 are recorded in real time and transmitted to the control unit 24 for navigation.

[0058] The patient, who in Figure 1 is represented solely by the tracker 28 attached to him for reasons of clarity, is registered with reference to his CT scan or MRI scan 38, with a transformation 40 of the registration being stored in the control unit 24. The navigation system is thus configured with reference to a coordinate system of the patient, so that the instrument 32 and the visualization system 18 and their cameras 20, 22 can be navigated with reference to the patient coordinate system.

[0059] During the procedure, the surgical microscope 20 and the overview camera 22 acquire up-to-date, intracorporeal images 42, 46 with a respective field of view 44, 48 and digitally provide them to the control unit 24. The data modality of the two images 42, 46 is identical and, according to the disclosure, is referred to as the "first data modality." The patient scan 38, however, is not up-to-date, but rather acquired preoperatively. Furthermore, it has a data modality that differs from that of the visualization system 18, which is due to the fact that it does not involve optically acquired data, as is the case with images 42 and 46. Any data modality that differs from the first is referred to below as a "second data modality," according to the disclosure. One could also speak of an "additional / supplementary / further data modality."

[0060] A multimodal data set of the assistance system 1 according to the invention, which can alternatively be referred to as a "multimodal waypoint" MMW, thus comprises, according to Figure 1, the position 50 of the robot arm 8 determined from the tracking of the tracker 30 or the joint angle 36, and the positions of the surgical microscope 20 and the overview camera 22 determined therefrom. Furthermore, it comprises the position of the instrument 32 determined from the tracking of the tracker 34, as well as the position of the patient determined from the tracking of the tracker 28. As up-to-date data of the "first data modality," it comprises the up-to-date acquired intracorporeal images 42 and 46, their position relative to the patient (tracker 28), and their fields of view 44, 48. As preoperatively acquired data with a "second data modality" that differs from the visualization system 18, the data set of the assistance system 1 comprises the patient scan 38.

[0061] Figure 2 shows a compact representation of a multimodal data set 60 of an assistance system according to another preferred embodiment. A visualization system of the assistance system, on which the data set according to Figure 2 is based, is supplemented—compared to the assistance system according to Figure 1—by a navigated endoscope camera 52. The data set according to Figure 2 therefore includes a further up-to-date, intracorporeal image of the "first data modality," a time-correct image of the endoscope. Furthermore, the data set is supplemented by histological image data 54 and electrophysiological measurement data 56. Both represent externally generated, not up-to-date data with a "second data modality" that differs from the visualization system 18, which cannot be captured by the navigation system during their generation and are therefore initially not navigable.

[0062] According to the disclosure, these “second data modality” data, here the histological image data 54 and the electrophysiological measurement data 56, are also assigned a respective position in a coordinate system 58 of the patient, which is symbolized by the double arrows to the coordinate system 58.

[0063] A multimodal data set—or waypoint (MMW)—thus consists, as disclosed, of a series of heterogeneous surgical data organized temporally and spatially according to a fixed reference system: the patient. In order to visualize, in addition to the images generated in real time and visualized live by the visualization system 18 (“first data modality”), data that is not generated in real time and is not a generic part of the images and thus not initially navigable (“second data modality”), a specific location or coordinate in the patient's anatomy, i.e., in the patient's coordinate system 58, is assigned to these data using a data visualization method as disclosed.

[0064] Figure 3 shows such a method for data visualization according to a preferred embodiment. Based on the data set 60 shown in Figure 2, a complete process is shown, beginning with step 100 "Navigating the visualization system 18" and ending with step 800 "Outputting a view of the data set 60." The starting point is the registered patient.

[0065] In a first step 100, the visualization system 18 is navigated to the area of ​​the procedure with spatial reference to the patient. This can be achieved, for example, through the interaction of the aforementioned input means with the control unit. In a step 200, the intracorporeal overview, surgical microscope, and endoscope images are continuously and up-to-date via the visualization system 18 and digitally provided to the control unit 24. In a step 300, the position of the visualization system 18—and thus all positions of all up-to-date intracorporeal images—is determined by the navigation system with spatial reference to the patient. In step 400, the control unit assigns the up-to-date intracorporeal images their respective position in the patient's coordinate system 58 and stores them in a data information system of the control unit.In the following, it is assumed that data from a second data modality is present, for example, the digitally received result / image of a histological examination of a tissue sample previously taken from the patient. This is followed by a step 500 in which this data is imported into the data information system via a data interface. As already mentioned above, in step 600, it is assigned a position in the patient's coordinate system 58 and stored in the data information system. This can be done, for example, by the surgeon assigning it a position in the current image—and thus on the patient—via a user interface, for example, via a touchscreen or via a user interface on the input device or manually guided instrument 32.Alternatively, the data can initially be pre-determined in the center or at another predetermined position within the current image and subsequently moved by the surgeon. However, if the data is acquired directly in the intervention area, for example, electrophysiological measurement data from the intervention area of ​​the brain, the surgeon can, for example, use his or her navigated instrument to specify the position at which the data should be acquired. The position of the data in the coordinate system 58 is then directly determined via the acquisition location and stored in the data information system. In the next step 700, a view of the current intracorporeal image is generated together with the imported data, which are spatially arranged within the current intracorporeal image based on their previously assigned position.In step 800, the view is then output on a single display device, in particular a single surgical monitor.

[0066] A multimodal data set or waypoint generated and saved in this way can, of course, be retrieved and explored by the surgeon at any time from the set of stored multimodal data sets. This retrieval then results in the joint visualization of all data stored in the multimodal data set on the patient's body, regardless of the data modality. For example, if the data are saved images or are suitable for another reason, the visualized data can be visualized hierarchically from coarser (c, coarse) to finer (f, fine), or vice versa. This is illustrated in Figure 4 for a data set 62, for which the image 46 of the overview camera 22 with the large field of view 48 is visualized on the left, the image 42 of the surgical microscope 20 with the significantly smaller field of view 44 is visualized in the middle, and the histological image 54 with the smallest field of view is visualized on the right.

[0067] Figure 5 illustrates a further possibility offered by the multimodal data sets according to the disclosure. Here, the aforementioned up-to-date images 42, 46 of the surgical microscope 20 and the overview camera 20, as well as a up-to-date extracorporeal image 62 of the surgeon, the stored histological image 54, and the position of the data set or waypoint 64 are visualized in the context of the patient scan 38. The surgeon can thus observe the position of the data set or waypoint 64 directly on the patient scan 38 and, if desired, call up a specific other data set or waypoint corresponding to its anatomical position by selecting a specific anatomical location on the patient scan 38. Alternatively, the surgeon can call up another data set or waypoint by setting or reproducing the recording conditions of a stored data set or waypoint.In this case, the robot 2 and the visualization system 18 are configured according to the joint angle 36 stored in the retrieved data set or waypoint, or the stored position of the tracker 30. This allows for the exact reproduction of visualization conditions. This function can be particularly useful for comparing a specific anatomical position during the course of the operation.

[0068] As already mentioned, the surgeon can call up any data set or waypoint and, as needed, select which data from the data set is to be displayed. Selected data is displayed, while deselected data is not. Depending on the indication, the surgeon can thus freely choose between all available images and other imported data or functional data stored in the data set or waypoint. Figure 6 shows, as an example, a multimodal data set 66 for which only the current image 42 of the surgical microscope 20, the instrument 32 positioned therein, and the preoperative patient scan 38, in which the position of the instrument 32 is visualized, have been selected for display / output.

[0069] According to Figure 7, a large number of multimodal data sets or waypoints 68, 70, 72, 74, 76 can be generated and stored during an operation. The data sets or waypoints 68, 70, 72, 74, 76 can be used to obtain a comprehensive overview of the procedure and to document it clearly. They are stored in the data information system according to their temporal acquisition, i.e., as shown in Figure 7, as a function of time t, and can be displayed according to a spatial hierarchy native to the data. This information is particularly useful for creating comprehensive and meaningful surgical documentation. In contrast to image data from conventional assistance systems, the disclosed data sets or waypoints 68, 70, 72, 74, 76 are organized both temporally and spatially.In addition, the data sets or waypoints 68, 70, 72, 74, 76 can of course also be organized hierarchically internally, as already explained in Figure 4.

[0070] Figure 8 shows the use of a multimodal dataset or waypoint 78 for functional mapping. The surgeon can use the dataset or waypoint 78 for live interpretation of functional data and display this intuitively on the patient's anatomy. In Figure 8, for example, the results of a histological examination are mapped directly onto the patient's anatomy. In reality, the mapping can be done, for example, using color marking. An area marked with a T, for example, indicates tumorous tissue, while areas marked with a G indicate healthy tissue. The multimodal dataset or waypoint 78 functionally mapped in this way contributes to optimal tumor resection.

[0071] List of reference symbols

[0072] 1 surgical assistance system

[0073] 2 robots

[0074] 4 Robot base

[0075] 8 Robot arm

[0076] 10, 12 Robot arm segment

[0077] 14 joint

[0078] 16 End section robot arm

[0079] 18 Visualization system

[0080] 20 Operating microscope

[0081] 22 Overview camera

[0082] 24 Control unit

[0083] 26 navigation camera

[0084] 28 Tracker Patient

[0085] 30 Tracker visualization system

[0086] 32 surgical instrument

[0087] 34 Tracker surgical instrument

[0088] 36 patient scans

[0089] 40 Transformation of the registration

[0090] 42 Surgical microscope image

[0091] 44 field of view surgical microscope

[0092] 46 Overview camera recording

[0093] 48 field of view overview camera

[0094] 50 layer robot arm

[0095] 52 Endoscope

[0096] 54 histological data

[0097] 56 electrophysiological data

[0098] 58 Patient coordinate system

[0099] 60 multimodal data sets

[0100] 62 Recording Surgeon Camera

[0101] 64 - 78 multimodal dataset step navigation visualization system

[0102] Step Create and provide data of the first data modality

[0103] Step Determine location of visualization system

[0104] Step Assign Location

[0105] Step Import second data modality data

[0106] Step Assign Position

[0107] Step Create View

[0108] Step Output View

Claims

Claims 1 . Surgical assistance system (1) with a navigated, surgical robot (2) for use in a surgical procedure on a patient, comprising: a robot base (4) as a local connection point of the robot (2) and a movable robot arm (8) connected to the robot base (4) and having at least one robot arm segment (10, 12), a visualization system (18) connected to the robot arm (8) and adapted to create and provide at least one up-to-date, intracorporeal image (42, 46), a navigation system (26, 30) adapted to determine, with spatial reference to the patient, at least one position of the visualization system (18) and thus a position of the at least one up-to-date, intracorporeal image (42, 46), and a control unit (24) adapted to assign a position in a coordinate system (58) of the patient to the up-to-date, intracorporeal image (42, 46) and to store, characterized bythat the control unit (24) is adapted to import data (38, 54, 56) which are recorded in a second data modality which is different from the visualization system into a data information system of the control unit (24) via a data interface, to assign at least one position, in particular a location, to the imported data (38, 54, 56) in the patient's coordinate system (58) and to store it together with the position in the data information system, to generate a view of the current, intracorporeal image (42, 46) together with those of the imported data (38, 54, 56) which, due to their position, are spatially arranged within the current, intracorporeal image (42, 46), and to output the view via a display device, in particular an operating room monitor.

2. Surgical assistance system (1) according to claim 1, characterized in that the data (38, 54, 56) acquired with the second data modality are image data (38, 54) and / or measurement data (56) of the patient and / or operating data of the surgical robot (2).

3. Surgical assistance system (1) according to one of the preceding claims, characterized in that the control unit (24) is adapted to permanently store a currently recorded data record (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) in the data information system, and / or to call up a permanently stored, previously currently recorded data record (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) from the data information system.

4. Surgical assistance system (1) according to claim 3, characterized in that the time-acquired data set (60, 64, 66, 68, 70, 72, 74, 76, 78) comprises at least one time-current, navigated position of the robot (2) and / or the at least one time-current, intracorporeal image (42, 46) and its position and / or preferably a time-current, navigated position of the instrument (32), in each case in the coordinate system (58) of the patient.

5. Surgical assistance system (1) according to claim 3 or 4, characterized by a user interface which is adapted to selectively select elements of the data set (60, 64, 66, 68, 70, 72, 74, 76, 78) for output or to deselect them from output.

6. Surgical assistance system (1) according to one of the preceding claims, characterized in that the control unit (24) is adapted to output at least the intracorporeal image (42, 46) of the visualization system (18) and / or the data (38, 54, 56) acquired in the second data modality in an ordered manner depending on their data modality and / or a size of a field of view (44, 48) and / or a viewing angle of a field of view (44, 48) and / or depending on a recording time (t).

7. A method for data visualization during a surgical, in particular neurosurgical, intervention on a patient, in particular for a surgical assistance system (1) according to one of the preceding claims, comprising the steps: (100) Navigating a visualization system (18) connected to a robot arm (8) with spatial reference to the patient by means of a control unit (24), (200) Creating and providing, in particular digitally, at least one up-to-date intracorporeal image (42, 46) by means of the visualization system (18), (300) Determining a position of the visualization system (18), and thus a position of the at least one current intracorporeal image (42, 46), with spatial reference to the patient, by means of a navigation system (26, 30), and (400) Assigning a position in a coordinate system (58) of the patient to the current intracorporeal image (42, 46) and storing the position and image (42, 46) in a data information system of the control unit (24) by means of the control unit (24), characterized by the steps: (500) Importing data (54, 56) that are transferred to the visualization system (18) different, second data modality, by means of a data interface of the control unit (24) into the data information system, (600) Assigning a position in the patient's coordinate system (58) to the imported data (54, 56) and storing the position and the imported data (54, 56) by means of the control unit (24), (700) generating a view of the current intracorporeal image (42, 46) together with those of the imported data (54, 56) which, due to their position, are spatially arranged within the current intracorporeal image (42, 46), via the control unit (24), and (800) Outputting the view via one, preferably exactly one, display device, in particular an operating room monitor.

8. A method for data visualization according to claim 7, characterized by the step: Permanent storage of a currently recorded data set (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) in the data information system, by means of a user interface and the control unit (24), and / or Retrieving a permanently stored, previously recorded data set (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) from the data information system by means of a user interface and the control unit (24), wherein the data set (60, 64, 66, 68, 70, 72, 74, 76, 78) comprises at least one current, navigated position of the robot (2), preferably its joint angle (36), and / or the at least one current, intracorporeal image (42, 46) and its position, and / or the current, navigated position of the instrument (32), in each case in the coordinate system (58) of the patient.

9. A method for data visualization according to claim 8, characterized in that the method further comprises the step: Retrieving the permanently stored, previously time-acquired data set (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) from the data information system, which is triggered by the assistance system (1), in particular the robot arm (8), moving to a position of this stored data set (60, 64, 66, 68, 70, 72, 74, 76, 78) via the control unit (24).

10. A method for data visualization according to claim 8 or 9, characterized by the step: Assigning a selection of data acquired before the procedure and / or during the procedure with the second data modality to the stored data set (60, 64, 66, 68, 70, 72, 74, 76, 78) by means of a user interface and the control unit (24).

11. Method for data visualization according to one of claims 8 to 10, characterized by the step: Displaying a position of the stored data set (64) in a scan (38) or image of the patient.

12. A method for data visualization according to one of claims 8 to 11, characterized by the step: Creating a histone of data sets (68, 70, 72, 74, 76, 78).

13. Method according to one of claims 8 to 12, characterized by the further step: Measure differences between images and / or scans of the same data modality and location, but with different acquisition times.

14. Method according to one of claims 7 to 13, characterized in that the method further comprises the steps: Marking an area and / or a coordinate within the current intracorporeal image and / or in the data acquired with the second data modality different from the visualization system, Assigning a function and / or a property and / or a parameter (G, T), and Saving the marking and assignment in the data information system.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method steps of the method for data visualization according to any one of claims 7 to 14.

Images