Surgical assistance system and method for data visualization of surgical surgical interfaces

Abstract

The present disclosure relates to a surgical assistance system (1) having a navigated surgical robot (2) for use in performing a surgical intervention on a patient, the assistance system having a robot base (4) as a local attachment point of the robot (2) and a movable robot arm (8) attached to the robot base (4), the robot arm has at least one robot arm segment (10, 12); a visualization system (18) coupled to the robotic arm (8) and adapted to create and provide at least one real-time, in-vivo photograph (42, 46); a navigation system (26, 30) adapted to determine, in spatial association with the patient, at least one orientation of the visualization system (18) and thus the orientation of at least one real-time, in-vivo picture (42, 46); and a control unit (24) adapted to assign an orientation in a coordinate system (58) of the patient for the real-time photographs (42, 46) of the body and to store the orientation. In addition, the present disclosure relates to a method for data visualization according to the paratactic claims and a computer-readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to a surgical assistance system, particularly a neurosurgical assistance system, comprising a navigated surgical robot for use during surgical interventions on a patient. The robot comprises a robot base serving as the robot's local connection point and, to a certain extent, as a local, fixed-position coordinate system. A movable robot arm having at least one robot arm segment is connected to the robot base. The assistance system also comprises a visualization system comprising one or more cameras, which is connected to, or particularly supported on, the robot arm, particularly on the side of the robot arm at its distal end. The visualization system is adapted to create at least one real-time, preferably in-vivo, image of the patient and preferably provide the image digitally. The navigation system of the assistance system allows for the determination of at least the position, i.e., the location and orientation, of the visualization system in spatial relation to the patient, and thereby the position of the at least one real-time, in-vivo image. Furthermore, surgical instruments are typically provided, either manually operated or operated by the navigated surgical robot. If this is the case, the position of the surgical instruments relative to the patient can also preferably be tracked by the navigation system. Furthermore, a control unit is provided. The control unit is adapted at least for assigning a position in the patient's coordinate system to at least one real-time, in-vivo image and for storing the position. The control unit is also preferably adapted for controlling the aforementioned subsystems of the assistance system—that is, at least for controlling the robot arm, the visualization system, and the navigation system—and for controlling their interaction. Background Art

[0002] Automation and the consequent integration of digitally controllable technical devices are gaining increasing importance in the fields of medicine and medical technology. Robots are increasingly being used in surgical interventions, particularly to support precise, minimally invasive procedures. These robots are not only used as standalone robots that simply perform operations, but are also being increasingly used as collaborative robots (cobots), meaning they are used directly in the surgical area as assistive or support robots that interact with medical professionals, particularly surgeons.

[0003] For such robotically supported surgical interventions, particularly neurosurgery of the brain, various visualization and measurement systems are often used to provide visual, physiological, and functional assessments of the area of ​​intervention and to assist in decision-making. Conventional data collected before and during the surgical intervention include, for example, pre-operative MRI and CT images of the patient, positional data of surgical instruments provided by a navigation system, images from multimodal digital microscopes, data from robots used to precisely position surgical instruments and visualization systems, electrophysiological monitoring data, tissue samples, and intraoperative ultrasound images.

[0004] Existing surgical assistance systems, particularly their navigation systems, can only integrate a limited amount of the aforementioned data. Preoperative patient scans are used as a common reference. State-of-the-art navigation systems can be used to position and navigate digital microscopes, combining the position of surgical instruments with the real-time microscope image (live image) captured during surgery and displaying them together.

[0005] On the other hand, current robotic visualization systems allow the position of the robotic arm to be linked to the image from a microscope fixedly connected to the robot's end effector. Consequently, these systems enable the precise reproduction of the conditions under which micrographs were taken. This key feature of robotic visualization systems is often referred to as "waypoints." Waypoints are defined and stored as, at a minimum, defined postures of the robotic arm and its segments relative to one another, for example, in the form of a set of joint angles. Stored waypoints can be recalled by the user, allowing the robotic arm to be precisely readjusted to the stored position and accurately reproducing the visualization conditions for the associated camera represented by the waypoint. Furthermore, the visualization system can be rotated or translated around a stored point on the patient, a so-called "lock-on target." However, existing waypoint functionality for visualization systems and surgical navigation systems is not currently integrated, resulting in inconsistencies between real-time images and instrument navigation. As a result, existing waypoint functionality cannot compensate for undesired patient movement or positional changes.

[0006] For the surgeon, the challenge lies in having to track the data from the visualization and navigation systems on separate systems using separate displays. Furthermore, other external, surgery-related data, such as those that cannot be visualized in real time (because they are acquired before or during surgery and therefore first prepared externally or are located outside the surgical intervention area), must be tracked on additional systems. Typical examples of this are histological data (either fixed or not) or electrophysiological data (the quality of local (electrochemical) signal transmission). These are not integrated into either the navigation system or the robotic positioning system, making their availability and interpretation significantly more difficult. Furthermore, since multiple different systems and screens must be used, as described above, this data cannot be presented to the surgeon in a simple manner. This is a significant drawback of existing surgical assistance systems. Furthermore, it is known to organize and store surgery-related data along a timeline. However, the surgeon can only access this data in a post-operative environment, based on the time at which the data was acquired. Summary of the Invention

[0007] Accordingly, the object of the present disclosure is to avoid or at least reduce the disadvantages of the prior art and, in particular, to provide a surgical assistance system, a (computer-implemented) method for data visualization using a surgical assistance system, and a computer-readable storage medium that allow a user to interpret surgically relevant data during surgery in a particularly simple, intuitive, and reliable manner. One object may be, in particular, to process, combine, and display different forms of medical information so that this information can be clearly displayed on a single display, depending on the requirements, wherein, in particular, for a specific tissue region of a patient, also the corresponding relevant combined information about that tissue region is displayed.

[0008] According to the invention, this object is achieved with respect to a surgical assistance system by the features of claim 1, with respect to a (computer-implemented) method for data visualization by the features of claim 7, and with respect to a computer-readable storage medium by the features of claim 15.

[0009] The basic idea of ​​the present invention is to realize a surgical auxiliary system, which is adapted to enable a spatial arrangement or distribution of operation-related data of each data form and to display these data together in space. The data form describes a corresponding method for data collection in the sense of the present disclosure. The data form can be, for example, imaging or functional, at the patient or at the auxiliary system and / or real-time or pre-operative or post-operative. The spatial arrangement of these data can preferably be spatially associated with the patient, in particular directly spatially associated with the patient's anatomical structure, preferably associated with the patient's coordinate system. This common arrangement and display according to the present disclosure allows the user to interpret all operation-related data more simply, more intuitively and more reliably during the operation, regardless of the data form of these data.

[0010] In other words, a surgical assistance system is provided that is adapted to merge or aggregate surgically relevant data in different data formats acquired during, before, and / or after surgery. The surgical navigation system of the assistance system is configured to spatially locate and spatially visualize data that differs based on its different data formats. The data are interconnected via their relative position relative to the patient, in particular, via the patient's coordinate system.

[0011] Specifically, a surgical assistance system, particularly a neurosurgical assistance system, includes a navigated surgical robot for use during surgical interventions on a patient. The robot includes a robot base as its local attachment point and a movable robot arm connected to the robot base, the robot arm having at least one robot arm segment. A visualization system (or visualization unit) is connected to, or in particular supported by, the robot arm, particularly on a distal side portion of the robot arm. The visualization system is adapted to create at least one real-time in-vivo image of the patient and subsequently provide the image in a digital or computer-readable format. Furthermore, the assistance system includes a navigation system adapted to determine at least one position, i.e., location and orientation, of the visualization system in spatial relation to the patient, and thereby determine the position of the at least one real-time in-vivo image (e.g., by transforming the visualization system into the image, e.g., using depth information). For this purpose, the navigation system may include, in particular, a 3D camera and rigid bodies or rigid bodies (with markers) tracked by the 3D camera, at least one of the rigid bodies being fixedly coupled to the visualization system. The position of the visualization system can be determined by triangulating, in particular, the 3D image of the rigid body (preferably performed in the control unit of the assistance system). Alternatively or additionally, the navigation system can include a detection unit for detecting the joint angles of the robot arm (i.e., the configuration of the robot arm segments). Given otherwise known kinematics of the robot arm, the control unit can also determine the position of the visualization system. Furthermore, the assistance system preferably includes at least one surgical instrument, either manually operated or operated by a navigated surgical robot, whose position relative to the patient, in particular the position of the instrument's end effector, is tracked by the navigation system. The control unit of the assistance system is specifically adapted to assign at least one position, in particular an orientation, in the patient's coordinate system to at least one real-time in-vivo image and to store the at least one position, preferably in a data information system of the control unit. The at least one real-time in-vivo image has a first data format resulting from an acquisition method. According to the present invention, the control unit is adapted to input additional surgery-related data, preferably from the patient and / or the assistance system, acquired in a second data format different from that of the visualization system, into the data information system via a data interface. The data in the second data format can be collected preoperatively and / or in real time (and / or even postoperatively, for example, to verify the surgery). Thus, the data information system contains at least one real-time, in-vivo photograph of the patient and other surgically relevant data in any data format. Furthermore, the control unit is adapted to assign a position to the input data in the patient's coordinate system and store the position in the data information system. Thus, the data in the second data format are spatially associated with the (virtual) patient through an image representation.All surgically relevant data is thus assigned a spatial position in the patient's (central) coordinate system. The data information system thus forms the basis for a common spatial display not only of the images but also of other data, regardless of their format. Thus, data that is non-spatially located on the patient, such as robot operating data, or non-real-time patient data, such as pre-operative scans or functional measurement data, are also assigned at least one spatial position on the patient. Finally, the control unit is adapted to generate at least one real-time, in-vivo image view along with the input data and to output this view via a display device of the auxiliary system, in particular a surgical monitor. Preferably, the control unit is adapted to generate the at least one real-time, in-vivo image view along with only that portion of the input data that, based on the data's position, is spatially located within the at least one real-time, in-vivo image.

[0012] In particular, a central reference is provided by a central coordinate system into which all medical data of different forms can be integrated, for example in order to centrally process different information and, for example, obtain coordinated views (e.g. overlays, augmented reality and the like), link or associate and output important medical information of the patient centrally for the surgeon.

[0013] In this way, a solution has been found for transplanting surgical data that, due to its data format, is not a photograph or image data of the patient (in vivo), into the same field of view as the surgeon's, such as live images, photographs, or other image data. This data is displayed in / on the best possible reference frame for this purpose: the patient's body. Thus, surgical data that differ in their data format can be output or displayed on a single, shared display device. The result for the surgeon is that their line of sight is no longer directed to multiple displays, but only to a single display to provide an overview of all surgical data. This combination—the output of data on a single display device and the spatial display of all surgical data on the patient's body—enables simpler, more intuitive, and more reliable interpretation of surgical data during surgery. This results in safer surgery and reduced surgical intervention time.

[0014] Therefore, a surgical assisting system is provided, which avoids or at least reduces the above-mentioned disadvantages of the prior art.

[0015] In the present disclosure, the term "end effector" refers to a medical device, such as an instrument or device, that can be used to perform a surgical intervention, such as when performing a surgical intervention on a patient. In particular, the following may be considered end effectors: an instrument, a medical instrument (such as an endoscope or a suction tube), an optical instrument with a visualization axis, a pointer / pointer with a distal tip for surgical navigation, and the like.

[0016] The term "robot arm segment" here particularly refers to the robot part of the robot arm that is supported between bearings or joints or, in the case of a robot arm segment located at the end, particularly to the robot part that is connected in series between the end effector and the preceding robot arm segment (corresponding to the robot base in the case of only one robot arm segment).

[0017] The term "position" refers to a geometric position in three-dimensional space, which is in particular specified by means of coordinates of a Cartesian coordinate system. A position can in particular be specified by the three coordinates X, Y and Z.

[0018] The term "orientation" in turn designates an orientation in space (approximately a position). Alternatively, orientation can be expressed using a direction or rotation in three-dimensional space. Orientation can be expressed, in particular, using three angles.

[0019] The term "position" includes both location and orientation. The position can be specified in particular by means of six coordinates: three position coordinates X, Y and Z and three angular coordinates for orientation.

[0020] Advantageous embodiments are claimed in the dependent claims and are explained in particular below.

[0021] According to a preferred embodiment, the visualization system can have a panoramic camera with a field of view, which is arranged and constructed to capture the surgical intervention area and its surrounding environment, and / or can have a microscope camera with a field of view (in particular with a magnifier, the magnifier has a zoom function), which is arranged and constructed to capture the surface of the surgical intervention or tissue, and / or can have an endoscopic camera with a field of view, which is arranged and constructed to capture inside the body, in particular below the surface of the surgical intervention area.

[0022] According to a preferred embodiment, the assistance system can be adapted to register the patient via the navigation system, so that the navigation system is configured to navigate the instrument and / or the visualization system.

[0023] According to a preferred embodiment, any of the above-mentioned components or cameras of the visualization system can be calibrated relative to the patient's coordinate system.

[0024] The corresponding intrinsic parameters (of the visualization unit), in particular focal length, zoom, distortion coefficients etc., can preferably be stored together with the corresponding photograph in the data information system.

[0025] The data acquired using the second data modality are preferably preoperative image data or scans of the patient, in particular CT images and / or MRI images and / or histological images and / or 3D images. Alternatively or additionally, patient measurement data, in particular electrophysiological data, and / or surgical robot operating data may be acquired, such as a set of joint angles of a robot arm or information about the currently used end effector, its rated data or operating parameters, such as torque, speed, etc.

[0026] For inputting electrophysiological measurement data, the auxiliary system can, according to a preferred embodiment, have a serial interface. For inputting histological measurement data, a DICOM interface can preferably be provided.

[0027] To avoid overloading the view with data, in one embodiment, the control unit is adapted to continuously determine the coordinates of the field of view of at least one live, in-vivo image in the patient's coordinate system and adjust these coordinates relative to the positions assigned to the input data (second data form). The control unit is then adapted to output only those data from the input data whose positions lie within the coordinates of the field of view. In particular, the surgeon can fade in and out specific forms, for example, histological data, by inputting them.

[0028] In a preferred embodiment, the control unit can be adapted to permanently store the data sets acquired in real time by the assistance system in a data information system and to be able to retrieve the data sets again later. The term "retrieval" is understood in the disclosed sense to mean restoring the state of the assistance system stored in the data set, in particular the pose or position of the robot arm.

[0029] Preferably, the data set includes at least one real-time, navigated pose or position of the robot, preferably a set of joint angles of the robot, and / or at least one real-time, in-vivo photograph and its orientation, and / or the real-time, navigated position of an instrument, preferably each located in the patient's coordinate system. If the data set only contains the pose or position of the robot and instrument, and real-time, in-vivo photographs and their positions, then the data set is a single-format data set, since it only contains at least one photograph of the patient in the first data format. Therefore, when this data set is called up, the stored pose or position of the robot and / or instrument is adjusted, and / or at least one real-time, in-vivo photograph is output / displayed. This allows the so-called "waypoints" of the assistance system to be replayed and navigated to at any time during the surgical intervention.

[0030] According to one embodiment, the dataset is supplemented with a selection of data collected and input using a second data form before and / or during the surgical intervention. In this case, this is a multimodal dataset, as the dataset contains data from the first data form as well as data from the second data form regarding the patient. The selection can include all input data for which the data from the second data form are located within the field of view, or only a subset thereof.

[0031] The call of a specific data set or "waypoint" can be carried out in the following manner, namely, either driving to the robot's posture or position stored using the data set and being navigated, wherein the output of the remaining data of the data set is triggered by reaching the posture or position, or selecting and / or activating the data set, thereby triggering driving to the posture or position of the robot and outputting the remaining data of the data set.

[0032] In order to be able to limit the aforementioned selections or subsets and / or to avoid overloading the aforementioned views with too much display data, the user interface of the assistance system is adapted to selectively select or deselect elements of the data set. In the case of selection, these elements are output together and / or assigned to the data set to be stored; in the case of deselection, these elements are not output and / or not assigned to the data set to be stored.

[0033] Preferably, the control unit is adapted to focus the visualization system on the patient and to control the robotic arm such that the position of the focus is maintained in any orientation of the visualization system.

[0034] In order to improve the clarity of the view and simplify the interpretation of the data, according to another embodiment, the controller is adapted to output or display real-time, in-vivo photographs and / or data acquired in a second data form in at least a hierarchical arrangement according to the data form of the data and / or the size of the field of view and / or the perspective of the field of view and / or according to the acquisition time.

[0035] According to a preferred embodiment, the control unit, in particular the data information system, is adapted to store and retrieve data sets in chronological order, thereby making it possible to create a history of surgical interventions and to output intermediate results of surgical interventions, including operation-related data, in a targeted manner.

[0036] According to a preferred embodiment, the control unit, in particular the data information system, is adapted to perform, in particular measure, deltas (differences) between stored data sets for determined or acquired data sets. This measurement can be performed, in particular, between photographs and / or input data of the same data format and orientation or position, but acquired at different times, in order to determine the course of a surgical intervention. In particular, the distances between the data sets or differences in acquisition times can be automatically calculated from the collection of stored data sets. This information can be used by the surgeon to perform measurements on the patient, even if the patient's anatomy has changed during surgery.

[0037] According to a preferred embodiment, the control unit is adapted to mark regions and / or coordinates within a real-time, in vivo image and / or in data acquired in a second data format different from that of the visualization system, assign functions and / or features and / or parameters—in the sense of functional mapping—to the marked regions and / or marked coordinates, and store the markings and assignments in the data information system. For example, the results of a histological examination can be selectively marked as tumor tissue or healthy tissue on a patient's CT scan and displayed in different colors. The user can interpret the functional data in real time and display the functional data directly and intuitively on the patient's anatomical structure.

[0038] According to a preferred embodiment, the control unit is adapted to reproduce the operating settings on the digital twin of the assistance system based on at least one stored data set. Since the poses, positions, or configurations of the robot arm and the visualization system are stored in the corresponding data sets, this information can be used to reproduce specific image settings both in the real world and on the digital, and therefore virtual, twin.

[0039] The object of the present disclosure is achieved according to the present disclosure in a method for visualizing data of a surgical assistance system by the following steps: the method comprises the following steps: navigating a visualization system spatially relative to a patient by means of a control unit, the visualization system being connected, in particular supported, on a robot arm, in particular on a side portion of the robot arm located at the end; creating and providing, in particular digitally creating and providing, at least one real-time in-vivo photograph of the patient by means of the visualization system; determining the position, in particular the position and orientation, of the visualization system spatially relative to the patient by means of the navigation system, and thereby determining the position of at least one real-time in-vivo photograph; assigning the position in the patient's coordinate system to the real-time in-vivo photograph by means of the control unit, and storing the position and the photograph; inputting data acquired in a second data form different from that of the visualization system into a data information system by means of a data interface; assigning the position in the patient's coordinate system to the inputted data; storing the position and the inputted data in the data information system by means of the control unit; generating, by means of the control unit, a view of the real-time in-vivo photograph together with those data of the inputted data which are spatially arranged within the real-time in-vivo photograph based on the position of the data; and outputting the view by means of preferably only one display device, in particular a surgical monitor.

[0040] With regard to a computer-readable storage medium or a computer program, the object is achieved in that the storage medium or the computer program comprises instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to the present disclosure.

[0041] Any disclosure associated with the surgical assisting system according to the present disclosure is applicable to the method according to the present disclosure, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be explained in detail below based on preferred embodiments with the aid of the accompanying drawings.

[0043] Figure 1 A view showing a surgical assistance system according to a first preferred embodiment;

[0044] Figure 2 shows a multi-modal data set of an assistance system according to a preferred embodiment,

[0045] Figure 3 A method for data visualization of a multi-modal dataset according to a preferred embodiment is shown.

[0046] Figure 4 shows a multi-modal dataset according to another preferred embodiment,

[0047] Figure 5 shows a multi-modal dataset according to another preferred embodiment,

[0048] Figure 6 shows a multi-modal dataset according to another preferred embodiment,

[0049] Figure 7 shows a multi-modal dataset according to another preferred embodiment, and

[0050] Figure 8 A multimodal dataset according to another preferred embodiment is shown.

[0051] The accompanying drawings are schematic and serve to illustrate the present invention. Identical elements are provided with identical reference numerals. Features of different embodiments may be interchanged with one another. DETAILED DESCRIPTION

[0052] Figure 1 A schematic side view of a surgical assistance system 1 according to a preferred embodiment is shown. The assistance system 1 comprises a surgical robot 2 having a robot base 4, which in the illustrated embodiment is fixed locally. Alternatively, the robot base 4 can be designed to be movable, for example, so that the robot 2 can be used at different locations in an operating room in a hospital as required. In any case, the robot base 4 forms a local reference point, to which a multi-segmented robot arm 8 having a plurality of robot arm segments 10, 12 is connected, each of which is connected to one another via joints 14. In this way, the robot arm segments 10, 12 can be actively moved relative to one another, and the robot arm 8 can be controlled as a whole.

[0053] The visualization system 18 of the assistance system 1 is attached to the end portion 16 of the robot arm 8. In the illustrated embodiment, the visualization system 18 includes a surgical microscope 20 as the robot's end effector and a panoramic camera 22. In addition to the illustrated embodiment, at least one medical instrument can be fixedly or movably mounted on the end segment 12 of the robot arm 8 as an end effector. In the illustrated embodiment, the assistance system 1 includes a manually operated instrument 32 that is tracked by the navigation system of the assistance system 1.

[0054] The position (position and orientation) of visualization system 18 and thus the corresponding positions of surgical microscope 20 and panoramic camera 22 can be controlled and adjusted by robot arm 8. For this purpose, a specially adapted control unit 24, which is arranged on robot base 4 and will be explained again below, serves as a central control unit.

[0055] In order to move the robot arm 8 according to user requirements, the assistance system 1 has an input device (not shown). The input device can be configured as a joystick or a 3D space mouse, for example. The joystick or 3D space mouse is adapted to manually receive control command inputs for not only translations along three mutually perpendicular axes but also rotations about these three axes through tactile operation, and transmit these control commands to the control unit 24 as computer-readable digital control signals or control commands, so that the control unit 24 can actively control the robot arm 8 according to the control commands.

[0056] Assistance system 1 also includes a navigation system with multiple components. The navigation system comprises a navigation camera 26, which in the illustrated embodiment is implemented as an infrared stereo camera; a "tracker" 28 and a "tracker" 30, each tracked by navigation camera 26. Tracker 28 is fixedly connected to the patient, while tracker 30 is fixedly connected to visualization system 18 and, therefore, not only to surgical microscope 20 and panoramic camera 22, but also to end segment 12 of robotic arm 8. Cameras 20 and 22 are optically calibrated, and their relative positions relative to tracker 30 are known and fixed. This allows the navigation system to determine the positions of cameras 20 and 22, provided the position of tracker 30 is known. Furthermore, the navigation system comprises a "tracker" 34, tracked by navigation camera 26 and fixedly connected to a manually manipulated instrument 32. Navigation camera 26 is in signal communication with control unit 24 and transmits information about tracked trackers 28, 30, and 34 to control unit 24, which then determines all necessary positions. Furthermore, the navigation system includes angle sensors (not shown) on joints 14, which detect the joint angles 36 and joint configurations of multi-jointed robot arm 8 in real time and transmit them to control unit 24 for navigation.

[0057] exist Figure 1 In FIG, the patient, represented for reasons of clarity only by the tracker 28 fixed to the patient, is registered in connection with his CT scan or MRT scan 38, wherein the registered transformation 40 is stored in the control unit 24. The navigation system is thus arranged in connection with the patient's coordinate system, so that the instrument 32 and the visualization system 18 with its cameras 20, 22 can be navigated in connection with the patient's coordinate system.

[0058] During the surgical intervention, surgical microscope 20 and panoramic camera 22 create real-time, in-vivo images 42 and 46 with corresponding fields of view 44 and 48, and digitally provide them to control unit 24. The data format of both images 42 and 46 is identical and, in accordance with the present disclosure, is referred to as the "first data format." In contrast, patient scan 38 is not acquired in real time, but rather preoperatively. Furthermore, patient scan 38 has a different data format than visualization system 18 because it does not involve optically acquired data, as is the case with images 42 and 46. Each data format that differs from the first data format is hereinafter referred to as a "second data format" in accordance with the present disclosure. This may also be referred to as an "additional / supplementary / further data format."

[0059] Thus, the multimodal data set according to the invention of the assistance system 1, which may alternatively be referred to as "multimodal waypoints" MMW, is Figure 1 This includes the position 50 of robot arm 8, determined by tracking or joint angles 36 of tracker 30, and the positions of surgical microscope 20 and panoramic camera 22 determined thereby. Furthermore, the data set includes the position of instrument 32, determined by tracking of tracker 34, and the position of the patient, determined by tracking of tracker 28. As real-time data, the "first data form" includes in-vivo images 42 and 46 acquired in real time, their positions relative to the patient (tracker 28), and their fields of view 44 and 48. As preoperative data acquired in a "second data form" different from that of visualization system 18, the data set of assistance system 1 includes patient scan 38.

[0060] Figure 2 A compact diagram shows a multi-modal data set 60 of an assistance system according to another preferred embodiment. Figure 2 Visualization system of auxiliary system based on the dataset of Figure 1 Compared to the auxiliary system - is supplemented by a guided endoscope camera 52. Therefore, according to Figure 2 The data set has additional real-time, in-vivo images, real-time endoscopic images, in a "first data form." Furthermore, the data set is supplemented with histological image data 54 and electrophysiological measurement data 56. Both are external, non-real-time generated data in a "second data form" different from the visualization system 18, which could not be acquired by the navigation system during their generation and, therefore, could not be navigated in the first place.

[0061] According to the present disclosure, these data of the “second data form”, here histological image data 54 and electrophysiological measurement data 56 , are also assigned corresponding positions in the patient's coordinate system 58 , which is indicated by a double arrow to the coordinate system 58 .

[0062] Thus, according to the present invention, a multimodal dataset—or waypoint MMW—consists of a series of differentiated surgical data organized in time and space according to a fixed reference system: according to the patient. In order to visualize, in addition to the images ("first data modality") generated in real time and visualized live by the visualization system 18, also data ("second data modality") that were not generated in real time and are not generally part of the images and therefore not initially navigable, these data are assigned specific positions or coordinates in the patient's anatomy, that is, in the patient's coordinate system 58, by the method for data visualization according to the present disclosure.

[0063] Figure 3 This method for data visualization according to a preferred embodiment is shown. Figure 2 The dataset 60 shows the complete process from step 100 “navigate the visualization system 18 ” to step 800 “output the view of the dataset 60 ”. The initial position is the registered patient.

[0064] In a first step 100, visualization system 18 is navigated to the area of ​​surgical intervention in spatial relation to the patient. This can be achieved, for example, through the interaction of the aforementioned input device and a control unit. In step 200, visualization system 18 continuously and in real time generates panoramic, surgical microscopic, and endoscopic images of the body and digitally provides them to control unit 24. In step 300, the navigation system determines the position of visualization system 18 in spatial relation to the patient—and thus the positions of all real-time, in-vivo images. In step 400, the control unit assigns the real-time, in-vivo images their respective positions in the patient's coordinate system 58 and stores them in the control unit's data information system. The further process assumes the presence of data in a second data form, such as digitally input results / images of a histological examination of a tissue sample previously taken from the patient. Next, step 500 is performed, in which this data is input into the data information system via a data interface. As already mentioned above, in step 600, this data is then assigned positions in the patient's coordinate system 58 and stored in the data information system. This can be done, for example, by the surgeon assigning the data to a location in the live image—and therefore on the patient—via a user interface, such as a touchscreen or an input device, or a manually operated instrument 32. Alternatively, the data can initially be pre-positioned at another predetermined location in the center or live image and subsequently moved by the surgeon. Conversely, if the data is acquired directly in the surgical intervention area, such as electrophysiological measurement data in the brain, the surgeon can, for example, use their navigated instrument to specify the location at which the data should be acquired. The acquisition location then directly determines the location of the data in coordinate system 58 and stores it in the data information system. In the next step 700, a view of the live, in-vivo image is generated, along with those data from the input data that are spatially arranged within the live, in-vivo image based on the previously assigned locations. The view is then output in step 800 on a single display device, particularly a single surgical monitor.

[0065] Of course, the multimodal datasets or waypoints generated and stored in this way can be called up and explored by the surgeon at any time from the collection of stored multimodal datasets. This call-up then results in a comprehensive visualization of all data stored in the multimodal dataset (of the same data format) on the patient's body. In this case, if this involves stored photographs or for other reasons, the visualized data can be hierarchically visualized, for example, from a coarser (c, coarse) to a finer (f, fine) or vice versa. This is done in Figure 4 , in which for data set 62 a picture 46 of panoramic camera 22 with a large field of view 48 is visualized on the left, a picture 42 of surgical microscope 20 with a significantly smaller field of view 44 is visualized in the center, and a histological picture 54 with a minimal field of view is visualized on the right.

[0066] Figure 5 Another possibility provided by the multimodal datasets disclosed herein is illustrated. Here, the already mentioned live images 42, 46 of the surgical microscope 20 and panoramic camera 20, as well as the surgeon's live in vitro image 62, stored histological images 54, and the positions of datasets or waypoints 64 are visualized against the backdrop of a patient scan 38. The surgeon can thus observe the positions of datasets or waypoints 64 directly on the patient scan 38 and, if desired, call up another dataset or waypoint based on a specific anatomical location by selecting it on the patient scan 38. Alternatively, the surgeon can call up another dataset or waypoint by adjusting or reproducing the image conditions of a stored dataset or waypoint. In this case, the robot 2 and visualization system 18 are configured based on the joint angles 36 or the stored orientation of the tracker 30 stored in the called dataset or waypoint. This enables precise reproduction of visualization conditions. This functionality can be particularly useful for comparing specific anatomical locations during surgery.

[0067] As already mentioned, the surgeon can call up any data set or waypoint and select which data of the data set to display to him as needed. The selected data is displayed, while the unselected data is not. Thus, according to the instructions, the surgeon can freely choose between all available photos and other input data or functional data stored in the data set or waypoint. Figure 6 By way of example, a multimodality data set 66 is shown, for which only the live picture 42 of the surgical microscope 20 , the instrument 32 positioned therein, and the preoperative patient scan 38 , in which the position of the instrument 32 is visualized, are selected for display / output.

[0068] refer to Figure 7 During a surgical intervention, a plurality of multi-modal data sets or waypoints 68, 70, 72, 74, 76 may be generated and stored. The data sets or waypoints 68, 70, 72, 74, 76 may be used to obtain a broad overview of the surgical intervention and to clearly record it. The data sets or waypoints may be generated and stored according to their temporal acquisition, i.e., according to the Figure 7The data sets or waypoints 68, 70, 72, 74, 76 according to the present disclosure are organized both temporally and spatially. Furthermore, the data sets or waypoints 68, 70, 72, 74, 76 can also be organized hierarchically internally, such as those already organized according to the spatial hierarchy. Figure 4 As stated.

[0069] Figure 8 The application of a multi-modal data set or waypoint 78 for functional mapping is shown. To this end, the operator can use the data set or waypoint 78 to interpret the functional data in real time and intuitively display the functional data on the patient's anatomical structure. Figure 8 In the example, the results of the histological examination are directly mapped onto the patient's anatomy. In practice, this mapping can be achieved, for example, using colored markings. Areas marked with a T, for example, identify tumor tissue, while areas marked with a G identify healthy tissue. This functionally mapped, multi-modal data set or waypoint 78 facilitates optimal tumor resection.

[0070] Reference Signs List

[0071] 1Surgical auxiliary system

[0072] 2 robots

[0073] 4Robot base

[0074] 8 robotic arms

[0075] 10, 12 robot arm segments

[0076] 14 joints

[0077] 16 robot arm end section

[0078] 18 Visualization System

[0079] 20 surgical microscope

[0080] 22 panoramic cameras

[0081] 24 control units

[0082] 26 navigation cameras

[0083] 28 Patient Trackers

[0084] 30 Visual System Tracker

[0085] 32 surgical instruments

[0086] 34Surgical Instrument Tracker

[0087] 36 patients scanned

[0088] 40 Registration Transformation

[0089] 42 Surgical microscope photos

[0090] 44 surgical microscope field of view

[0091] 46 panoramic camera photos

[0092] 48 panoramic camera fields of view

[0093] 50 robot arm positions

[0094] 52 Endoscope

[0095] 54 Histogenesis Data

[0096] 56 Electrophysiology data

[0097] 58 Patient coordinate system

[0098] 60+ datasets

[0099] 62 Surgeon Camera Photos

[0100] 64-78 multi-modal datasets

[0101] 100 steps of navigation visualization system

[0102] 200 Create and provide data step of the first data form

[0103] 300 Determine the location of the visualization system

[0104] 400 Allocation Steps

[0105] 500 Input data of the second data form step

[0106] 600 Allocation Steps

[0107] 700 Generate View Steps

[0108] 800 Output view step.

Claims

1. A surgical assistance system (1) comprising a navigated surgical robot (2) for use in performing a surgical intervention on a patient, the assistance system comprising: a robot base (4) as a local connection point of the robot (2) and a movable robot arm (8) connected to the robot base (4), the robot arm having at least one robot arm segment (10, 12), a visualization system (18) connected to the robotic arm (8) and adapted to create and provide at least one real-time, in-vivo picture (42, 46), a navigation system (26, 30) adapted to determine at least one position of the visualization system (18) and thus the position of the at least one real-time, in-vivo image (42, 46) in spatial relation to the patient, and a control unit (24) adapted to assign a position in the patient's coordinate system (58) to the real-time in-vivo images (42, 46) and to store the position, It is characterized in that The control unit (24) is adapted to, - inputting the data (38, 54, 56) collected in a second data format different from that of the visualization system into the data information system of the control unit (24) via a data interface, - assigning at least one position, in particular an orientation, to the input data (38, 54, 56) in the coordinate system (58) of the patient and storing it together with the position in a data information system, - generating a view of the real-time in-vivo photograph (42, 46) together with those data of the input data (38, 54, 56) which are spatially arranged within the real-time in-vivo photograph (42, 46) based on the position of the data, and outputting the view via a display device, in particular a surgical monitor.

2. The surgical auxiliary system (1) according to claim 1, characterized in that The data (38, 54, 56) acquired in the second data form are image data (38, 54) and / or measurement data (56) of the patient and / or operating data of the surgical robot (2).

3. The surgical assistance system (1) according to any one of the preceding claims, characterized in that The control unit (24) is adapted to permanently store the data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) acquired in real time in the data information system and / or to retrieve permanently stored data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) acquired previously in real time from the data information system.

4. The surgical auxiliary system (1) according to claim 3, characterized in that The data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) acquired in real time include at least one real-time navigated position of the robot (2) in the coordinate system (58) of the patient and / or at least one real-time in-vivo photograph (42, 46) and the orientation of the photograph and / or preferably include the real-time navigated position of the instrument (32).

5. The surgical auxiliary system (1) according to claim 3 or 4, characterized in that An operator interface is adapted to selectively select elements of the data set (60, 64, 66, 68, 70, 72, 74, 76, 78) for output or to deselect elements of the data set from output.

6. The surgical assistance system (1) according to any one of the preceding claims, characterized in that The control unit (24) is adapted to output the in-vivo photographs (42, 46) of the visualization system (18) and / or the data (38, 54, 56) acquired in the second data form at least in an organized manner according to the data form of the data and / or the size of the field of view (44, 48) and / or the viewing angle of the field of view (44, 48) and / or according to the acquisition time (t).

7. A method for visualizing data during a surgical, in particular neurosurgical, intervention on a patient, in particular for a surgical assistance system (1) according to any one of the preceding claims, comprising the following steps: (100) navigating a visualization system (18) spatially associated with the patient by means of a control unit (24), said visualization system being connected to the robotic arm (8); (200) creating and providing, in particular digitally creating and providing, at least one real-time in-vivo image (42, 46) by means of the visualization system (18); (300) determining the position of the visualization system (18) and thus the position of the at least one real-time, in-vivo image (42, 46) in spatial relation to the patient using a navigation system (26, 30); (400) assigning, by means of the control unit (24), a position in the patient's coordinate system (58) to the real-time, in-vivo images (42, 46) and storing the position and the images (42, 46) in a data information system of the control unit (24); It is characterized by the following steps: (500) inputting data (54, 56) acquired in a second data format different from that of the visualization system (18) into the data information system by means of a data interface of the control unit (24); (600) assigning a position in the patient's coordinate system (58) to the input data (54, 56) and storing the position and the input data (54, 56) by means of the control unit (24); (700) generating, by means of the control unit (24), a view of the live, in-vivo photograph (42, 46) together with those of the input data (54, 56) that are spatially arranged within the live, in-vivo photograph (42, 46) based on the position of the data; as well as (800) Outputting the view via preferably a display device, in particular a surgical monitor.

8. The method for data visualization according to claim 7, characterized by the following steps: - permanently storing the data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the assistance system (1) acquired in real time in the data information system by means of an operator interface and the control unit (24), and / or - calling up permanently stored, previously real-time acquired data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the auxiliary system (1) from the data information system with the aid of an operator interface and the control unit (24), wherein the data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) include at least one real-time navigated position of the robot (2) in the coordinate system (58) of the patient, preferably including joint angles (36) of the robot, and / or the at least one real-time, in-vivo photograph (42, 46) and the orientation of the photograph, and / or the real-time navigated position of the instrument (32).

9. The method for data visualization according to claim 8, characterized in that The method further comprises the following steps: - calling up permanently stored data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) of the auxiliary system (1) from the data information system, The above steps are triggered by the control unit (24) of the assistance system (1), in particular the control unit of the robot arm (8), moving to the position of the stored data set (60, 64, 66, 68, 70, 72, 74, 76, 78).

10. The method for data visualization according to claim 8 or 9, characterized by the following steps: - Assigning a selection of data acquired in a second data form before and / or during a surgical intervention to the stored data sets (60, 64, 66, 68, 70, 72, 74, 76, 78) by means of a user interface and the control unit (24).

11. The method for data visualization according to any one of claims 8 to 10, characterized by the following steps: - Displaying the location of the stored data set (64) in a scan (38) or image of the patient.

12. The method for data visualization according to any one of claims 8 to 11, characterized by the following steps: - History of building datasets (68, 70, 72, 74, 76, 78).

13. The method according to any one of claims 8 to 12, characterized in that Here is another step: - Measure the differences between photos and / or scans of the same data format and orientation, but acquired at different times.

14. The method according to any one of claims 7 to 13, characterized in that The method further comprises the following steps: - marking regions and / or coordinates within a real-time, in vivo photograph and / or in data acquired in a second data format different from that of the visualization system, - assigning functions and / or characteristics and / or parameters (G, T), and - storing said marking and said assignment in a data information system. 15 . A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to implement the method steps of the method for data visualization according to claim 7 .

Images