Robotic telemanipulation system with adaptable level of autonomy, use of a robotic telemanipulation system, autonomous medical robot system

Abstract

Robotic telemanipulation system with adaptable level of autonomy, including: - a control device for a user; - at least one manipulation device; and - a bidirectional data transmission device; wherein the control device is designed to convert user inputs into control signals for controlling the manipulation device and to transmit the control signals to the manipulation device via the data transmission device, wherein the manipulation device is designed to receive control signals from the bidirectional data transmission device, to convert the control signals into actions of the manipulation device and to return feedback signals via the data transmission device to the control device, wherein the control device is further configured to output the feedback signals of the manipulation device to the user and wherein the control device comprises a database and / or a database interface and a control unit, wherein the control unit is designed to receive initial information from the database and / or a database interface and / or to store secondary information in the database and / or to transfer it to the database interface for storage, where the control unit selects a degree of autonomy for converting user inputs into control signals for the manipulation device, depending on the initial information. characterized by the fact that The control unit is trained to choose the level of autonomy in a process of reciprocal learning, whereby the robotic telemanipulation system initially acts with little or no autonomy and, over the course of a user's usage time, the system acts more autonomously based on knowledge of the user's preferences.

Description

[0001] The present invention relates to a robotic telemanipulation system with an adaptable degree of autonomy, a control device for a robotic telemanipulation system, a method for adapting the degree of autonomy of a robotic telemanipulation system, the use of a robotic telemanipulation system for performing a medical procedure, in particular in the field of soft tissue surgery, an autonomous medical robot system for diagnostics and / or therapy, and a method for adapting a degree of autonomy of a telemanipulation system.

[0002] A general trend in surgery is the desire to minimize trauma caused by medical procedures in order to reduce patient pain and accelerate healing. While this benefits the patient, it necessitates more complex techniques in medical interventions, such as operations, and an increasing reliance on technology in the operating room, as the surgeon loses direct access to the surgical field. These conditions place ever greater cognitive demands on medical personnel, potentially leading to overload in critical situations and for less experienced staff.

[0003] Relieving the burden on medical personnel, such as surgeons, through technical systems would therefore be desirable. However, due to the critical nature of such applications, any such technical solution must meet high safety requirements and, given increasing cost pressures, offer clear added value to medical personnel and / or the patients being treated.

[0004] Medical robotic systems, through their combination of sensory and manipulation capabilities, have the potential to relieve the burden on medical personnel during surgery. However, the currently commercially available medical robotic systems for soft tissue surgery (da Vinci X, da Vinci Xi, Senhance) are purely telemanipulation systems. Autonomous medical robotic systems are currently commercially available, for example, in orthopedics (ROBODOC) and radiation therapy (CyberKnife), where patient referencing is simpler and the manipulated structures are less mobile compared to, for example, surgery and / or soft tissue surgery.

[0005] US 9,867,668 B2 describes a robotic system with at least one camera that calculates the mechanical properties of tissue, such as its stiffness, based on a tissue model and the tissue's behavior during interaction with an end effector of the robot, and visualizes this data as a property map. This property map is combined with the actual camera image to create a combined representation that is displayed on a screen for the surgeon or medical personnel.

[0006] US patent 10383694 B1 describes a machine-learning-based system for generating haptic feedback from endoscopic images. This involves identifying specific instrument-tissue interactions within the endoscope video stream. These interactions are then assigned specific force levels, which have been previously annotated for comparable interactions. These force levels are then relayed back to the surgeon as haptic feedback.

[0007] US Patent 2019 / 0175062 A1 describes a navigation aid for an endoscopic instrument used in the medical field of bronchoscopy. Based on data from a suitable sensor, the position and orientation of the instrument tip are calculated and visualized in a 3D model of the hollow organ and the target structure based on preoperative data. The perspective of the rendering may differ from the perspective of the endoscopic instrument.

[0008] US 10 229 753 B2 describes a gesture control system for navigating through medical image datasets, such as computed tomography (CT) scans. The link between gestures and image movements / manipulations is user-specific and configured via a database of presets.

[0009] Similarly, US 2020 / 0205914 A1 describes an interaction concept with a virtual three-dimensional patient model on a touchscreen, which can optionally be combined with gesture recognition. The user can move and rotate the model, adjust contrast and transparency, and hide or highlight individual objects.

[0010] US 10,786,315 B2 describes the control of the graphical user interface of a training simulator using the input devices on a surgeon's console of the da Vinci Xi system from Intuitive Surgical. For navigation within the user interface, degrees of freedom of the hand controllers can be locked, so that movement is limited to a plane. This movement is preferably chosen to maintain hand-eye coordination. Optionally, the hand controllers provide haptic feedback (e.g., via rotary and / or translational vibration devices) when interacting with the user interface.

[0011] WO 2019 / 136342 A1 describes a user interface for controlling a robotic endoscope. This includes two height-adjustable hand controllers (to allow the surgeon to work while sitting or standing), foot pedals, and a user interface consisting of a display / touchscreen and other input devices such as a keyboard, mouse, or microphone.

[0012] WO 2019 / 117926 A1 describes a graphical user interface for a medical robotics system. In addition to the endoscope image, this interface can display to the user the type and condition of the instruments used, information on the individual surgical steps, preoperative image data such as CT or ultrasound images, excerpts from the patient's medical record, or information on the robotic system (setup, surgery time, etc.), and allows the marking and annotation of areas in the endoscope image.

[0013] US patent 2009 / 0088774A1 discloses an interaction method with a medical robotics system in which individual degrees of freedom can be locked by dedicated activation of the hand controller (closing twice within a short time, then closing completely). This allows, for example, a gripper for holding tissue to be locked, or a gripper with a cautery function to be closed for the duration of the cauterization procedure. Unlocking the degrees of freedom and switching back to normal control is achieved by closing completely and then closing twice.

[0014] US 2018 / 0353245A1 describes the visualization of the movement possibilities of the end effectors of a single-port system using a graphical user interface. This involves, on the one hand, a 2D projection of the workspace in the direction of the positioning mechanism, and on the other hand, a linear scale along the axis of the positioning mechanism. Alternatively, a combination of a schematic view of the single-port system and a 2D projection of the workspace for the left and right instruments can be used.

[0015] WO 2020 / 261 956 A1 describes a system for controlling medical instruments or tools using a robotic telemanipulation system, in which different levels of autonomy for control can be manually specified.

[0016] Purely robotic telemanipulation systems, as are common in laparoscopy, do improve ergonomics and manipulation possibilities in the patient for the surgeon as the user of the system, but do not relieve the patient cognitively, especially since the control of the system must also be learned beforehand.

[0017] In autonomously operating medical robot systems, the physician, as medical personnel, only has a monitoring function; the actual patient interaction is performed solely by the robot. This can negatively impact acceptance, and furthermore, deviations from the planned procedure are often difficult to detect because the physician is no longer actively involved in the process (out-of-the-loop problem). Currently existing robotic systems are often inadequately able to react to deviations between the actual medical intervention and the planned procedure.

[0018] In contrast, a hybrid overall system of humans and robotic systems is desirable, combining their respective strengths (robotic system: factual knowledge, continuity, precision, repeatability, optimization capability, analytical, rational, objective ⇔ human: use of (implicit) experiential knowledge, flexibility, adaptability, responsiveness to unforeseen events, association, cognition, intuition).

[0019] The present invention is based on the objective of improving a robotic telemanipulation system and the use of a robotic telemanipulation system for performing a medical procedure, particularly in the field of soft tissue surgery, as well as an autonomous medical robot system for diagnostics and / or therapy, in such a way as to be able to significantly improve the degree of autonomy by automatically adjusting it while taking user preferences into account.

[0020] According to the invention, the aforementioned problem is solved by a robotic telemanipulation system according to claim 1, the use of a aforementioned telemanipulation system according to claim 10, by an autonomous medical robot system according to claim 11, by a control device for a robotic telemanipulation system with an adaptable degree of autonomy according to claim 12, and by a method for adapting the degree of autonomy of a telemanipulation system. The dependent claims relate to advantageous embodiments of the respective subject matter of the invention.

[0021] A robotic telemanipulation system according to the invention with an adaptable degree of autonomy comprises a control unit for a user, at least one manipulation unit, and a bidirectional data transmission unit. The control unit is configured to convert user inputs into control signals for controlling the manipulation unit and to transmit the control signals to the manipulation unit via the data transmission unit. The manipulation unit is configured to receive the control signals from the bidirectional data transmission unit, to convert the control signals into actions of the manipulation unit, and to return feedback signals to the control unit via the data transmission unit.The control device is further configured to output the feedback signals from the manipulation device to the user, the control device comprising a database and / or a database interface and a control unit.

[0022] The control unit is designed to receive information from the database and / or a database interface and / or to store second pieces of information in the database and / or to transfer them to the database interface for storage.

[0023] The control unit is further designed in such a way that, depending on the initial information, it selects the degree of autonomy for converting user inputs into control signals for the manipulation device, and the control unit is further trained to select the degree of autonomy in a process of reciprocal learning, whereby the robotic telemanipulation system initially acts with little or no autonomy and, over the course of a user's usage, the system acts more autonomously based on knowledge of the user's preferences.

[0024] The phase in question could be, for example, the current stage of an operation or medical procedure being performed by the manipulation device on a human or animal body. Based on sensor data (poses and movements of the manipulation device, such as a robot), the system is able to recognize the forces acting on the manipulation device and any connected medical instruments, and based on formalized prior knowledge (the type of current operational phase of the medical procedure, the steps involved in this procedure). This allows the system to offer the user the most relevant functions from a wide range of possible support options.

[0025] The degree of autonomy of the system can change depending on the user's knowledge of the system and their preferences over the course of the collaboration with the respective user, such as a surgeon who has identified themselves to the robotic system using their personal user profile.

[0026] In a process of reciprocal learning, the robotic telemanipulation system initially operates with little or no autonomy. This serves two purposes: firstly, to facilitate the user's familiarization with and understanding of the system's functions, and secondly, to learn the user's preferences. Over time, the system becomes increasingly familiar with the user's preferences and can thus act more autonomously without its behavior becoming disruptive to the user. Conversely, the user becomes so familiar with the characteristics and processes of the system's functions that even autonomous execution of these functions does not result in opaque system behavior.

[0027] The degree of autonomy can also be changed based on the task, such that for critical tasks, such as the preparation of structures using the manipulation device during a surgical procedure, a lower degree of autonomy, i.e., a greater involvement of the user in the task definition, is offered than for non-critical tasks, such as guiding an endoscope through the manipulation device or guiding a suction device over the manipulation device.

[0028] Preferably, the first piece of information for choosing the degree of autonomy can be selected based on criteria from the group of: a current phase of a task to be performed by the manipulation device, the system knowledge and preference of the user and / or the task to be performed by the manipulation device.

[0029] The level of autonomy can be selected from the following groups, which are particularly preferred: manual control, action support, batch processing, joint control, decision support, mixed decision-making, rigid system, automated decision-making, higher-level control, or full automation.

[0030] In the aforementioned definitions, Manual Control corresponds to manual control, Action Support to action support, Batch Processing to batch processing, Shared Control to shared control, Decision Support to decision support, Blended Decision Making to mixed decision making, Rigid System to rigid system, Automated Decision Making to automated decision making, Supervisory Control to higher-level control, and Full Automation to full automation.

[0031] In the collaboration between humans and (semi-)autonomous technical systems, different levels of autonomy can be distinguished. Endsley and Kaber consider the four tasks of monitoring (receiving all information relevant for perceiving the system status), generation (formulating options or strategies for achieving a goal), selection (deciding on an option or strategy), and implementation (carrying out and implementing the selected option through control measures at an interface) of actions in order to distinguish ten levels of autonomy of technical systems based on these tasks. [Kaber DB, Endsley MR. The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science 2004; 5(2):113-53.]

[0032] As in Fig. As shown in Figure 2, the aforementioned degrees of autonomy differ in the agency of the two actors, human (M) and computer (C). In this application, medical robotics systems can also be used instead of computers. The human actor is the user of the medical robotics system, and the computer is the combination of the control unit with at least one manipulation device. The [details of the following are missing from the original text] Fig. The two autonomy levels highlighted in bold – 2 Action Support, 6 Blended Decision Making and 9 Supervisory Control – are presented below for various tasks in the areas of navigation and endoscope guidance: Action Support: The computer assists the human in performing selected actions. This requires human oversight. A common example is a teleoperation system, where the operator works based on human input through a control device. Blended Decision Making: The computer generates a list of decision options, selects one, and executes it with the human's consent. Alternatively, the human can select another option generated by the computer or create their own options. The computer then executes the selected action, for example, using a manipulation device.

[0033] Supervisory Control: The computer generates options, selects one for implementation, and executes it. A human monitors the actions and intervenes only when necessary. In the event of intervention, the human selects a different option, either generated by the computer or by the human themselves.

[0034] In reciprocal learning of a robotic system, both the behavior of the system user and the behavior of the robotic system change over the course of the collaboration. This alters how, in Fig. 3 shown, using the two functions of intraoperative navigation, as in Fig. 3a shown and endoscope guidance, as in Fig. Figure 3b illustrates the gradual division of responsibilities between humans and robotic systems over time. With complementary functional design, as in the case of intraoperative navigation, actions originally performed by humans are increasingly taken over by the system. With context-sensitive functional design, as in the case of endoscope guidance, the system operates largely autonomously from the outset (supervisory control), but improves its autonomous behavior over time by learning the surgeon's individual habits and preferences.

[0035] Telemanipulability: The user must be able to fully remotely manipulate the system during critical surgical steps, in case of complications or unclear surgical situations, and in the event of potential malfunctions. This requires, on the one hand, appropriate input and output options, such as those already offered by some currently available commercial systems. On the other hand, the user must be able to pause and / or completely abort the autonomous processing of tasks at any time.

[0036] Use of different input modalities: Depending on the tasks to be controlled, various input options should be used. The primary goal is ease of use and intuitive understanding. While hand controllers can be used for precise movements, or gesture recognition with limitations (no haptic feedback, precise movements difficult over extended periods), buttons, switches, and / or foot pedals allow for the dedicated triggering of functions (e.g., activating cautery current for sclerotherapy).

[0037] Touchscreens and voice input devices are particularly suitable for intraoperative interaction with the robotic system (e.g., marking and naming structures, activating support functions). Eye tracking offers a possible alternative to hand controller control for moving the endoscope camera.

[0038] Use of different feedback modalities: Similarly, depending on the type of feedback to the user, different output options should be used. These can stimulate the user's visual (display on the (3D) screen, status LEDs), auditory (warning tones, attention tone after activating voice input), or haptic (force feedback or vibrotactile feedback via the hand controllers or additional devices such as the Vibrotac wristband) sensory channels. Care must be taken to ensure that no sensory channel is overloaded and that the user is not cognitively overwhelmed by a large number of messages. The latter requires a clear prioritization and labeling of the messages (e.g., in ascending order of importance: Notification => Warning => Hazard Warning => Critical Warning).

[0039] According to the invention, various databases or interfaces to databases can be used, such as in particular: A functional database: This persistent database, present from the start of the system and subsequently expandable by the user and / or the system manufacturer, stores the existing prior knowledge for individual procedures. This includes, for example, prior knowledge of anatomy (to automatically name vessels, for instance), process models for the sequence of various surgical procedures, and descriptions of individual elementary actions (e.g., tightening knots). These elementary actions are preferably described in a parameterizable manner so that they can be adapted to individual surgeons and procedures by loading corresponding parameter sets.

[0040] A user database: This persistent database, populated by system users and the system itself, stores the support functions requested by the surgeon, which the surgeon can select and modify via a graphical user interface (GUI). It also stores the surgeon's learned preferences, for example, as parameter sets for basic actions in the function database.

[0041] An external memory: This temporary database for intraoperative data stores information such as markings and timers placed by the surgeon. Additionally, the materials and instruments used, tissue samples taken, and a preliminary version of the surgical protocol are stored there and can be viewed and modified at any time by the surgeon and / or the operating room staff.

[0042] An interface to the patient database of a medical facility, such as a hospital: This connection serves, on the one hand, to make patient data (e.g., preoperative image data) directly available on the system. On the other hand, it enables the export of all information from the external memory that should also be permanently available after the procedure, such as the surgical protocol, video recordings, photos of the procedure, or logged system data from the robot.

[0043] The control device may preferably also include at least one input interface for receiving user input and at least one output interface for outputting feedback signals from the manipulation device to the user.

[0044] The at least one input interface can consist of at least one element selected from the group of a hand controller, a gesture recognition system, a system for tracking the user's eyes or eye movements, switches, buttons, foot pedals, touchscreens and / or voice input.

[0045] The at least one output interface can be formed by at least one element selected from the group of a visual output device, an auditory output device and / or a haptic output device.

[0046] According to the invention, multiple input and output interfaces can be provided. For example, additional interfaces for sterile operating room personnel at the operating table can be provided, such as RGB LEDs on the robot as outputs, and buttons and torque sensors integrated into the robot as inputs for direct control of the robot (hands-on operation). In the system according to the invention, when multiple input devices are provided, a prioritization is specified for how the different inputs are to be prioritized or, if necessary, blocked (so that, for example, the assistant physician at the table does not interfere with the surgeon's telemanipulation by moving the robot).

[0047] The database can be formed from at least one element selected from the group of a function database, a user database and / or an external memory or external data storage.

[0048] A database interface can be formed by an interface to a patient database of a medical institution, such as a clinic.

[0049] The interface to the clinic's patient database serves two purposes: firstly, to make patient data, such as pre-operative image data, directly available on the robotic telemanipulation system; and secondly, to export all information from the external memory or data storage that should remain permanently available even after the procedure, such as the protocol of the medical procedure, video recordings or photos of the procedure, or locked system data from the telemanipulation device, such as a robot.

[0050] The at least one manipulation device can be designed to carry out medical interventions, in particular for therapeutic measures and / or surgical procedures on the animal and / or human body.

[0051] According to the invention, the use of a robotic telemanipulation system according to the first aspect of the present invention for performing a medical procedure, particularly in the field of soft tissue surgery, can also be provided. In particular, an autonomous medical robotic system for performing diagnostic tasks and / or medical therapies, comprising a robotic telemanipulation system according to the first aspect of the present invention, can also be provided.

[0052] According to a further aspect, the present invention relates to a control device for a robotic telemanipulation system with an adaptable degree of autonomy, wherein the control device is designed to convert user inputs into control signals for controlling a manipulation device and to transmit the control signals to the manipulation device via a data transmission device.

[0053] The control device is further configured to output feedback signals from the manipulation device to the user, and the control device comprises a database and / or a database interface and a control unit, the control unit being configured to receive first information from the database and / or a database interface and / or to store second information in the database and / or to transfer it to the database interface for storage.Depending on the initial information, the control unit selects a degree of autonomy for converting user inputs into control signals for the manipulation device. The control unit is further trained to select the degree of autonomy in a reciprocal learning process, whereby the robotic telemanipulation system initially acts with little or no autonomy, and over the course of a user's usage, the system becomes more autonomous based on knowledge of the user's preferences.

[0054] The aforementioned control device can have the same optional features as previously described with regard to the robotic telemanipulation system.

[0055] The following describes an embodiment of the invention for minimally invasive surgery. However, the fundamental concepts of the invention are also applicable to robotic systems for diagnostics and therapy in other medical fields (e.g., orthopedics or neurosurgery). The surgeon described is the user of the robotic telemanipulation system according to the invention with an adaptable degree of autonomy, in which the surgeon provides inputs for controlling the telemanipulation system via the control unit.

[0056] According to an additional aspect, the present invention relates to a method for automatically adapting the degree of autonomy of a robotic telemanipulation system. The robotic telemanipulation system preferably has the features from one of the preceding aspects of the present invention. The method according to the invention comprises the steps: - Retrieving initial information from a database and / or via a database interface; - Selecting the degree of autonomy depending on the initial information retrieved for converting user inputs into control signals of a manipulation device, wherein the selection of the degree of autonomy takes place in a process of reciprocal learning, with the robotic telemanipulation system initially acting with little or no autonomy and, over the course of a user's usage time, the system acting more autonomously based on knowledge of the user's preferences; - Converting the user's inputs into control signals to control the manipulation device, taking into account the previously selected level of autonomy; - Execution of control signals by the manipulation device and simultaneous reception of feedback signals; - Outputting feedback signals to the user; and optionally - Storing secondary information in the database and / or passing secondary information to a database interface for storage.

[0057] The procedure may also provide for the initial selection of the degree of autonomy based on criteria selected from the group of: a current phase of a task to be performed by the manipulation device, the user's system knowledge and preference, and / or the task to be performed by the manipulation device.

[0058] Furthermore, it can be provided that the level of autonomy is selected from the group of: manual control, action support, batch processing, joint control, decision support, mixed decision-making, rigid system, automated decision-making, higher-level control or full automation.

[0059] The method according to the invention can also be combined with the optional features already described in the preceding aspects of the invention. Furthermore, it is possible to implement the method using a computer.

[0060] The Fig. Figure 1 schematically depicts a medical robotics system for minimally invasive surgery: One or more robots, positioned near the patient on the operating table (right), manipulate the endoscope and surgical instruments during the procedure. They are controlled by a surgeon at a console (left). Often, the surgeon is in the same operating room as the patient, as this simplifies coordination with the other surgical staff. However, telesurgery also works when the surgeon and patient are geographically separated, allowing, for example, a remote expert to be consulted for a complex procedure with minimal effort.Bidirectional data transmission takes place between the surgeon's console as the control unit and the robots as manipulation devices: On the one hand, control commands from the surgeon are transmitted to the robot arms, which then perform specific manipulation tasks. On the other hand, information from the surgical field (e.g., images from the endoscope camera or tissue interaction forces) is transmitted to the control unit, for example, in the form of the surgeon's console, and fed back to the surgeon as the user in a suitable manner (e.g., visually or haptically).

[0061] For quick and intuitive navigation through the application, a flat interaction architecture should be used if possible (see below). Fig. 4) In the present application concept, the system is controlled via three levels: user profile, procedure overview, and home screen. The user profile allows the surgeon to define individual preferences and workflow, as well as access the system's learning status. The procedure overview contains all relevant information and data about the patient and the procedure. Additionally, preoperative data can be selected and the patient referenced via this level. Fig. Figure 4 was divided onto two sides of the figure to improve the visibility of the features, with the two parts of the figure connecting to each other in the area of ​​the dashed lines.

[0062] The one in the Fig. The home screen shown in Figures 5a to 5d forms the system's main interface for the intraoperative phase. Here, the surgeon can access all system functions relevant to the intraoperative period. The home screen interface is visually unobtrusive, displaying only the information necessary for the situation, so as not to distract the surgeon from the primary task of operating.

[0063] The HOMEBAR contains all the information and functions that the surgeon needs to have constantly available. This includes instrument and endoscope status, as well as options for intervening in the autonomous endoscope guidance. These are operated via voice control. QUICK ACCESS consists of three main functions: setting an overlay, showing or hiding the overlay, and manual endoscope guidance. These functions are frequently used and therefore must be quickly accessible. Like the HomeBAR, Quick Access is also operated via voice control. It is hidden by default and is only displayed and activated by a voice command using a code word.

[0064] The sub-functions of each main quick access function can be accessed via the right-hand sidebar. A hand controller is used as the input and output interface for this purpose. Behind each sidebar is an element directory, which manages and organizes the elements created during the interaction, such as markers, lines, or locks. This directory can be opened by double-closing the hand controller. The direct selection window contains all the settings that can be applied to each overlay type (line, marker, point). It appears after placing an overlay or after selecting an existing overlay with a click.

[0065] The left sidebar provides access to the preoperative image data loaded in the procedure overview and the associated editing and manipulation tools. The left sidebar also has a corresponding element directory, which can be opened, for example, by double-clicking on the left edge of the screen and manages further image data and existing elements.

[0066] The Fig. 6, Fig. 7, Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15, Fig. 16, Fig. 17, Fig. 18, Fig. 19 to Fig. 20 describe methods for the user to interact with the robotic system at the three different levels of autonomy: Action Support, Blended Decision, and Supervisory Control.

[0067] The in Fig. The procedure for preparing the procedure, as shown in Figure 6, is identical for the different levels of autonomy of the robotic system. To register the patient with the robotic system, the surgeon, in hands-on mode, moves the robot arm or manipulation device as described under Referencing. This is supported by a visualization of the points to be approached.

[0068] In the Action Support autonomy level, the system is fully manually instructed and controlled by the surgeon. Functions are selected and executed manually. The system only passively supports the surgeon, for example, through safety features. During this phase, the surgeon learns about the functions, their advantages, and their application.

[0069] Fig. Figure 7 illustrates the workflow of autonomous endoscope guidance for the Action Support autonomy level. The two screenshots demonstrate the visualization of the current (top) and the next focus area (bottom), which informs the surgeon about the endoscope movement planned by the system (change in orientation and increase / decrease of the distance to the surgical field). The robotic system automatically ensures compliance with the trocar conditions for the endoscope.

[0070] Fig. Section 8 describes the two possible procedures for manually controlling the endoscope in the Action Support autonomy level: Either the surgeon activates manual endoscope guidance via voice command and can then control the endoscope movements directly using the hand controllers. Alternatively, the surgeon can navigate to saved endoscope positions, which were previously saved using the "MIRO - Save" command. To do this, the hand controllers are released using "MIRO - Release," which is confirmed by an increasing vibration frequency. The right sidebar can then be displayed by clicking. A double-click (i.e., double-closing the hand controllers) expands the element directory. After another double-click on the corresponding saved position, the endoscope moves to that position.

[0071] At the Action Support autonomy level, locking a security area via overlay, as in Fig. Figure 9 illustrates the numerous manual steps required by the surgeon: After activating the overlay function, the robotic system first decouples the hand controllers and instruments and displays a quick access bar with a pre-selected marking tool to the surgeon. By closing and then moving a hand controller (typically the surgeon's dominant hand; alternatively, two-handed movement is possible after both hand controllers are coupled), the surgeon marks the desired overlay area in the endoscope image. This area is saved after the hand controller is opened (here, the surgeon's two-dimensional marking is mapped onto the three-dimensional anatomical structures in the patient underlying the endoscope image), and a direct selection window with manipulation options for the marking is displayed to the surgeon.The surgeon can name the marker, specify whether it should be automatically expanded during the procedure, and define the type of restriction (no-go area or vibration restriction). In the first case, a repulsive virtual fixture prevents [the procedure]. Fig. 10a) In the first case, the surgeon receives a vibration alarm (vibrotactile feedback) as a warning upon penetration of the surgical instrument into the restricted area. The visualization of the restricted area via overlay differs depending on the type of restriction and can be displayed by the surgeon at any time (in which case the restriction is permanently visible as a colored overlay; a light blue overlay is used in the illustrations, as this color does not occur inside the body) or hidden (in which case the restriction is only displayed when the restricted area is violated in the vicinity of the violating instrument).

[0072] The Fig. Sections 10a to 10d provide an overview of the different forms of virtual fixtures: On the one hand, a distinction can be made between regional constraints ( Fig. 10a), which restrict the user's movement space, and guiding constraints ( Fig. 10b), which guide the user along a predetermined path. The latter are often perceived as more dominant by users. On the other hand, with regard to the type of movement influence, a distinction can be made between Attractive ( Fig. 10c), i.e. attractive, and repulsive ( Fig. 10d), i.e., repulsive, constraints are distinguished. In the case of the no-go area, for example, repulsive regional constraints are used. Depending on the parameterization of the virtual fixtures, they can feel very rigid ("virtual wall" / "guide rail") or only slightly influence the surgeon's movements. It is also possible to make these fixtures overridable, i.e., to deactivate them as soon as the surgeon has exceeded their boundary far enough to indicate a conscious action on the part of the surgeon. The appropriate parameterization of the virtual fixtures is a complex task, which is influenced by numerous objective factors (e.g., size and position of the geometries, procedure sequence) as well as by the surgeon's subjective preferences.

[0073] The in Fig. Figure 11 shows an overlay of the endoscope image with a fluorescence view, where the fluorescence view is superimposed on the entire endoscope image, in which Fig. Figure 11 shows highly perfused areas in the fluorescence view, represented as hatched areas. The overlay allows for the identification of near-surface perfused structures, such as large vessels. At the Action Support autonomy level, the sidebar (right) is opened via voice command to activate or deactivate this view, and the fluorescence view option is manually selected using the hand controllers. At the Blended Decision autonomy level, the system automatically selects the fluorescence view as the most suitable option after the user's voice command. At the Supervisory Control autonomy level, both the decision to display an overlay and the selection of overlays are made autonomously by the system.

[0074] Fig. Section 12 describes the procedure for creating reminder messages linked to structures or objects for the Action Support autonomy level. The surgeon activates the overlay function via voice command, then selects the point tool in the sidebar using the hand controllers (which are decoupled from the instruments), and moves it to the desired structure / object. Closing the hand controller selects the structure / object, which can then be named, for example, via voice input. Optionally, a timer can also be activated to continuously monitor ischemia times, for example, when clamping vessels. While the timer is active, it is displayed to the surgeon above the home bar, and its expiration is indicated both visually and haptically.

[0075] As in Fig. As can be seen in Figure 13, in the Action Support autonomy level, the optimized alignment of the image data takes place in two phases. First, the 3D model of the surgical area calculated from the preoperative image data is aligned using the preoperative referencing (see Figure 13). Fig. 6) Roughly aligned. If further improvement of the alignment is desired, the surgeon must select additional reference points on the 3D model and in the endoscope image in a second phase (for example, during the surgical procedure). The alignment of the 3D model is optimized after each reference point is set by minimizing the deviation between real and virtual structures across all reference points. Reference points can be set or deleted at any time during the procedure. Alternatively, the alignment optimization could be performed not automatically after the reference points are set, but only upon user request. This is particularly useful if the optimization requires a comparatively high amount of processing time.

[0076] Once the preoperative image data are referenced to the endoscope image, they can be used as described in the Fig. Figures 14a to 14d show that the data can be made available to the surgeon in a variety of ways directly on the home screen: The surgeon can selectively highlight individual structures in the datasets and freely move and measure the 3D model. Direct access to the 2D data is also possible.

[0077] At the Blended Decision autonomy level, the robotic system independently makes suggestions to the user, which the user can either confirm directly or modify.

[0078] How the comparison of Fig. 15 and Fig. As illustrated in Figure 9, when defining and locking the overlay area in the Blended Decision scenario, the system automatically suggests manipulation functions, the expansion of the locked area, and the appropriate locking method. The surgeon can confirm or adjust these suggestions.

[0079] Compared to Fig. 12 selects the system when defining memories in the Blended Decision scenario (see Fig. 16) automatically selects the appropriate overlay tool for the situation, and the system also automatically determines the appropriate type of reminder (e.g., timer).

[0080] At the in Fig. In the Blended Decision scenario (shown in section 17), the system activates coarse global alignment by default and automatically displays preferred editing options when referencing preoperative image data. When the referencing tool is activated, the system automatically suggests two matching reference points in the 3D model and the endoscope image, which the surgeon can either adjust or accept directly.

[0081] From the Blended Decision autonomy level onwards, the user has access to the system's learning status. This is displayed in an overview view (see below). Fig. 18) to deactivate or delete individual actions of the autonomously performed actions of the system (in the example the automatic locking of marked areas, the slide switch is used to deactivate, the cross to delete).

[0082] At the Supervisory Control autonomy level, the system executes most functions independently following an initial command from the surgeon. The surgeon is only informed by the visual effects of an action. If an action needs to be changed or reversed, deliberate intervention by the surgeon is required.

[0083] In the case of the overlay ( Fig. 19) In Supervisory Control autonomy mode, the system can intelligently adjust the previously selected settings for the type of blockage and visibility intraoperatively as needed to the specific surgical situation (e.g., automatic change of the blockage to vibration if the surgeon needs to manipulate the blocked structure according to the surgical plan). Furthermore, defined blockage areas can be automatically enlarged to include freely dissected areas; that is, the blockage grows continuously from the original structure during the course of the procedure.

[0084] The alignment of preoperative image data is performed automatically in the Supervisory Control scenario (see below). Fig.20): During the procedure, the system automatically and in the background sets additional reference points and updates the orientation of the 3D model to ensure optimal alignment of the preoperative image data at all times. Furthermore, global alignment is enabled by default, and preferred editing options are automatically displayed. This also applies to all two-dimensional image data.

Claims

[1] Robotic telemanipulation system with adaptable level of autonomy, comprising: - a control device for a user; - at least one manipulation device; and - a bidirectional data transmission device; wherein the control device is designed to convert user inputs into control signals for controlling the manipulation device and to transmit the control signals to the manipulation device via the data transmission device, wherein the manipulation device is designed to receive control signals from the bidirectional data transmission device, to convert the control signals into actions of the manipulation device and to return feedback signals via the data transmission device to the control device, wherein the control device is further configured to output the feedback signals of the manipulation device to the user and wherein the control device comprises a database and / or a database interface and a control unit, wherein the control unit is designed to receive initial information from the database and / or a database interface and / or to store secondary information in the database and / or to transfer it to the database interface for storage, wherein the control unit selects a degree of autonomy for converting user inputs into control signals for the manipulation device, depending on the initial information, characterized by , that The control unit is trained to choose the level of autonomy in a process of reciprocal learning, whereby the robotic telemanipulation system initially acts with little or no autonomy and, over the course of a user's usage time, the system acts more autonomously based on knowledge of the user's preferences. [2] Robotic telemanipulation system according to claim 1, wherein the first information for choosing the degree of autonomy is based on the criteria from the group of: a current phase of a task to be performed by the manipulation device, the system knowledge and preference of the user and / or the task to be performed by the manipulation device is selected. [3] Robotic telemanipulation system according to claim 1 or 2, wherein the degree of autonomy is selected from the group consisting of: manual control, action support, batch processing, joint control, decision support, mixed decision making, rigid system, automated decision making, higher-level control or full automation. [4] Robotic telemanipulation system according to one of the preceding claims, wherein the control device further comprises at least one input interface for receiving user inputs and at least one output interface for outputting feedback signals from the manipulation device to the user. [5] Robotic telemanipulation system according to claim 4, wherein the at least one input interface is formed from at least one element selected from the group consisting of: hand controller, gesture recognition system, eye tracking system, biosignal measurement devices (such as EMG for measuring muscle activity), foot pedals, touchscreens and / or voice input. [6] Robotic telemanipulation system according to claim 4 or 5, wherein the at least one output interface is formed from at least one element selected from the group consisting of: a visual output device, an auditory output device and / or a haptic output device, such as in particular a vibrotactile output device. [7] Robotic telemanipulation system according to one of the preceding claims, wherein the database is formed from at least one element selected from the group of: a function database, a user database and / or an external memory. [8] Robotic telemanipulation system according to one of the preceding claims, wherein the database interface is formed by an interface to a patient database of a medical facility, such as a clinic. [9] Robotic telemanipulation system according to one of the preceding claims, wherein at least one manipulation device is designed to perform medical interventions, in particular for therapeutic measures and / or surgical interventions on the animal and / or human body. [10] Use of a robotic telemanipulation system according to any one of claims 1 to 9 for performing a medical procedure, in particular in the field of soft tissue surgery. [11] Autonomous medical robot system for diagnostics and / or therapy, comprising a robotic telemanipulation system according to any one of claims 1 to 9. [12] Control device for a robotic telemanipulation system with an adaptable degree of autonomy, wherein the control device is designed to convert user inputs into control signals for controlling a manipulation device and to transmit the control signals to the manipulation device via a data transmission device, wherein the control device is further configured to output feedback signals from the manipulation device to the user and wherein the control device comprises a database and / or a database interface and a control unit, wherein the control unit is designed to receive initial information from the database and / or a database interface and / or to store secondary information in the database and / or to transfer it to the database interface for storage, wherein the control unit selects a degree of autonomy for converting user inputs into control signals for the manipulation device, depending on the initial information, characterized by , that The control unit is trained to choose the level of autonomy in a process of reciprocal learning, whereby the robotic telemanipulation system initially acts with little or no autonomy and, over the course of a user's usage time, the system acts more autonomously based on knowledge of the user's preferences. [13] Method for adapting the degree of autonomy of a robotic telemanipulation system, the system preferably having the features according to any one of claims 1 to 9, the method comprising the steps: - Retrieving initial information from a database and / or via a database interface; - Selecting the degree of autonomy depending on the initial information retrieved for converting user inputs into control signals of a manipulation device, wherein the selection of the degree of autonomy takes place in a process of reciprocal learning, with the robotic telemanipulation system initially acting with little or no autonomy and, over the course of a user's usage time, the system acting more autonomously based on knowledge of the user's preferences; - Converting the user's inputs into control signals to control the manipulation device, taking into account the previously selected level of autonomy; - Execution of control signals by the manipulation device and simultaneous reception of feedback signals; - Outputting feedback signals to the user; and optionally - Storing secondary information in the database and / or passing secondary information to a database interface for storage. [14] Method according to claim 13, wherein the first information for choosing the degree of autonomy is selected based on the criteria from the group consisting of: a current phase of a task to be performed by the manipulation device, the system knowledge and preference of the user and / or the task to be performed by the manipulation device. [15] Method according to claim 13 or 14, wherein the degree of autonomy is selected from the group consisting of: manual control, action support, batch processing, joint control, decision support, mixed decision making, rigid system, automated decision making, higher-level control or full automation.

Images