Graphical user guidance for robotic surgical systems

Abstract

Graphical user guidance for a robotic surgical system is provided. In one embodiment, a graphical user interface for a robotic surgical system includes a first area and a second area. The first area is used to display an endoscopic view of a surgical site within a patient's body captured by an endoscopic camera of the robotic surgical system, and the second area is used to display user feedback information. The graphical user interface overlays guidance messages on top of the endoscopic view of the surgical site in the first area to provide user instructions for interacting with a user input device to engage a robotic arm of the robotic surgical system. The present invention provides other embodiments.

Description

Technical Field

[0001] The following embodiments relate generally to the field of robotic surgery, and more particularly to graphical user guidance for robotic surgical systems. Background Art

[0002] Minimally invasive surgery (MIS) such as laparoscopic surgery involves techniques designed to reduce tissue damage during surgical procedures. For example, laparoscopic surgery typically involves making multiple small incisions in a patient's body (e.g., in the abdomen) and introducing one or more surgical instruments (e.g., end effectors, at least one camera, etc.) into the patient's body through the incisions. The surgical procedure can then be performed using the introduced surgical instruments, with visualization assistance provided by the camera.

[0003] Generally speaking, MIS provides multiple benefits, such as reducing scarring, alleviating pain, shortening recovery time, and reducing medical expenses associated with patient recovery. In some embodiments, MIS can be performed with a robotic system that includes one or more robotic arms for manipulating surgical instruments based on commands from an operator. The robotic arms can, for example, support various devices at their distal ends, such as surgical end effectors, imaging devices, cannulas for providing access to a patient's body cavities and organs, and the like.

[0004] In some embodiments, the operator may provide commands for manipulating the surgical instrument while viewing images provided by the camera and displayed to the user on a display. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1A An example of an operating room arrangement with a robotic surgical system and a user console of an embodiment is depicted.

[0006] Figure 1B is a schematic illustration of an exemplary variation of a robotic arm manipulator, tool driver, and cannula with a surgical tool of an embodiment.

[0007] Figure 1C is a schematic illustration of an exemplary user console of an embodiment.

[0008] Figure 2 is a schematic illustration of an exemplary variation of a user console of a robotic surgical system of an embodiment for communicating with one or more third party devices.

[0009] Figure 3 is a schematic diagram of a surgical robotic platform having an implementation of a graphical user interface (GUI) module, wherein the surgical robotic platform communicates with multiple medical data resources.

[0010] Figure 4A and Figure 4B 2 are a perspective view and a longitudinal cross-sectional view, respectively, of an exemplary variation of a handheld user input device of an embodiment.

[0011] Figure 5 is an illustration of a GUI for a robotic surgical system instructing a user to pick up an embodiment of a controller.

[0012] Figure 6 is a flow chart of a method of an embodiment for displaying data on a GUI of an embodiment robotic surgical system.

[0013] Figure 7 is an illustration of a GUI for a robotic surgical system instructing a user to match an embodiment of a grip.

[0014] FIG. 8A to FIG. 8F is an illustration of various graphical tools that may be used to instruct a user on an embodiment of a matching grip.

[0015] Fig. 9 is an illustration of a GUI for a robotic surgical system of an embodiment.

[0016] Fig.10 is an illustration of a GUI for a robotic surgical system showing an embodiment of a hover indicator when a user hovers his foot over a pedal.

[0017] Fig.11 is an illustration of a GUI for a robotic surgical system showing an embodiment of an energy activation indicator when a user depresses an energy pedal.

[0018] Fig.12 is an illustration of a GUI of a robotic surgical system displaying an embodiment of a notification of approaching a workspace limit.

[0019] Fig.13 is an illustration of a GUI of a robotic surgical system displaying an embodiment of a notification that a workspace limit has been exceeded.

[0020] Fig.14 is an illustration of a GUI of a robotic surgical system displaying an embodiment of a notification that a rotation limit has been reached.

[0021] Fig.15 is an illustration of a GUI of a robotic surgical system displaying an embodiment of a notification that eye contact has been lost.

[0022] Fig.16 is an illustration of a GUI of a robotic surgical system instructing a user to match grips after temporarily losing eye contact.

[0023] Fig.17is an illustration of a GUI of a robotic surgical system instructing a user to mate a grip after eye contact has been lost for more than thirty seconds.

[0024] Fig.18 is an illustration of a GUI of a robotic surgical system displaying an embodiment of a notification that a user input device has been dropped.

[0025] Fig.19 is an illustration of a GUI of a robotic surgical system of an embodiment instructing a user to tap a clutch pedal after a user input device has been dropped.

[0026] Figure 20 to Figure 23 is an illustration of a GUI for a robotic surgical system displaying an embodiment of an application in a display area of ​​a second region of the GUI.

[0027] Figure 24 to Figure 25 is an illustration of a GUI for a robotic surgical system of an embodiment displaying a CT image in a first area of ​​the GUI.

[0028] Fig.26 is an illustration of a GUI for a robotic surgical system of an embodiment in which a user instructs the robotic surgical system to fix a CT image to a display area in a second region.

[0029] Fig. 27 is an illustration of a GUI of a robotic surgical system of an embodiment in which a CT image is pinned to a display area in a second region. DETAILED DESCRIPTION

[0030] Non-limiting examples of various aspects and variations of the embodiments are described herein and illustrated in the accompanying drawings.

[0031] Overview of Robotic Surgery Systems

[0032] Figure 1A is a diagram of an exemplary operating room environment with a robotic surgical system. Figure 1A As shown, the robotic surgical system includes a user console 100 (sometimes referred to herein as a "surgeon bridge" or "bridge"), a control tower 133, and one or more robotic arms 160 located at a robotic platform (e.g., a table, a bed, etc.), wherein surgical instruments (e.g., with end effectors) are attached to the distal ends of the robotic arms 160 to perform surgical procedures. The robotic arms 160 are shown as a table-mounted system, but in other configurations, one or more robotic arms may be mounted on a cart, ceiling or sidewall, or other suitable support surface.

[0033] As a further illustration, Figure 1BAs shown in the exemplary schematic diagram of , the robotic surgical system may include at least one robotic arm 160 and a tool driver 170 generally attached to the distal end of the robotic arm 160. A cannula 180 coupled to the end of the tool driver 170 may receive and guide a surgical instrument 190 (e.g., an end effector, a camera, etc.). In addition, the robotic arm 160 may include a plurality of links that are actuated to position and orient the tool driver 170, which actuates the surgical instrument 190.

[0034] Usually, if Figure 1A As shown, user console 100 can be used to interface with robotic surgical system 150. A user (such as a surgeon or other operator) can use user console 100 to remotely manipulate robotic arm 160 and / or surgical instruments (e.g., in teleoperation). User console 100 can be located in the same operating room as robotic system 150, such as in FIG. Figure 1A In other embodiments, the user console 100 may be located in an adjacent or nearby room, or remotely operated from a remote location in a different building, city, or country. In one example, the user console 100 may include a seat 110, a foot control (pedal) 120, one or more handheld user input devices 122, and at least one user display 130 configured to display, for example, a view of a surgical site within a patient's body (e.g., captured with an endoscopic camera), and / or other surgical or medical information.

[0035] exist Figure 1C In the exemplary user console shown, a user located in the seat 110 and viewing the user display 130 can manipulate the foot control 120 and / or the handheld user input device 122 to remotely control the robotic arm 160 and / or the surgical instrument mounted to the distal end of the arm. The foot control 120 and / or the handheld user input device 122 can be used in addition or alternatively to control other aspects of the user console 100 or the robotic system 150. For example, in a variation where the user typically controls (at any given time) a designated "left hand" robotic arm / instrument and a designated "right hand" robotic arm / instrument, the foot control 120 can enable the user to specify which robotic arms / instruments include the "left hand" and "right hand" robotic arms / instruments from a larger set of available robotic arms / instruments (e.g., via switching or rotating when selecting among the available robotic arms / instruments). Other examples include adjusting or configuring the seat 110, the foot control 120, the user input device 122, and / or the user display 130.

[0036] In some variations, a user may operate the surgical robotic system in an "on the bed" (OTB) mode, wherein the user is positioned to one side of the patient and simultaneously manipulates both the robotically driven instrument / end effector (e.g., a handheld user input device 122 held in one hand) and the manual laparoscopic tools attached to the patient. For example, the user's left hand may manipulate the handheld user input device 122 to control the robotic surgical components, while the user's right hand may manipulate the manual laparoscopic tools. Thus, in these variations, the user may perform both robotic-assisted MIS and manual laparoscopic procedures on the patient.

[0037] During an exemplary procedure or surgery, the patient is prepared and draped in a sterile manner, and anesthesia is achieved. The initial approach to the surgical site can be performed manually using the robotic system 150 in a stowed configuration or a retracted configuration to facilitate access to the surgical site. Once access is completed, the initial positioning and / or preparation of the robotic system can be performed. During the surgical procedure, the surgeon or other user in the user console 100 can manipulate various end effectors and / or imaging systems to perform the procedure using the foot control 120, the user input device 122, and / or other suitable controls. Manual assistance can be provided at the procedure table by other personnel, who can perform tasks including but not limited to: retracting tissue, or performing manual repositioning or tool replacement involving one or more robotic arms 160. Other personnel may be present to assist the user at the user console 100. Medical and surgical related information to assist other medical personnel (e.g., nurses) can be provided on additional displays, such as a display 134 on a control tower 133 (e.g., a control system for a robotic surgical system) and / or a display 132 located near the patient's bedside. For example, as described in further detail herein, some or all of the information displayed to the user in the user console 100 may also be displayed on at least one additional display of other personnel and / or provide additional pathways for communication between personnel. Upon completion of a procedure or surgery, the robotic system 150 and / or the user console 100 may be configured or arranged to be in a state that facilitates one or more post-operative procedures, including, but not limited to, robotic system cleaning and / or disinfection, and / or such as medical record input or printout via the user console 100, whether electronic or hard copy.

[0038] In some variations, communication between the robotic system 150, the user console 100, and any other display may be through a control tower 133, which may translate user commands from the user console 100 into robotic control commands and transmit these commands to the robotic system 150. The control tower 133 may also transmit status and feedback from the robot 150 back to the user console 100 (and / or other displays). The connections between the surgical robot 150, the user console 100, other displays, and the control tower 133 may be via wired and / or wireless connections, and may be proprietary and / or implemented using any of a variety of data communication protocols. Any wired connection may be built into the floor and / or walls or ceiling of the operating room. The surgical robotic system may provide video output to one or more displays, including displays within the operating room and remote displays accessible via the Internet or other networks. The video output or feed may be encrypted to ensure privacy, and all or one or more portions of the video output may be saved to a server, an electronic health care record system, or other suitable storage medium.

[0039] In some variations, additional user consoles 100 may be provided, for example to control additional surgical instruments and / or to control one or more surgical instruments on the primary user console. This would allow, for example, a surgeon to take over or demonstrate a technique to medical students and physicians-in-training during a surgical procedure, or to assist during a complex surgery that requires multiple surgeons to act simultaneously or in a coordinated manner.

[0040] In some variations, such as Figure 2 As shown in the schematic diagram of , one or more third-party devices 240 can be configured to communicate with the user console 210 and / or other suitable parts of the robotic surgical system. For example, as described elsewhere herein, a surgeon or other user can sit in the user console 210, which can communicate with the control tower 230 and / or the robotic instruments in the robotic system 220. Medical data (e.g., endoscopic images, patient vital signs, tool status, etc.) can be displayed at the user console 210, the control tower 230, and / or other displays. At least a subset of the surgical and other medical related information can also be displayed on the third-party device 240, such as a remote computer display, which is viewed by a surgical collaborator in the same room or outside the room. Other communications, such as teleconferences using audio and / or visual communications, can be further provided to and from the third-party device. The surgical collaborator can be, for example, a supervisor or trainer, a medical colleague (e.g., a radiologist) or other third party, who can, for example, view and communicate via the third-party device 240 to assist in the surgical procedure.

[0041] Figure 3is a schematic illustration of an exemplary variation of a system 300 including a robotic surgical system and its interactions with other devices and other parties. Figure 3 Specific architectures of various connections and communication systems are depicted in the drawings, but it should be understood that in other variations, other suitable architectures may be used, and Figure 3 The arrangement shown in is for illustrative purposes. System 300 may include a surgical robotic platform 302 that facilitates integration of medical data from discrete medical data resources generated from various parties. Data from discrete medical data resources may, for example, be used to form temporally coordinated medical data. As further described herein, a multi-panel display of temporally coordinated medical data may be configured and presented.

[0042] Platform 302 may be, for example, a machine having one or more processors 310 connected to one or more input / output devices 312 via bus 314. The at least one processor may include, for example, a central processing unit, a graphics processing unit, an application specific integrated circuit, a field programmable logic device, or a combination thereof.

[0043] The surgical robot platform 302 may include one or more input ports to receive medical data from discrete medical data resources. For example, the surgical robot port 329 may receive surgical robot data from the surgical robot 330. For example, such data may include position data or other suitable state information. The imaging port 331 may receive imaging data from an imaging device 332 (such as an endoscope) that is configured to capture images (e.g., still images, video images) of the surgical site. The endoscope may be inserted, for example, through a natural orifice or through a hole in the surgical patient. As another example, one or more medical instrument ports 333 may receive patient vital sign information from a medical instrument 334 (e.g., a pulse oximeter, an electrocardiogram device, an ultrasound device, etc.). In addition, as another example, one or more user control data ports 335 may receive user interaction data from one or more control devices that receive user input from a user to control the system. For example, one or more handheld user input devices, one or more foot pedals, and / or other suitable devices (e.g., eye tracking, head tracking sensors) may receive user input.

[0044] The surgical robotic platform 302 may also include one or more output ports 337 configured to connect to one or more displays 338. For example, the display 338 may include an open display (e.g., a monitor screen) in a user console, an immersive display or head-mounted device with a display, a supplemental display such as on a control tower display (e.g., a team display), a bedside display (e.g., a nurse display), an overhead "stadium" style screen, etc. For example, the graphical user interface disclosed herein may be presented on one or more displays 338. One or more displays 338 may present a three-dimensional image. In some variations, one or more displays 338 may include a touch screen. One or more displays 138 may be a single display with multiple panels, each of which presents different content. Alternatively, one or more displays 138 may include a collection of separate displays, each of which presents at least one panel.

[0045] In some variations, a network interface 316 may also be connected to the bus 314. For example, the network interface 316 may provide connectivity to a network 317, which may be any combination of one or more wired and / or wireless networks. For example, the network 317 may facilitate communication between the surgical robotic platform 302 and other data sources or other devices. For example, one or more third-party data sources 340 may also be connected to the network 317. The third-party source 340 may include a third-party device (e.g., another computer operated by a third party such as another doctor or medical specialist), a repository of video surgical procedure data (e.g., the data may be related to a procedure performed by a surgeon), or other suitable additional information sources related to a surgical procedure. For example, the third-party device data may be transmitted to a panel displayed to the surgeon before, during, or after the procedure.

[0046] As another example, one or more application databases 342 can be connected to the network 317 (or alternatively, stored locally in the memory 320 within the surgical robot platform 302). The application database 342 can include software applications (e.g., as described in further detail below) that may be of interest to the surgeon during a procedure. For example, the software application can provide access to stored patient medical records, provide a checklist of surgical tasks for a surgical procedure, perform machine vision techniques for assisting the procedure, perform machine learning tasks to improve surgical tasks, and the like. Any suitable number of applications can be called. During the procedure, information associated with the application can be displayed in a multi-panel display or other suitable display. Additionally or alternatively, the information provided by one or more applications can be provided by a separate resource (e.g., a machine learning resource) that is otherwise in appropriate communication with the surgical robot platform 302.

[0047] In some variations, one or more of the software applications may run as separate processes that use an application program interface (API) to draw objects and / or images on a display. APIs of varying complexity may be used. For example, a simple API may include several templates with fixed widget sizes and positions that may be used by a GUI module to customize text and / or images. As another example, a more complex API may allow a software application to create, place, and delete different widgets, such as labels, lists, buttons, and images.

[0048] Additionally or alternatively, one or more software applications may render themselves for display. This may, for example, allow for advanced customization and complex behavior of the application. For example, this approach may be implemented by allowing the application to pass frames rendered by a graphical user interface (GUI) module 324, which may be a computer-readable program code executed by the processor 310. Alternatively, the image buffer may be used as a repository to which the application renders itself.

[0049] In some variations, one or more software applications may run and render themselves independently of the GUI module 324. However, the GUI module may still launch such applications, instruct the application or the operating system where to position the application on the display, and so on.

[0050] As another approach, in some variations, one or more applications may run completely separate from the GUI rendered by the GUI module. For example, such applications may have a physical video connection and a data connection to the system (e.g., through appropriate input / output devices, a network, etc.). The data connection may be used to configure the application's video feed to the appropriate pixel size (e.g., full screen, half screen, etc.).

[0051] like Figure 3 As shown, in some variations, a memory 320 may also be connected to the bus 314. The memory 320 may be configured to store data processed according to embodiments of the methods and systems described herein.

[0052] In some variations, the memory 320 may be configured to store other kinds of data and / or software modules for execution. For example, a user console may include a memory 320 that stores a GUI module 324 having executable instructions to implement the operations disclosed herein. The GUI module may, for example, combine and aggregate information from various software applications and / or other medical data resources for display. In some exemplary variations, one or more software applications may be incorporated into the base code of the GUI module so that the module draws graphics and displays text in appropriate locations on the display. For example, the module may obtain images from a database, or images may be pushed to an interface from an instrument (e.g., an endoscopic camera) in an operating room via a wired or wireless interface.

[0053] In some variations, medical data may be collected from discrete medical data resources (e.g., surgical robot 330, endoscope 332, medical instrument 334, control device 336, third-party data source 340, application database 342, etc.). In addition, at least some of the medical data may be temporally coordinated so that, when necessary, time-sensitive information from different medical data resources is aligned on a common timeline. For example, surgical robot position data may be temporally coordinated with endoscope data, which is coordinated with operator interaction data from a control device. Similarly, network resources such as information provided by one or more software applications may be presented at appropriate points in time with other temporally coordinated data. A multi-panel display and / or other suitable display may be configured to convey medical information (e.g., including temporally coordinated medical data) as part of a graphical user interface (GUI).

[0054] Various exemplary aspects of a GUI for a robotic surgical system are described herein. In some variations, the GUI may be displayed in a multi-panel display at a user console that controls the robotic surgical system. Additionally or alternatively, the GUI may be displayed at one or more additional displays, such as at a control tower of the robotic surgical system, at a patient's bedside, etc. In general, the GUI may provide more efficient communication of information to a user in a user console and / or other personnel, as well as more efficient communication and collaboration between different parties involved in a surgical procedure, as further described below.

[0055] Graphical User Interface (GUI) Interaction

[0056] In one embodiment, the GUI is displayed on a display 130 in a user console 100 that is used to control the robotic surgical system 150 (e.g., by a surgeon), and at least some of the interactive graphical objects displayed on the display 130 can be controlled, selected, or otherwise interacted with via one or more user controls that are also used to control aspects of the surgical system (e.g., surgical instruments). For example, a user can selectively control aspects of the robotic surgical system 150 and selectively interact with the GUI using one or more handheld user input devices 122 and / or one or more foot pedals 120. By enabling the same user controls to control both the robotic surgical system 150 and the GUI, the user can advantageously avoid having to switch between two different types of user controls. Enabling the user to use the same input device to control the robotic system 150 and the GUI streamlines surgical procedures and improves efficiency, as well as helping the user maintain sterility throughout the surgical procedure.

[0057] like Figure 4A and Figure 4B As generally shown in FIG. 4 , an exemplary variation of a handheld user input device 122 for controlling a robotic system may include a member 410, a housing 420 disposed at least partially around the member 410 and configured to be held in a user's hand, and a tracking sensor system 440 configured to detect at least a position and / or orientation of at least a portion of the device. The housing 420 may be flexible (e.g., made of silicone). In some instances, the detected position and / or orientation of the device may be associated with a control of the robotic system. For example, the user input device 122 may control at least a portion of a robotic arm, an end effector or tool (e.g., a grasper or clamp) coupled to a distal end of the robotic arm, a GUI, or other suitable aspects or features of the robotic surgical system 150. Additionally, in some instances, the detected position and / or orientation of the device 122 may be associated with a control of the GUI. Furthermore, in some variations, the user input device 122 may include one or more sensors for detecting other manipulations of the user input device 122, such as squeezing the housing 420 (e.g., via one or more pressure sensors, one or more capacitive sensors, etc.).

[0058] In general, a user interface for controlling a robotic surgical system may include at least one handheld user input device 122, or may include at least two handheld user input devices 122 (e.g., a first user input device to be held by a user's left hand, and a second user input device to be held by a user's right hand), or any suitable number. Each user input device 122 may be configured to control one or more different aspects or features of the robotic system. For example, a user input device held in a user's left hand may be configured to control an end effector represented on the left side of a camera view provided to the user, while a user input device held in a user's right hand may be configured to control an end effector represented on the right side of the camera view.

[0059] In some variations, the handheld user input device 122 may be a groundless user input device configured to be held in the hand and manipulated in free space. For example, the user input device 122 may be configured to be held between the user's fingers and freely (e.g., translated, rotated, tilted, etc.) moved by the user when the user moves his or her arm, hand, and / or finger. Additionally or alternatively, the handheld user input device 122 may be a body-grounded user input device, because the user input device 122 may be directly or via any suitable mechanism (such as a glove, hand strap, sleeve, etc.) coupled to a portion of the user (e.g., the user's finger, hand, and / or arm). Such a body-grounded user input device may still enable the user to manipulate the user input device in free space. Therefore, in variations where the user input device 122 is groundless or body-grounded (as opposed to being permanently mounted or grounded to a fixed console, etc.), the user input device 122 may be ergonomic and provide ambidextrous control, such as by enabling the user to control the user input device with natural body movement that is not hindered by the fixed nature of the grounding system.

[0060] The handheld user input device 122 may include a wired connection, for example, to provide power to the user input device 122, carry sensor signals (e.g., from a tracking sensor assembly and / or other sensors such as capacitive sensors, optical sensors, etc.). Alternatively, the user input device may be wireless, such as Figure 4A As shown, and commands and other signals are communicated via, for example, radio frequency signals (e.g., WiFi or short range such as a 400 mm to 500 mm range) or other suitable wireless communication protocols (such as Bluetooth). Other wireless connections may be facilitated using optical reader sensors and / or cameras (which are configured to detect optical indicia on the user input device 122), infrared sensors, ultrasonic sensors, or other suitable sensors.

[0061] The handheld user input device may include a clutch mechanism for switching between controlling a robotic arm or end effector and controlling a graphical user interface, etc., and / or switching between other control modes. One or more of the various user inputs described in further detail below may be used as a clutch in any suitable combination. For example, a gesture touch area of ​​a touch device, squeezing a housing, flicking or rotating a user input device, etc. may act to engage a clutch. As another example, a combination of squeezing and holding a user input device and rotating a user input device may be used as a clutch. However, any suitable gesture combination may be used as a clutch. Additionally or alternatively, user inputs to other user input devices (e.g., a foot pedal assembly) may be used as a clutch alone or in combination with user inputs to a handheld user input device.

[0062] In some variations, engagement and disengagement of the clutch mechanism may enable transitioning between using the handheld user input device as a control for the robotic system and using the handheld user input device as a control for a GUI (e.g., operating a cursor displayed on a screen). When the clutch mechanism is engaged so that the user input device is being used to control the GUI, the position or pose of the robotic arm may be substantially locked in place to "pause" operation of the robotic system so that subsequent movement of the user input device while the clutch was engaged will not inadvertently cause movement of the robotic arm.

[0063] GUI Layout

[0064] Any suitable layout of the GUI may be used. For example, Figure 5 As shown, in one embodiment, the GUI includes a first area 500 and a second area 510. Each of these areas 500, 510 can have one or more display areas. Figure 5 In the example shown, the first area 500 is positioned in the middle of the display, and the second area 510 has two display areas displayed on the left and right sides of the first area 500. In this embodiment, the first area 500 displays an endoscopic view (e.g., a video image or a still image) of a surgical site within a patient's body captured by an endoscopic camera of the robotic surgical system 150.

[0065] The GUI of the robotic surgical system 150 may be used to provide information and guidance to the user. To make the GUI in this embodiment intentionally simple and non-distracting, the processor 310 of the robotic surgical system 150 determines which area 500, 510 to use to display data based on the type of data to be displayed. Figure 6 600 is a flow chart of a method that may be implemented by processor 310 to perform this operation. Figure 6As shown, the processor 310 (e.g., using code in the GUI module 324) displays a GUI on the display device 130, wherein the GUI has a first area 500 and a second area 510 (action 610). Next, the processor 310 receives data to be displayed on the display 130 (action 620). The data may originate from one or more applications being executed by the processor 310 or from an external location. The processor 310 then determines whether the data is user feedback information or a guidance message for engaging the robotic arm 160 (action 630). If the data is a guidance message for engaging the robotic arm 160, the processor 130 displays the data in the first area 500 by superimposing the guidance message on top of the endoscopic view of the surgical site in the first area 500 (action 640). However, if the data is user feedback information, the processor 310 displays the data in the second area 510 (action 650). The following paragraphs and figures will now be discussed to illustrate this embodiment.

[0066] return Figure 5 , the first area 500 displays an endoscopic view of the surgical site within the patient's body taken by the endoscopic camera of the robotic surgical system 150. This view typically shows the patient tissue and the end effector of the robotic arm. The second area 510 (here, the left panel and the right panel) can be used to display user feedback information, such as but not limited to errors, warnings, alarms, and notifications. For example, in this illustration, the upper portion of the left side of the second area 510 shows the initials of the user logged into the robotic surgical system 150, the type of mirror used (here, a 30-degree mirror pointing upward), and a horizontal line indicator. The other corners of the second area 510 display messages indicating the tools (here, graspers, bipolar window openers, and suture shears) located on the three robotic arms in the robotic arm.

[0067] As described above, in this embodiment, the robotic surgical system 150 determines whether to display data in the first area 500 or the second area 510 depending on the type of data to be displayed. In one embodiment, the message superimposed on top of the image of the surgical site in the first area 500 is only a guidance message that provides the surgeon with user instructions to engage the robotic arm 160. As used herein, the robotic arm "engages" when the movement of at least one of the plurality of user input devices is converted into movement of the end effector of the robotic arm in real time. Since the surgeon is usually focused on the image of the surgical site in the first area 500, superimposing the guidance information on the image of the surgical site in the first area 500 is likely to attract the surgeon's attention. Displaying the user feedback information in the second area 510 (and not above the image of the surgical site in the first area 500) avoids distracting the surgeon during surgery (minimizing cognitive load), thereby allowing the surgeon to focus on the surgical procedure rather than the GUI. If and when the surgeon wants to view the user feedback information, he can do so by glancing at the second area 510. With this layout, the surgeon can focus on the procedure and the patient because the display is clean and the noise is reduced.

[0068] exist Figure 5 In the example shown, a guidance message "Pick up the controller to get started" is superimposed on the image of the surgical site in the first area 500, thereby instructing the surgeon to pick up the handheld left and right user input devices 122 (the handheld devices 122 are used to illustrate this example, but it should be understood that other types of user input devices may be used). This may occur, for example, when the surgeon first sits down at the user console 100 or after the surgeon sets down or drops the user input device 122. In one embodiment, one of the foot controls acts as a "clutch pedal" that can switch the functionality of the user input device 122 between controlling the robotic arm and controlling the graphical user interface.

[0069] After the surgeon picks up the handheld left and right user input devices 122, the message superimposed on the image of the surgical site in the first area 500 changes to "Matching Grips", as shown in FIG. Figure 7 "Match Grip" instructs the surgeon to open and close the controller's grip to match the instrument's current state (e.g., the grip angle of the end effector). After the surgeon "matches grip," he fully engages the robotic arm and can perform surgery.

[0070] Various aids are available to help the surgeon match the grip. Figure 7As shown, icons for the left and right user input devices 122 are also superimposed on the image of the surgical site. As the surgeon opens and closes the grip of each controller, the color of the icons may change to provide feedback to the surgeon if he needs to provide a tighter or looser grip to align his grip with the state of the instrument. Other ways of providing feedback to the surgeon on "matching grips" may be provided. For example, FIG. 8A to FIG. 8F is an illustration of various graphical tools that can be used to instruct the user in matching grips. Fig. 8A In , the grip to be matched is represented by the inner circle, and the outer circle represents the current grip of the user input device. As the user grips the user input device more tightly, the outer circle will move closer to the inner circle. Figure 8B In , the grip to match is represented by the middle bar, and the upper bar represents the current grip of the user input device. As the user grips the user input device more tightly, the upper bar will move closer to the middle bar. Figure 8C In , the grip to be matched is represented by the inner shape, and the outer shape represents the current grip of the user input device. As the user grips the user input device more tightly, the outer shape will be formed closer to the inner shape. Fig.8D In FIG. 1 , a representation of the physical shape of a user input device is shown, wherein arrows represent pressing or releasing a user input device handle. Fig. 8E In , the grip of the user input device represents the actual state of the user input device, and the line path indicates the amount of squeezing of the UID to match the grip. Fig.8F , the left point represents the state of the user input device, and the right point represents the state of the end effector. The path between the points indicates the steps that the user input device needs to go through.

[0071] Once the surgeon matches the grip of the left and right user input devices 122 with the current state of the instrument, the surgeon has successfully engaged the robotic arm and can manipulate the left and right user input devices 122 to perform surgery on the patient. During surgery, the graphical user interface can present feedback information to the surgeon in the second area 510 in a timely, perceptible, but non-invasive manner. For example, Fig. 9 As shown, the lower right portion of the second area 510 shows the "lighted" "cut" and "coagulation" bars, indicating that the tool is active. These bars are color-coded to the pedal 120 of the robotic surgical system 150, and the length of the bar indicates the amount of energy applied to the tool by the pedal 120. Fig.10 The long "cut" bar in the diagram is a hover indicator. In this embodiment, the pedal 120 has a hover sensor when the surgeon's foot is over the pedal 120 but not pressing the pedal. The hover indicator can be pulsating or animated to get the surgeon's attention. This is a safety feature because it may be desirable to get the surgeon's attention before actually pressing the pedal 120. Fig.11As shown, when the pedal 120 is pushed and cutting energy is applied, the entire tool package is filled with yellow, thereby showing the type and level of energy.

[0072] Fig.12 and Fig.13 is an example of when the user feedback information displayed in the second area 510 is related to the position of the user input device 122. In one embodiment, the robotic surgical system 150 uses an electromagnetic tracker to track the position of the user input device 122. The space within which the user input device 122 can operate is defined by a boundary (e.g., a cube), and the system 150 defines a buffer zone around the edge of the boundary. When the surgeon moves the user input device 122 into the buffer zone, the robotic surgical system 150 displays a message in the second area 510 to inform the surgeon that he is about to exceed the working space limit. This is shown in Fig.12 As shown in the figure, the right side of the second area 510 shows a user feedback message, which notifies the doctor that he is approaching the working space limit of the right user input device. The message is displayed on the right side of the second area 510 because the message is related to the right user input device. If the message is related to the left user input device, the message can be displayed on the left side of the second area 510.

[0073] The circle displayed around the message provides an indication of how close the user input device 122 is to exceeding the limit. As the user approaches the workspace limit, larger circles may be added. Many alternative graphics may be used. For example, a line / bar may be used instead of an enlarged circle. In addition, the origin of the circle may be located near an edge of the second area 510 that corresponds to the location of the limit that the user is approaching (e.g., the bottom of the second area 510 if the user is approaching the lower edge workspace limit). Various graphics and / or animations may be used.

[0074] like Fig.13 As shown, if the limit is exceeded, the robotic surgical system 150 displays a message (now completely blue) in the second area 510, informing him of this and how to reposition the user input device to reengage the robotic arm. After the surgeon moves the user input device back into the workspace, the surgeon will tap the clutch pedal and repeat the matching grip process described above.

[0075] Fig.14 5 shows a message displayed in the second area 510 when the surgeon over-rotates the end effector. Fig.14 As shown, the message instructs the surgeon to rotate the user input device in the opposite direction (again, the message is located on the right side of the second area 510 because the message is associated with the right user input device). Fig.14The message in one embodiment is shown to be high contrast (here white text on a black background) to draw the surgeon's attention while still not distracting from the surgical view in the first area 500. It may also be desirable to provide white space around the message to avoid cluttering the message area and to provide room for translations in other languages, where the translations may be longer than the English message.

[0076] In some embodiments, the robotic surgical system 150 has an eye sensor on the user console / surgeon bridge 100 to sense whether the surgeon is looking at the display screen 130. If the surgeon is not looking at the display screen 130, the surgeon may not be focusing on the surgical site, and movement of the user input device may cause movement of the robotic arm that may injure the patient. Therefore, when the robotic surgical system 150 detects that eye contact has been lost, the robotic surgical system 150 suspends operation of the instrument and displays a message to that effect in the second area 510 (here, displayed on both the left and right sides to indicate that eye contact has been lost between the eyes).

[0077] After reestablishing eye contact, the surgeon needs to reengage with the robotic surgical system 150. In one embodiment, the required action depends on whether the loss of eye contact was temporary or not temporary (e.g., less than thirty seconds, or thirty seconds or longer). If the loss of eye contact was temporary, the robotic surgical system 150 displays a message over the image of the surgical site in the first area 500, instructing the surgeon to match the grips ( Fig.16 ). Once the grips have been mated, the surgeon has reengaged with the robotic surgical system 150 and can continue with the surgery. However, if eye contact is lost for more than five seconds in this example, the robotic surgical system 150 displays a message over the image of the surgical site in the first area 500 instructing the surgeon to tap the clutch pedal ( Fig.17 ), after which the robotic surgical system 150 will display a message above the image of the surgical site in the first area 500, instructing the surgeon to match the grip.

[0078] Some messages displayed in the second area 510 may be more important than other messages, and the robotic surgical system 150 may use different ways to indicate the importance of the message. Fig.18 As shown, in one embodiment, when the surgeon drops the user input device, the robotic surgical system 150 displays a message with a red background in the second display area 510, notifying the surgeon that the user input device has been dropped and that the user input device needs to be picked up to continue. Once the surgeon picks up the dropped user input device, the robotic surgical system 150 displays a message over the image of the surgical site in the first area 500, instructing the surgeon to tap the clutch pedal ( Fig.19 ), after which the robotic surgical system 150 will display a message above the image of the surgical site in the first area 500, instructing the surgeon to match the grip.

[0079] In one embodiment, the second area 510 can also be used for purposes other than displaying messages. Figure 20 to Figure 23 As shown, the second area 510 may be used to display one or more applications running on the robotic surgical system 150 . Fig. 20 Several applications are shown in the second area 510. The layout and location of the applications may be selected in any suitable manner and may be reconfigured. Fig. 20 In the example shown, the upper left corner of the second area 510 includes an image viewing application that communicates with a medical record database or other suitable repository so that the image viewer application can receive medical images related to the surgical procedure. For example, the image viewer application can receive and display preoperative images (e.g., X-ray, CT, MRI, ultrasound, etc.) of the patient. Such display of preoperative images can allow surgeons and / or other users to easily view preoperative images before, during, and / or after the surgical procedure, and can help the surgical team make better and more informed decisions related to the surgical procedure. The lower left corner of the second area 510 includes a procedure template application that can be used to generate a list of items related to the execution of the surgical procedure. For example, the procedure template application can display a checklist of surgical tasks that are part of the surgical procedure, a list of instruments or tools required for the surgical procedure, a list of operating room setup tasks, a schematic diagram of the port position and arm / surgical instrument settings, etc. The upper right corner of the second area 510 includes a timer application that can, for example, track the duration of the surgical procedure and / or the duration of a segment of the surgical procedure (e.g., a separate surgical task and other tasks performed). Other types of applications and implementations may be used, such as those described in U.S. patent application 15 / 842,485, “Multi-Panel Graphical User Interface for a Robotic Surgical System,” filed on December 14, 2017, which is hereby incorporated by reference.

[0080] The user input device 122 can be used to switch between various displays and applications. For example, in one embodiment, Fig. 20 The display in is the result of the surgeon depressing the clutch pedal and squeezing the two user input devices 122, Fig.21 The display in is the result of the surgeon again squeezing the two user input devices 122, and Fig. 22 and Fig.23The display in is the result of the user input device 122 hovering over various parts of the display. In one embodiment, the border of the display area over which the cursor is hovering is highlighted to indicate which display area is active.

[0081] The panels of the first display area 500 and / or the second display area 510 may be changed dynamically. Fig.24 and Fig.25 As shown, the CT image is shown Fig.23 The display panel in the second area 510 in the display area 500 can be moved to the larger first area 510, and the image of the surgical site can be moved to the panel in the second area 510. Each thumbnail can be associated with a single image, or can be associated with a series of images (e.g., multiple images taken on the same date). Using the large CT image shown, the surgeon can use the user input device to "quickly browse" through the different images in the CT scan group. If the surgeon finds an image of interest, the surgeon can "pin" that particular image to the second display area 510 (e.g., by squeezing two user input devices). When this occurs, a message is displayed above the CT image in the first display area 500 confirming that the pinning operation has occurred ( Fig.26 ), and the CT image is later fixed in the second display area 510 ( Fig. 27 ).

[0082] For the purpose of explanation, the foregoing description uses specific nomenclature to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that specific details are not required to practice the present invention. For the purposes of illustration and description, the foregoing description of specific embodiments of the present invention has been provided. They are not intended to be exhaustive or to limit the present invention to the specific forms disclosed; various modifications and changes may be made to the present disclosure in light of the above teachings. The embodiments are selected and described in order to best explain the principles of the present invention and its practical application, so that other persons skilled in the art can best utilize the present invention and various embodiments with various modifications suitable for the specific purposes contemplated. The following claims and their equivalents are intended to define the scope of the present invention.

Claims

1. A method for a robotic surgical system, the method comprising: In a robotic surgical system including a plurality of robotic arms, a plurality of user input devices, and a display device, the following operations are performed: displaying, in a first area of ​​a graphical user interface on the display device, an endoscopic view of a surgical site within a patient's body captured by an endoscopic camera of the robotic surgical system; determining whether data to be displayed in the graphical user interface is (i) a guidance message providing user instructions for interacting with at least one of the plurality of user input devices to engage the plurality of robotic arms, or (ii) user feedback information regarding control of the plurality of robotic arms in response to the plurality of user input devices; in response to determining that the data to be displayed in the graphical user interface is the guidance message, superimposing the guidance message on top of the endoscopic view of the surgical site in the first area; as well as In response to determining that the data to be displayed in the graphical user interface is user feedback information, the user feedback information is displayed in the second area.

2. The method of claim 1 , wherein one of the plurality of user input devices comprises a handheld user input device configured to remotely control one of the plurality of robotic arms, and wherein the guidance message instructs a user to match a grip of the handheld user input device with a grip of the one of the plurality of robotic arms. 3 . The method of claim 2 , wherein the guidance message comprises a graphic that changes as the grip of the one of the plurality of user input devices changes.

4. The method of claim 1, wherein one of the plurality of user input devices comprises a foot control, and wherein the guidance message instructs the user to depress the foot control. 5 . The method of claim 1 , wherein the user feedback information displayed in the second area comprises one or more of the following: an error, a warning, an alert, or a notification.

6. The method of claim 1 , wherein the user feedback information displayed in the second area comprises one or more of: information about an energy level applied by engaging one of the plurality of user input devices, a warning that one of the plurality of user input devices is approaching a workspace limit, a message that one of the plurality of user input devices has exceeded a workspace limit, a message that a rotation limit has been reached, a message that eye contact has been lost, and a message that one of the plurality of user input devices has been dropped. The method according to claim 1 , further comprising displaying an application in the second area.

8. The method of claim 7, further comprising exchanging the display of the endoscopic view of the surgical site and the display of the application between the first area and the second area.

9. A robotic surgical system, comprising: Robotic surgical instruments; one or more handheld user input devices configured for controlling a robotic arm coupled to the robotic surgical instrument; Display devices; as well as A processor configured to: displaying in a first area of ​​the display device a view of a surgical site within a patient captured by an image capture device of the robotic surgical system; as well as displaying in a second area of ​​the display device user feedback information regarding control of the robotic arm in response to the handheld user input device; wherein the processor is further configured to overlay guidance messages on top of the view of the surgical site in the first area, wherein the guidance messages provide user instructions for interacting with the one or more handheld user input devices to engage the robotic surgical instrument, and wherein only guidance messages are overlaid on top of the view of the surgical site.

10. The robotic surgical system of claim 9, wherein the guidance message instructs a user to match a grip of the one or more handheld user input devices with a grip of the robotic surgical instrument.

11. The robotic surgical system of claim 10, wherein the guidance message comprises a graphic that changes as the grip of the one or more handheld user input devices changes.

12. The robotic surgical system of claim 9, wherein the guidance message instructs the user to depress a foot control.

13. The robotic surgical system of claim 9, wherein the user feedback information displayed in the second area comprises one or more of: an error, a warning, an alarm, or a notification.

14. The robotic surgical system of claim 9, wherein the user feedback information displayed in the second area includes one or more of: information about an energy level applied by engaging one of the plurality of user input devices, a warning that one of the plurality of user input devices is approaching a workspace limit, a message that one of the plurality of user input devices has exceeded a workspace limit, a message that a rotation limit has been reached, a message that eye contact has been lost, and a message that one of the plurality of user input devices has been dropped.

15. The robotic surgical system of claim 9, wherein the processor is further configured to display an application in the second area.

16. The robotic surgical system of claim 15, wherein the processor is further configured to exchange the display of the endoscopic view of the surgical site and the display of the application between the first area and the second area.

17. A robotic surgical system, comprising: a display device for displaying a graphical user interface, wherein the graphical user interface includes a first area and a second area, the first area displaying a view of a surgical site with a surgical instrument within a patient's body taken by an endoscopic camera of the robotic surgical system; as well as a processor configured to cause display in the second area of ​​user feedback information regarding control of a robotic arm or the surgical instrument in response to a user input device and to overlay a guidance message on top of the view of the surgical site in the first area, wherein the guidance message provides user instructions for interacting with the user input device to engage a robotic arm of the robotic surgical system connected to the surgical instrument.

18. The robotic surgical system of claim 17, wherein the guidance message instructs a user to match a grip of the user input device with a grip of the robotic arm.

19. The robotic surgical system of claim 18, wherein the guidance message comprises a graphic that changes as the grip of the user input device changes.

20. The robotic surgical system of claim 17, wherein the user feedback information displayed in the second area comprises one or more of: an error, a warning, an alarm, or a notification.

21. The robotic surgical system of claim 17, wherein the guidance message instructs the user to depress a foot control.

22. The robotic surgical system of claim 17, wherein the user feedback information displayed in the second area includes one or more of: information regarding an energy level applied by engaging the user input device, a warning that the user input device is approaching a workspace limit, a message that the user input device has exceeded a workspace limit, a message that a rotation limit has been reached, a message that eye contact has been lost, and a message that the user input device has been dropped.

23. The robotic surgical system of claim 17, wherein the processor is further configured to display an application in the second area.

24. The robotic surgical system of claim 23, wherein the processor is further configured to exchange the display of the view of the surgical site and the display of the application between the first area and the second area.

Images