Robotic surgical system and method for providing a stadium view with arm set-up guidance

Abstract

A robotic surgical system and method for providing a stadium view with arm setup guidance is provided. In one embodiment, a robotic surgical system includes a plurality of robotic arms, a display device, and a processor. The processor is configured to render, on the display device, a graphical representation of the plurality of robotic arms in their current positions, and user guidance information about how to move the plurality of robotic arms to different positions. Other embodiments are provided.

Description

TECHNICAL FIELD

[0001] The following embodiments relate generally to the field of robotic surgery, and more particularly to a graphical user interface (GUI) for a robotic surgical system that provides a stadium view with arm setup guidance. BACKGROUND

[0002] Minimally invasive surgical procedures (MIS), such as laparoscopic surgery, involve techniques intended to reduce tissue damage during a surgical procedure. For example, laparoscopic surgery often involves forming a plurality of small incisions in a patient (e.g., in the abdomen) and introducing one or more surgical instruments (e.g., an end effector, at least one camera, etc.) into the patient through the incisions. The surgical procedure can then be performed using the introduced surgical instruments, with visualization assistance provided by the camera(s).

[0003] Generally, MIS provides multiple benefits, such as reduced patient scarring, reduced patient pain, shortened patient recovery periods, and reduced medical costs associated with patient recovery. In some embodiments, MIS can be performed with a robotic system that includes one or more robotic arms for manipulating surgical instruments based on commands from an operator. The robotic arms can support, for example, at their distal ends, various devices such as surgical end effectors, imaging devices, cannulas for providing access to a patient’s body cavities and organs, etc.

[0004] In some embodiments, the operator can provide commands for manipulating the surgical instruments while viewing images provided by the cameras and displayed to the user on a display. BRIEF DESCRIPTION OF DRAWINGS

[0005] Figure 1A An example of an operating room arrangement of an embodiment with a robotic surgical system and a user console is depicted.

[0006] Figure 1B is a schematic illustration of one exemplary variation of a robotic arm manipulator, a tool driver, and a cannula with a surgical tool of an embodiment.

[0007] Figure 1C is a schematic illustration of an exemplary user console of an embodiment.

[0008] Figure 2 is a schematic illustration of an exemplary variation of a user console of a robotic surgical system of an embodiment for communicating with one or more third party devices.

[0009] Figure 3 is a schematic illustration of a surgical robotic platform of an embodiment with a graphical user interface (GUI) module, where the surgical robotic platform is in communication with a plurality of medical data resources.

[0010] Figures 4A-4E is an example variation of a graphical user interface including an embodiment of a stadium view application.

[0011] Figure 5 is an illustration of a graphical user interface including an embodiment of a stadium view application providing user guidance information on how to position a patient.

[0012] Figure 6 is an illustration of a graphical user interface including an embodiment of a stadium view application providing user guidance information on how to deploy a robotic arm.

[0013] Figure 7 is an illustration of a graphical user interface including an embodiment of a stadium view application providing instructions on how to drape a robotic arm.

[0014] Figure 8 is an illustration of a graphical user interface including an embodiment of a stadium view application providing instructions on how to move a robotic arm to a ready arm position.

[0015] Figure 9 is an illustration of a graphical user interface including an embodiment of a stadium view application providing instructions on how to place a port in a patient.

[0016] Figure 10 is an illustration of a graphical user interface including an embodiment of a stadium view application providing instructions on how to deploy a robotic arm to a docking arm position.

[0017] Figure 11 is an illustration of a graphical user interface including an embodiment of a stadium view application providing instructions on how to insert an instrument in a robotic arm.

[0018] Figure 12 is an illustration of a graphical user interface including an embodiment of a stadium view application providing instructions on how to deploy a robotic arm to an assignment arm position.

[0019] Figure 13 is an illustration of a graphical user interface including an embodiment of a stadium view application displaying user guidance information on how to move a plurality of robotic arms to an arm setup position.

[0020] Figure 14 is an illustration of a graphical user interface including an embodiment of a stadium view application displaying a plurality of robotic arms in a deployment position.

[0021] Figure 15is an illustration of a graphical user interface of an embodiment of a stadium view application that includes user guidance information about how to move a plurality of robotic arms to a stowed position.

[0022] Figure 16 is an illustration of a graphical user interface of another embodiment. DETAILED DESCRIPTION

[0023] Non-limiting examples of various aspects and variations of embodiments are described herein and illustrated in the accompanying drawings.

[0024] Robotic Surgical System Overview

[0025] Figure 1A is an illustration of an example operating room environment with a robotic surgical system. Generally, as shown in Figure 1A the robotic surgical system includes a user console 100 (sometimes referred to herein as a “surgeon’s bridge” or “bridge”), a control tower 133, and one or more robotic arms 160 located at a robotic platform (e.g., table, bed, etc.) with surgical instruments (e.g., having end effectors) attached to the distal ends of the robotic arms 160 to perform a surgical procedure. The robotic arms 160 are shown as a table-mounted system, but in other configurations, one or more robotic arms can be mounted on a cart, ceiling or sidewall, or other suitable support surface.

[0026] As a further illustration, as shown in the example schematic of Figure 1B the robotic surgical system can include at least one robotic arm 160 and a tool driver 170 that is generally attached to a distal end of the robotic arm 160. A cannula 180 coupled to the end of the tool driver 170 can receive and guide a surgical instrument 190 (e.g., end effector, camera, etc.). Further, the robotic arm 160 can include a plurality of links that are actuated in order to position and orient the tool driver 170, which actuates the surgical instrument 190.

[0027] Generally, as shown in Figure 1A the user console 100 can be used to interface with the robotic surgical system 150. A user, such as a surgeon or other operator, can use the user console 100 to remotely manipulate the robotic arms 160 and / or surgical instruments (e.g., in teleoperation). The user console 100 can be located in the same operating room as the robotic system 150, as shown in Figure 1AThe user console 100 can be located in the same room as the patient, as shown. In other embodiments, the user console 100 can be located in an adjacent or nearby room, or operated remotely from a remote location in a different building, city, or country. In one example, the user console 100 can include a seat 110, foot pedals 120, one or more handheld user input devices 122, and at least one user display 130 configured to display, for example, a view of a surgical site within a patient (e.g., captured with an endoscope camera), and / or other surgical or medical information.

[0028] In Figure 1C In the example user console shown, a user seated in the seat 110 and viewing the user display 130 can manipulate the foot pedals 120 and / or handheld user input devices 122 to remotely control the robotic arms 160 and / or surgical instruments mounted to the distal ends of the arms. The foot pedals 120 and / or handheld user input devices 122 can additionally or alternatively be used to control other aspects of the user console 100 or robotic system 150. For example, in variations where the user typically controls (at any given time) a designated “left-hand” robotic arm / instrument and a designated “right-hand” robotic arm / instrument, the foot pedals 120 can enable the user to designate which robotic arms / instruments comprise the “left-hand” and “right-hand” robotic arms / instruments from a larger set of available robotic arms / instruments (e.g., via dialing or rotating when selecting among available robotic arms / instruments). Other examples include adjusting or configuring the seat 110, foot pedals 120, user input devices 122, and / or user display 130.

[0029] In some variations, the user can operate the surgical robotic system in an “over the bed” (OTB) mode, where the user is positioned to one side of the patient and simultaneously manipulates a robotically-driven instrument / end effector attached to the patient (e.g., handheld user input device 122 held in one hand) and a manual laparoscope tool. For example, the user’s left hand can manipulate a handheld user input device 122 to control a robotic surgical component, while the user’s right hand can manipulate a manual laparoscope tool. Thus, in these variations, the user can perform both robotically-assisted MIS and manual laparoscopic surgery on the patient.

[0030] During an exemplary procedure or surgery, the patient is prepared and draped in a sterile manner, and anesthesia is achieved. Initial access to the surgical site can be performed manually with the robotic system 150 in a stowed or retracted configuration to facilitate access to the surgical site. Once access is complete, initial positioning and / or preparation of the robotic system can be performed. During the surgical procedure, the surgeon or other user in the user console 100 can manipulate various end effectors and / or imaging systems to perform the procedure with the foot pedals 120, user input devices 122, and / or other suitable controls. Manual assistance during the procedure can be provided at the procedure table by other personnel, who can perform tasks including but not limited to retraction of tissue, or manual repositioning or tool changes involving one or more robotic arms 160. There can be other personnel to assist the user at the user console 100. Medical and surgical related information for assisting other medical personnel (e.g., nurses) can be provided on additional displays, such as the display 134 on the control tower 133 (e.g., a control system for a robotic surgical system) and / or the display 132 located near the patient’s bedside. For example, as described in further detail herein, some or all of the information displayed to the user in the user console 100 can also be displayed on at least one additional display for the other personnel and / or provide an additional pathway for inter-personnel communication. When the procedure or surgery is complete, the robotic system 150 and / or the user console 100 can be configured or set into a state that facilitates one or more post-operative procedures, including but not limited to robotic system cleaning and / or sterilization, and / or medical record input or printout via the user console 100, whether electronic or hard copy.

[0031] In some variations, communication between the robotic system 150, the user console 100, and any other displays can be through the control tower 133, which can translate user commands from the user console 100 into robotic control commands and transmit those commands to the robotic system 150. The control tower 133 can also transmit status and feedback from the robot 150 back to the user console 100 (and / or other displays). Connections between the surgical robot 150, the user console 100, other displays, and the control tower 133 can be via wired and / or wireless connections, and can be proprietary and / or performed using any of a variety of data communication protocols. Any wired connections can be built into the floor and / or walls or ceiling of the operating room. The surgical robotic system can provide video output to one or more displays, including displays within the operating room as well as remote displays that can be accessed via the internet or other network. The video output or feed can be encrypted to ensure privacy, and all or one or more portions of the video output can be saved to a server, an electronic health care record system, or other suitable storage medium.

[0032] In some variations, additional user consoles 100 can be provided, for example, to control additional surgical instruments and / or to control one or more surgical instruments on the primary user console. This would allow, for example, a surgeon to take over or explain a technique to a medical student and a physician in training during a surgical procedure, or to assist during complex surgeries requiring multiple surgeons to act simultaneously or in a coordinated manner.

[0033] In some variations, as Figure 2 As shown in the illustrative diagram of FIG. 27, one or more third party devices 240 can be configured to communicate with the user console 210 and / or other suitable portions of the robotic surgical system. For example, as described elsewhere herein, a surgeon or other user can sit in the user console 210, which can communicate with the control tower 230 and / or robotic instruments in the robotic system 220. Medical data (e.g., endoscopic images, patient vital signs, tool status, etc.) can be displayed at the user console 210, the control tower 230, and / or other displays. At least a subset of the surgical and other medical related information can also be displayed at the third party device 240, such as a remote computer monitor, which is viewed by a surgical collaborator in the same room or outside the room. Other communications, such as a teleconference using audio and / or visual communications, can further be provided to and from the third party device. The surgical collaborator can be, for example, a supervisor or trainer, a medical colleague (e.g., a radiologist), or other third party, who can view and communicate, for example, via the third party device 240, to assist in the surgical procedure.

[0034] Figure 3 is an illustrative diagram of an exemplary variation of a system 300 that includes a robotic surgical system and its interaction with other devices and other parties. Although Figure 3 particular architecture of various connections and communication systems is depicted in Figure 3 the arrangement shown in FIG. 27 is for illustrative purposes. The system 300 can include a surgical robotic platform 302 that facilitates integration of medical data from discrete medical data resources generated from various parties. The data from the discrete medical data resources can be used, for example, to form temporally coordinated medical data. As described further herein, a multi-panel display of the temporally coordinated medical data can be configured and presented.

[0035] The platform 302 can be, for example, a machine with one or more processors 310 connected to one or more input / output devices 312 via a bus 314. The at least one processor can include, for example, a central processing unit, a graphics processing unit, an application-specific integrated circuit, a field-programmable gate array, or a combination thereof.

[0036] The surgical robotic platform 302 can include one or more input ports to receive medical data from discrete medical data resources. For example, a surgical robotic port 329 can receive surgical robotic data from a surgical robot 330. For example, such data can include position data or other suitable status information. An imaging port 331 can receive imaging data from an imaging device 332, such as an endoscope, configured to capture images (e.g., still images, video images) of a surgical site. The endoscope can be inserted, for example, through a natural orifice or through a hole in the surgical patient. As another example, one or more medical instrument ports 333 can receive patient vital sign information from medical instruments 334 (e.g., pulse oximeter, electrocardiogram device, ultrasound device, etc.). Additionally, as another example, one or more user control data ports 335 can receive user interaction data from one or more control devices that receive user input from a user to control the system. For example, one or more handheld user input devices, one or more foot pedals, and / or other suitable devices (e.g., eye tracking, head tracking sensors) can receive user input.

[0037] The surgical robotic platform 302 can further include one or more output ports 337 configured to connect to one or more displays 338. For example, the displays 338 can include open displays (e.g., monitor screens) in a user console, immersive displays or head-mounted devices with displays, supplemental displays such as team displays on a control tower display, bedside displays (e.g., nurse displays), overhead “stadium” style screens, etc. For example, the graphical user interfaces disclosed herein can be presented on the one or more displays 338. The one or more displays 338 can present three-dimensional images. In some variations, the one or more displays 338 can include touchscreens. The one or more displays 138 can be a single display with multiple panels, where each panel presents different content. Alternatively, the one or more displays 138 can include a collection of individual displays, where each individual display presents at least one panel.

[0038] In some variations, the network interface 316 can also be connected to the bus 314. For example, the network interface 316 can provide connectivity to a network 317, which can be any combination of one or more wired and / or wireless networks. The network 317 can help enable communication between the surgical robotic platform 302 and other data sources or other devices, for example. For example, one or more third-party data sources 340 can also be connected to the network 317. The third-party sources 340 can include a third-party device (e.g., another computer operated by a third party such as another doctor or medical specialist), a repository of video surgical procedure data (e.g., data that can be relevant to a procedure being performed by the surgeon), or other suitable additional sources of information relevant to a surgical procedure. For example, the third-party device data can be communicated to a panel displayed to the surgeon before, during, or after a procedure.

[0039] As another example, one or more application databases 342 can be connected to the network 317 (or alternatively stored locally within a memory 320 within the surgical robotic platform 302). The application databases 342 can include software applications (e.g., as described in further detail below) that the surgeon can be interested in during a procedure. For example, the software applications can provide access to stored patient medical records, provide a checklist of surgical tasks for a surgical procedure, perform machine vision techniques to assist with a procedure, perform machine learning tasks to improve surgical tasks, etc. Any suitable number of applications can be invoked. Information associated with the applications can be displayed in a multi-panel display or other suitable display during a procedure. Additionally or alternatively, information provided by one or more of the applications can be provided through a separate resource (e.g., a machine learning resource) that is additionally in suitable communication with the surgical robotic platform 302.

[0040] In some variations, one or more of the software applications can run as a separate process that uses an application program interface (API) to draw objects and / or images on the display. Different levels of complexity of the API can be used. For example, a simple API can include several templates with fixed widget sizes and locations that can be customized by the GUI module with custom text and / or images. As another example, a more complex API can allow the software application to create, place, and delete different widgets such as labels, lists, buttons, and images.

[0041] Alternatively, one or more software applications may render themselves for display. This can, for example, allow for advanced customization and complex behavior of the applications. For instance, this approach can be implemented by allowing the application to pass frames rendered by a graphical user interface (GUI) module 324, which may be computer-readable program code executed by processor 310. Alternatively, an image buffer can be used as a store for the application to render itself.

[0042] In some variations, one or more software applications can run and render themselves independently of the GUI module 324. However, the GUI module can still launch such applications, instruct applications or the operating system where to position applications on the display, and so on.

[0043] As another approach, in some variations, one or more applications can run completely independently of the GUI rendered by the GUI module. For example, such applications can have physical video and data connections to the system (e.g., via suitable input / output devices, networks, etc.). The data connection can be used to configure the application's video feed to appropriate pixel dimensions (e.g., full-screen, half-screen, etc.).

[0044] like Figure 3 As shown, in some variations, memory 320 may also be connected to bus 314. Memory 320 may be configured to store data processed according to implementations of the methods and systems described herein.

[0045] In some variations, memory 320 may be configured to store other types of data and / or software modules for execution. For example, a user console may include memory 320 storing a GUI module 324 with executable instructions to perform the operations disclosed herein. The GUI module may, for example, combine and aggregate information from various software applications and / or other medical data resources for display. In some exemplary variations, one or more software applications may be incorporated into the base code of the GUI module, causing the module to draw graphics and display text at appropriate locations on the display. For example, the module may retrieve images from a database or push images from instruments (e.g., endoscopic cameras) in the operating room to the interface via a wired or wireless interface.

[0046] In some variations, medical data can be collected from discrete medical data resources (e.g., surgical robot 330, endoscope 332, medical instrument 334, control device 336, third party data sources 340, application databases 342, etc.). Additionally, at least some of the medical data can be time-coordinated such that, if necessary, time-sensitive information from different medical data resources are aligned on a common timeline. For example, surgical robot position data can be time-coordinated with endoscope data, which is coordinated with operator interaction data from a control device. Similarly, web resources such as information provided by one or more software applications can be presented at appropriate points in time along with other time-coordinated data. Multi-panel displays and / or other suitable displays can be configured to communicate medical information (e.g., including time-coordinated medical data) as part of a graphical user interface (GUI).

[0047] In some embodiments, the GUI can be displayed in a multi-panel display at a user console that controls a robotic surgical system. Additionally or alternatively, the GUI can be displayed at one or more additional displays, such as at a control tower of the robotic surgical system, at a patient bedside, etc. In general, the GUI can provide more efficient communication of information to users in the user console and / or other personnel, as well as more efficient communication and collaboration between different parties involved in a surgical procedure.

[0048] The following section describes one particular embodiment that can be used with a GUI of robotic surgical system 150. Examples of other GUI embodiments are described in U.S. Patent Application 15 / 842,485, “Multi-Panel Graphical User Interface for a Robotic Surgical System,” filed December 14, 2017, which is incorporated by reference herein.

[0049] Stadium view application of the GUI

[0050] In one embodiment, the GUI runs a “stadium view” application that renders a graphical representation of the current positions of the staging table and the plurality of robotic arms 160 (i.e., not actual camera views). The stadium view can also show additional information, such as but not limited to graphical representations of the patient, operating room staff, and / or other elements of the operating room environment. The graphical representations can be two-dimensional or three-dimensional, and can be generated by the robotic surgical system 150, or rendered by a separate component and provided to the stadium view application for display.

[0051] The current position can be derived from real-time or near-real-time information related to the current position of the stage and multiple robotic arms 160. For example, a graphical representation of the robotic arm (sometimes referred to herein as "rendering") can be based at least in part on one or more motion algorithms that control the robotic arm. One or more motion algorithms can be fed into a modeling module that transforms the motion information into a rendered two-dimensional or three-dimensional model. As another example, the rendering of the robotic arm and / or stage can be based at least in part on one or more sensors (e.g., position sensors in the robotic arm, infrared sensors tracking markers placed on the robotic arm 160 or stage around the operating room, etc.).

[0052] This stadium-style view provides users with an "external" view of the robotic surgical system 150, the patient, and / or staff within the operating room. For example, users can monitor the status of the robotic system, such as tool status, potential collisions, etc., and communicate such statuses and solutions to any problems to other members of the surgical team. Furthermore, in some variants, users can interact with the graphical representation within the stadium-style view application and implement one or more variations of the robotic surgical system, as described below.

[0053] Figure 4A An exemplary implementation of a stadium-style view application is shown in the image. For example... Figure 4A As shown, the stadium-style view application 900 can display a patient on a patient table and a 3D rendering 910 of multiple robotic arms connected to the patient. A perspective guide 920 can also be displayed to indicate which view of the 3D rendering is currently being displayed (e.g., perspective view, floor plan, etc.). Furthermore, as for example... Figure 4C As shown, at least some of the robotic arms can be digitally labeled to distinguish different robotic arms (e.g., to help achieve better communication about the state of a particular arm). In another view within the stadium-type view application 900, additional information about the state of the robotic system (e.g., which types of tools are attached to the corresponding robotic arms, the activation status of the tools, etc.) can be additionally displayed near render 910.

[0054] The display of 3D rendering 910 in the stadium-style view application can be modified based on the state of the rendered objects. For example, as... Figure 4B As shown, render 910 is typically a nominal render, where specific parts of render 910 are not selected or highlighted. For example... Figure 4C As shown, the stadium-style view application can be configured to, for example, highlight at least one robotic arm in response to a user selection on the arm (in... Figure 4CThe user can select a particular robot arm, and in response, the stadium view application can display information about the status of the selected arm. As shown, for example Figure 4E In response to the user selection of the arm, the stadium view application can also display and / or highlight information related to the selected arm and its associated tool, such as the tool type (e.g., “scissors”), the tool status (e.g., an operational status such as “cutting” or “clotting,” and / or remaining staples, etc.), and the like. As another example, the user can select a particular robot arm, causing the robot arm to be highlighted in the user’s displayed GUI and in another displayed instance of the GUI (e.g., on a control tower display) to more easily communicate with other surgical staff regarding the robot arm, reducing confusion.

[0055] As another example, the user can select a robot arm rendered in the stadium view application and move it (e.g., through a click-and-drag interaction) to effect a change in the position (pose) of the actual selected robot arm. The movement of the selected rendered robot arm can be communicated, for example, to a robot arm controller, which resolves the new position into a series of one or more actuator commands to actuate joints in the robot arm so that the robot arm position matches the new position of the rendered robot arm. Thus, the stadium view application can provide a way to help the user to be able to “manually” reposition the robot arm from the user console without physically touching the robot arm. Similarly, the position of the patient table can be adjusted via adjustment of a rendered patient table within the stadium view application.

[0056] In some variations, the stadium view application can be configured to inform the user of collisions between robot arms. In some variations, collisions (e.g., impending or already occurred) can be detected based on proximity or contact sensors on the robot arms, machine vision techniques, and / or in any suitable manner. In response to receiving information indicating an impending or already occurred collision, the stadium view application can highlight one or more robot arms involved in the collision. For example, as shown Figure 4DAs shown, one or more rendered robotic arms (labeled "3" and "4") can be highlighted. Additionally or alternatively, a warning notification 930 explaining the collision can be displayed. An audio warning indicating the collision can additionally be provided by the stadium view application. It will be appreciated that in other variations, the stadium view application can provide warnings or notifications of other kinds of status updates (such as a surgical instrument error) in a similar manner. For example, rendered displays of other portions of the robotic system and / or other suitable portions of the operating room environment can be highlighted to indicate other kinds of status changes or to provide suitable updates. Notifications similar to the warning notification 930 of other kinds of status updates can also be provided via the stadium view application.

[0057] In the above examples, the stadium view is used intraoperatively during a procedure to provide the surgeon (using the display on the bridge 100) and / or staff (e.g., using the display 134 on the control tower 133 or the display 132 located at the patient's side / adjacent to the patient) with a real-time or near real-time view of the position of the robotic arms 160 to warn of possible collisions between the arms 160 and / or other objects. The above examples also show that the stadium view can be used by the surgeon during a procedure to guide staff during an instrument exchange on the robotic arms. As shown by these examples, because the stadium view provides the surgeon with a way to have a real-time or near real-time out-of-body view of the robotic arms 160, table, and patient from the surgeon's console 100, this implementation allows the surgeon to better communicate with the bedside staff during a procedure. That is, using the stadium view, the surgeon at the surgeon's console 100 can have a better understanding of the bedside situation and clearly communicate with the bedside staff because the surgeon will have a real-time (or near real-time) view of the arms 160 with identification of the arms 160 and tool names. The stadium view can also provide feedback, such as error or warning information, to help resolve problems (e.g., arm collisions).

[0058] In another example, the stadium view can be used pre-operatively and / or post-operatively to guide staff (e.g., nurses) in completing arm setup and / or teardown by providing user guidance information on how to move the plurality of robotic arms 160 to different positions. Pre-operative / post-operative arm setup / teardown is an important process that users of the robotic surgical system 150 must face. Users can find the visual guidance provided by the stadium view of this implementation easier to follow and more desirable than textual guides. For example, during setup of the robotic system, a particular robotic arm can be highlighted in the stadium view application to indicate that the next tool according to the procedure template application should be attached to that particular robotic arm. Other guidance can be provided via the stadium view application, such as textual descriptions and / or other graphical representations of tools and animations (videos), etc., to further assist surgical staff in setting up, tearing down, or otherwise facilitating the system. The following paragraphs and Figures 5-12 Further information is provided regarding this implementation.

[0059] Figure 5 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing user guidance information on how to position a patient. Figure 6 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing user guidance information on how to deploy a robotic arm. Movement of the arm can be accomplished using a user input device (e.g., remote control), for example. Figure 7 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing instructions on how to drape a robotic arm. Figure 8 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing instructions on how to move a robotic arm to a ready arm position. Figure 9 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing instructions on how to place a port in a patient. Figure 10 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing instructions on how to deploy a robotic arm to a docking arm position. Figure 11 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing instructions on how to insert an instrument in a robotic arm. Figure 12 is an illustration of a graphical user interface of an implementation of a stadium view application that includes providing instructions on how to deploy a robotic arm to an assigned arm position.

[0060] In one implementation, the rendered display of one or more parts of the robotic system can be modified to help guide the surgical team during the setup and / or disassembly of the robotic surgical system 150 (e.g., preoperative and / or postoperative procedures), or otherwise assist the system 150. That is, in addition to a graphical representation of the current positions of the display table and multiple robotic arms 160, the GUI can display user-guided information on how to move the multiple robotic arms 160 to different positions (e.g., arm setup positions or arm disassembly positions). This will now be combined with... Figures 13-15 Describe this implementation scheme.

[0061] Figure 13 A graphical representation of the current positions of the platform and multiple robotic arms 160 is shown. Arms 160 are in the "retracted" position under the platform. Figure 13 The document also includes user guidance information on how to move multiple robotic arms 160 to different positions (in this case, the fully deployed positions). The user guidance information in this example is a graphical representation of the multiple robotic arms in different positions. Figure 13 As shown, the current positions of the multiple robot arms 160 are represented by solid lines, while the positions of the multiple robot arms 160 at different locations are represented by imaginary lines. However, other methods can be used to display and deliver this user guidance information.

[0062] By being able to see where arm 160 should be positioned, the user can simply move arm 160 until the graphical representation of the current position of robot arm 160 overlaps with the graphical representation of the "destination position" of robot arm 160 (see...). Figure 14 Because the graphical representation of the current position of the robotic arm 160 moves as the user moves the arm, the user receives continuous visual feedback on how to position the robotic arm 160 appropriately. Optionally, the robotic surgical system 150 can provide audible user guidance on how to move multiple robotic arms 160 to different positions. For example, as the user is moving the robotic arm 160 closer to its intended position, the system 150 can provide auditory cues, such as a more frequent "beep" sound when the user is on the correct track, followed by different sounds to indicate successful positioning.

[0063] Figure 13 An example of an arm setup is shown, while Figure 15 The GUI, featuring a stadium-style view, illustrates how it can be used to retract the robotic arm 160 when the arm is detached. Figure 15 As shown, the arm in the retracted position is indicated by a dashed line, and the user will move the robot arm 160 to the position indicated by the dashed line to retract the arm 160.

[0064] Of course, these are merely examples, and other implementations of user guidance information can be used. For example, instead of indicating the final position of the robotic arms 160 with only static images, the GUI can provide an animation / video that accurately shows how the robotic arms 160 should be manipulated to move them to their final positions. As another example, the user guidance information can be a display of which contacts on the robotic arms 160 to press and how to move the robotic arms 160. As yet another example, in addition to the graphical representations of the arms 160, table, and patient, the stadium view can include an endoscopic view of the surgical site (captured by an endoscopic camera on one of the robotic arms 160). Further, these implementations can be used to guide the user in positioning the patient and table accessories. This can help enable the stadium view to reflect the precise position of the patient on the table. In addition, these implementations can be used to automatically detect when the user has skipped or missed a setup / disassembly step, and inform the user of this with guidance on how to resolve the issue. Finally, as noted above, the images shown in the figures are merely examples, and other graphics or representations can be used. For example, Figure 16 is an illustration of a graphical user interface of another implementation.

[0065] The foregoing description, for purposes of explanation, uses specific nomenclature to provide a thorough understanding of the application. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the application. The foregoing descriptions of particular embodiments of the application are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the application to the precise forms described. Many modifications and variations are possible in view of the above teachings. The implementation was chosen and described in order to best explain the principles of the application and its practical application, thereby enabling others skilled in the art to best utilize the application, and various implementations with various modifications as are suited to the particular use contemplated. It is intended that the following claims define the scope of the application and that methods equivalent to those shown in the disclosure and covered by the claims are within the scope of the application.

Claims

1. A robotic surgical system, comprising: Multiple robotic arms are mounted on the operating table; Display devices; as well as Processor, the processor being configured to: The motion model of the robot arm is input into the modeling module to render a graphical representation, and the graphical representation of the modeling module of the multiple robot arms at the current position is rendered on the display device. as well as The display device shows user guidance information on how to move the multiple robotic arms from their retracted positions to different positions or from their current positions to the retracted positions. The retracted position corresponds to the retraction of the plurality of robotic arms under the operating table surface, and the graphical representation of the plurality of robotic arms moves to reflect position changes in real time, so that the user receives continuous visual feedback on how to position the plurality of robotic arms in the appropriate position.

2. The robotic surgical system of claim 1, wherein the different locations are deployment locations.

3. The robotic surgical system of claim 1, wherein the current position is the deployment position.

4. The robotic surgical system of claim 1, wherein the graphical representation of the plurality of robotic arms at their current positions is a solid line, and wherein the user guidance information includes a graphical representation of the plurality of robotic arms at their different positions or at the retracted position as imaginary lines.

5. The robotic surgical system of claim 1, wherein the user guidance information includes instructions on how to manipulate the user input device to move the plurality of robotic arms to the different positions or the retracted position.

6. The robotic surgical system of claim 1, wherein the processor is further configured to provide audible user guidance information.

7. The robotic surgical system of claim 1, wherein the display device is positioned adjacent to the patient table.

8. The robotic surgical system of claim 1, wherein the user guidance information includes instructions on which contacts on the plurality of robotic arms to manipulate.

9. The robotic surgical system of claim 1, wherein the processor is configured to inform the user about collisions between the plurality of robotic arms.

10. The robotic surgical system of claim 9, wherein the processor is configured to highlight one of the plurality of robotic arms in the graphical representation, the highlighted robotic arm being involved in the collision.

11. A method for a robotic surgical system, the method comprising: Performing rendering and display of static images in a robotic surgical system that includes multiple robotic arms and display devices: The rendering includes rendering a graphical representation of the current positions of the plurality of robotic arms on the display device; and The display includes a graphical representation of the destination position of the plurality of robotic arms displayed on the display device, wherein the graphical representation of the destination position of the plurality of robotic arms is displayed differently from the graphical representation of the current position of the plurality of robotic arms, the current position or the destination position including the plurality of robotic arms retracted under the operating table, wherein the graphical representation of the destination position and the current position of the plurality of robotic arms each includes a drawn object different from the camera view; and The graphical representation of the current position is rendered by inputting the motion model of the plurality of robotic arms into the modeling module; The method described therein is performed before and / or after the procedure to guide staff in completing arm setup and / or disassembly; The graphical representation of the movement of the plurality of robotic arms is used to reflect changes in position in real time, providing the user with continuous visual feedback on how to position the plurality of robotic arms appropriately.

12. The method of claim 11, wherein the graphical representation of the destination positions of the plurality of robotic arms is displayed as imaginary lines, and wherein the graphical representation of the current positions of the plurality of robotic arms is rendered as solid lines.

13. The method of claim 11, wherein the destination location is the plurality of robotic arms retracted below the operating table.

14. The method of claim 11, further comprising providing audible user guidance information.

15. The method of claim 11, wherein the display device is positioned adjacent to the patient table.

16. The method of claim 11, further comprising displaying an indication of which contacts on the plurality of robotic arms to be manipulated.

17. A robotic surgical system, comprising: Multiple robotic arms; as well as A means for providing user guidance information on how to move the plurality of robotic arms to different positions, the user guidance information including highlighting a first robotic arm of the plurality of robotic arms, wherein a second robotic arm of the plurality of robotic arms is not highlighted, the first robotic arm being highlighted in response to user selection of the first robotic arm, and the different positions including a stowed position under the patient table; The device for providing user guidance information includes a device for rendering a graphical representation by inputting the motion model of the robot arm into a modeling module, and rendering the user guidance information into a graphical representation of the modeling module of the plurality of robot arms on a display device; The means for providing the means includes means for displaying graphical representations of the plurality of robotic arms at different positions on a display device, together with a rendered display of graphical representations of the plurality of robotic arms at their current positions, wherein the graphical representations of the plurality of robotic arms move to reflect position changes in real time, so that the user receives continuous visual feedback on how to position the plurality of robotic arms appropriately.

18. The robotic surgical system of claim 17, wherein the graphical representation of the plurality of robotic arms at the different locations is displayed differently from the rendered graphical representation of the plurality of robotic arms at the current location.

19. The robotic surgical system of claim 17, wherein the means for providing includes means for providing instructions on how to manipulate the user input device to move the plurality of robotic arms to the different positions.

20. The robotic surgical system of claim 17, wherein the means for providing includes means for providing audible user guidance information.

21. The robotic surgical system of claim 17, wherein the display device is positioned adjacent to the patient table.

Images