Situational awareness of surgical robots with variable arm positioning

Abstract

A surgical robotic system includes: a plurality of robotic arms, including a first robotic arm coupled to an instrument; and a console communicatively coupled to the robotic arms. The console includes a display device and a processor coupled to the display device and configured to display, at the display device, an instrument-to-arm mapping model including a graphical representation of the robotic arms, wherein the instrument-to-arm mapping model depicts the position of each robotic arm and indicates instrument data associated with the first robotic arm describing the instrument.

Description

[0001] priority This patent application claims priority to U.S. Provisional Patent Application No. 63 / 596,110, filed November 3, 2023, entitled “SITUATIONAL AWARENESS OF SURGICAL ROBOT WITH VARIED ARM POSITIONING”, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The systems and methods disclosed herein relate to devices and methods for indicating the position or orientation of surgical instruments, and more specifically to surgical robotic systems for indicating the position or orientation of surgical instruments. Background Technology

[0003] Robotic systems can be used to perform a wide variety of tasks and procedures. They are applicable to various industries, such as manufacturing, automotive, healthcare, and construction. For example, in the healthcare industry, robotic surgical systems have been used to perform a large number of medical procedures, including minimally invasive surgery (e.g., laparoscopic surgery) and non-invasive surgery (e.g., endoscopic surgery). Generally, a robotic system may include a robotic arm and a control console. The robotic arm is configured to control the movement of tools or instruments attached to it, and the user can control the movement of the robotic arm and / or tools through the control console. Summary of the Invention

[0004] In one embodiment, a surgical robotic system is disclosed. The surgical robotic system includes: a plurality of robotic arms, including a first robotic arm coupled to an instrument; and a console communicatively coupled to the robotic arms. The console includes a display device and a processor coupled to the display device and configured to display, at the display device, an instrument-to-arm mapping model including a graphical representation of the robotic arms, wherein the instrument-to-arm mapping model depicts the position of each robotic arm and indicates instrument data describing the instrument and associated with the first robotic arm.

[0005] In another embodiment, a method performed by a surgical robotic system is disclosed. The method includes: displaying an instrument-to-arm mapping model at a console of the surgical robotic system, the instrument-to-arm mapping model including a graphical representation of a plurality of robotic arms of the surgical robotic system, wherein the instrument-to-arm mapping model depicts the position of each robotic arm relative to a patient platform of the surgical robotic system, and indicating instrument data describing an instrument associated with a first robotic arm of the robotic arms, wherein the first robotic arm is coupled to an instrument.

[0006] In another embodiment, a non-transitory computer-readable medium storing instructions is disclosed. The non-transitory computer-readable medium stores instructions that, when executed by a processor of a surgical robotic system including multiple robotic arms, cause the processor to: display an instrument-to-arm mapping model at a console of the surgical robotic system, the instrument-to-arm mapping model including a graphical representation of the robotic arms relative to a patient platform of the surgical robotic system; indicate in the instrument-to-arm mapping model instrument data describing an instrument and associated with a first robotic arm among the robotic arms, wherein the first robotic arm is coupled to the instrument; and display in the instrument-to-arm mapping model a haptic interface device (HID) indicator along with the instrument data, wherein the HID indicator indicates whether a first HID or a second HID of the console is configured to control the instrument.

[0007] In another embodiment, the present invention discloses a surgical robotic system. The surgical robotic system includes: a patient platform; a plurality of robotic arms including a first robotic arm, wherein the first robotic arm is coupled to an instrument; and a console communicatively coupled to the robotic arms. The console includes a first haptic interface device (HID) and a second HID, and a display configured to display an instrument-to-arm mapping model, the instrument-to-arm mapping model including a graphical representation of the robotic arm relative to the patient platform of the surgical robotic system; instrument data in the instrument-to-arm mapping model describing the instrument and associated with the first robotic arm; and haptic interface device (HID) indicators displayed in the instrument-to-arm mapping model along with the instrument data, wherein the HID indicators indicate whether the first HID or the second HID of the console is configured to control the instrument.

[0008] It should be noted that the various examples described above can be combined with any other examples presented herein. The features and advantages described in the specification are not exhaustive, and in particular, many additional features and advantages will be apparent to those skilled in the art, taking into account the drawings, specification, and claims. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and guidance purposes and may not be intended to depict or limit the subject matter of the invention. Attached Figure Description

[0009] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects, wherein similar reference numerals denote similar elements.

[0010] Figure 1 A display is shown at the console of a robot system according to various embodiments of the present disclosure.

[0011] Figure 2A , Figure 2B, Figure 2C and Figure 2D Various embodiments of the present disclosure are shown, such as Figure 1 The example shown is an instrument-to-arm mapping model and settings menu displayed on the console's display device.

[0012] Figure 3 Various embodiments of the present disclosure are shown, such as Figure 1 The example shown is a stadium view model displayed on the console's display device.

[0013] Figure 4 Various embodiments of the present disclosure are shown, such as Figure 1 The example shown is a target screen model rendered at the display device of the console.

[0014] Figure 5A and Figure 5B Various embodiments of the present disclosure are shown, such as Figure 1 The example shown is a state model displayed on the console's display device.

[0015] Figure 6 Various embodiments according to this disclosure are shown in, for example Figure 1 The console shown is displayed on the display device. Figures 2A to 2C , Figure 3 , Figure 4 and Figures 5A to 5B Examples of various models.

[0016] Figure 7 This is a flowchart illustrating a first method according to an embodiment of the present disclosure.

[0017] Figure 8 This is a flowchart illustrating a second method according to an embodiment of the present disclosure.

[0018] Figure 9 This is a schematic diagram illustrating the electronic components of a surgical robot system according to some embodiments. Detailed Implementation

[0019] A. Robotic System A robotic system may include various components that work together to perform a specific task. While the specific design and functionality of a robotic system can vary based on industry and application, some examples of robotic systems may include one or more robotic arms communicatively coupled to a control console. The control console can be used to manipulate the robotic arms and instruments detachably attached to them. The robotic arm may include one or more joints and links interconnecting the joints. The robotic arm may also include end effectors (e.g., grippers, tools, instruments, etc.) positioned at or detachably attached to the robotic arm, such that the end effectors interact with the environment or a subject.

[0020] The control console of a robotic system can essentially serve as a control center or interface, allowing an operator or user to interact with and control the robot. The console acts as a command and feedback point for the robotic system, providing users with a mechanism to input instructions, monitor the status of various components of the robotic system, and receive information about the components and environment of the robotic system. For example, the console may include a display device configured to display images captured by the robotic system's cameras, data describing the components of the robotic system, the settings and status of the components, menus related to the control of the robotic system, and other data describing the robotic system. The console may also include one or more input devices configured to control the robot arm / end-effector of the system. Input devices may include, for example, buttons, switches, touch-sensitive surfaces, gimbals, haptic interface devices (HID), etc., located in various areas of the console (e.g., near the user's hand or foot). The user of the console can actuate or interact with the input devices to control the robot arm / end-effector, manipulate information displayed on the console, adjust console settings, etc. It should be understood that, in addition to the robot arm and console, a robotic system may also include many other components, such as, for example, processing hardware, sensors, power supplies, safety systems, programming interfaces, memory, etc.

[0021] For illustrative purposes, this document may describe various implementations in relation to surgical robotic systems used in the medical industry. However, it should be understood that robotic systems can be applied to many other industries, and the robotic systems described herein are not limited to surgical robotic systems.

[0022] To this end, an exemplary surgical robotic system may include multiple robotic arms whose movement can be controlled by a HID located on a console of the surgical robotic system. The robotic arms can be positioned in a manner convenient for performing surgery on the patient, and the healthcare provider can manipulate the HID to control the movement of end effectors of the robotic arms (e.g., which may include medical instruments, cameras, etc.), which can interact with or be positioned within the patient's body. At least one of the instruments may include a camera that can be used to capture real-time images of the patient's anatomy during surgery. The images may also show the movement of the end effectors relative to the patient's anatomy. A display device on the console can display the images in real time, thereby providing visual feedback to the healthcare provider during surgery, which may be crucial for making informed decisions during the procedure.

[0023] B. Display devices for robot systems In some cases, the display device can be implemented as a head-mounted device, where the structure of the display device essentially surrounds the area around the user's eyes to create a sense of presence and immersion in the images and data displayed by the device. When the user is immersed, they can insert their head into the display device in a way that blocks out ambient light during immersion. In this way, the user can focus on images (e.g., patient anatomy) and instrument movement while performing various tasks (e.g., surgery on a patient), all of which can be displayed on the device. The user can perform these tasks without being distracted by any movement or events occurring outside of immersion. In effect, immersion at the display device reduces the likelihood of a healthcare provider being distracted during surgery, which could otherwise lead to injury or harm to the patient.

[0024] However, immersion in the display device limits the amount of information a user can access while performing a task using a robotic arm. For example, the display device could be limited to showing images of the patient received by a camera that defines the patient's range. In this way, the information displayed on the device can generally be minimized to further reduce the risk of distracting the user from the primary focus of the surgery.

[0025] In some cases, additional information may also be displayed on the display device, but this information may be limited and not describe the environment around the robotic arm. For example, the information displayed on the display device may not include the position of each robotic arm relative to each other and relative to the subject using the robotic arm to perform a task (e.g., a patient positioned on a patient platform). This may be because such information display may not be spatially conservative, and too much information may distract the user during surgery.

[0026] However, this information about the robotic arms and attached instruments can be crucial for making informed decisions during surgery. For example, healthcare providers may need to continuously understand the spatial orientation of the robotic arms to have tableside visibility of the surgical robotic system. The term "tableside" can refer to the area encompassing the actual patient platform, robotic arms, instruments, and immediate surroundings on which the surgical robotic system is placed during surgery (i.e., excluding the console). Such perception of the tableside area of ​​the surgical robotic system can be important for healthcare providers when troubleshooting robotic arm and instrument collisions and orienting instruments in the workspace. This awareness can also be important for healthcare providers in understanding how the robotic arms move when adjusting the patient platform during surgery and can also be used to determine how to grasp instruments during surgery. In this way, failure to indicate the position of the robotic arms and the instruments attached to each of the robotic arms can be problematic, as healthcare providers may need access to this information at certain points before or during surgery for surgical setup, troubleshooting, collision avoidance, etc. However, to access this information, the healthcare provider may need to remove himself or her from the immersion, leave the console, and physically move to the location of the robotic arm to view the positioning of the robotic arm and attachment instruments. Alternatively, another healthcare provider or person physically present at the console may need to verbally or otherwise communicate the positioning of the robotic arm and attachment instruments to the healthcare provider so that the healthcare provider is aware of the information.

[0027] Furthermore, the information displayed on the display device may not indicate the system's hand-handling in an easily understandable format, and the format may be space-conservative (i.e., minimizing the display size on the display device). The term "hand-handling" can refer to the assignment of control of a device (and therefore, the associated robotic arm to which the device is attached) to a specific input device at the control console. Similarly, the information displayed on the display device may not indicate the difference between a device actively engaged by an input device and a device that can be attached to the robotic arm but not actively engaged by the input device. When a device is attached (e.g., locked in a detachably attached manner) to the robotic arm, the device may be assigned to an input device, but when an action performed at the input device actually causes movement or manipulation of the robotic arm and / or the attached device, the device may be actively engaged only by the input device.

[0028] Failure to indicate the bias of a robotic system to the user can lead to confusion and potentially errors and delays when using the robotic arm to perform tasks or surgeries. For example, a healthcare provider may need additional time before the procedure to manually identify the instruments controlled by each input device, or to manually test the input devices to identify the instruments that move with the movement of the corresponding input device. This is especially true in surgical robotic systems where the positioning of the robotic arm may not be indicated in a linear and sequential manner. When, for example, the robotic arm translates around the operating table, it may not be indicated in a linear and sequential manner, such that the order in which the robotic arm moves from one side (i.e., the left) to the other (i.e., the right) is not always sequential. Therefore, in addition to images of the subject and basic information on the instruments or robotic arm, the display device may not show details that might be important to the user or otherwise helpful during the performance of the surgery.

[0029] For example, Figure 1 Figure 1100 shows a console 240 displaying an image 1103 of the patient's anatomy and active instruments 212A, 212B controlled by the user of the console 240. Figure 1 The console 240 shown may include a display device 242, a touchscreen 232, and one or more user input devices 226, 228 (also referred to herein as "HID 226, 228"). Although HID 226, 228... Figure 1 The HID 226 and 228 are displayed as handles or joysticks, but it should be understood that the HID 226 and 228 can be implemented as any other type of input device (e.g., buttons, switches, touch-sensitive surfaces, toggle switches, etc.).

[0030] Console 240 may include display device 242, left HID 226 (also referred to herein as "first HID 226"), and right HID 228 (also referred to herein as "second HID 228"). However, it should be understood that console 240 may include in Figure 1 Other components not shown or described elsewhere.

[0031] The healthcare provider can sit at console 240 with his or her head immersed in display device 242. The healthcare provider's left hand can operate left HID 226, and the healthcare provider's right hand can operate right HID 228. Each of HIDs 226 and 228 can be assigned to control multiple robotic arms 210, but can only be engaged with one robotic arm 210 at a time and actively control one robotic arm, and thus engage with one instrument 212A and 212B at a time and actively control one instrument. The robotic arm 210 and instruments 212A and 212B that are actively engaged with and controlled by HIDs 226 and 228 are referred to herein as "active robotic arm 210" and "active instrument 212A and 212B".

[0032] Display device 242 may include a user interface that displays image 1103. Image 1103 may be an image of a patient's anatomy captured by a camera (such as, for example, an endoscopic camera or a laparoscopic camera). In one case, camera 606 may be coupled to one of the robotic arms 210 and capture image 1103 as a real-time stream while a healthcare provider performs surgery on the patient using the surgical robotic system. In another case, camera 606 may have already captured image 1103 before performing surgery on the patient using the surgical robotic system.

[0033] Image 1103 can not only show a portion of the patient's anatomy, but in some cases, it can also show one or more active devices 212A, 212B actively engaged by HID 226 and 228. In cases such as Figure 1 In the example shown, the first instrument 212A and the second instrument 212B are displayed in image 1103. However, it should be understood that in other cases, image 1103 may not display instruments 212A and 212B at all (e.g., instruments 212A and 212B may not have moved into the area captured by image 1103).

[0034] Figure 1 The tableside area 233 of the environment surrounding the surgical robot system 203 is also shown. When immersed in the display device 242, the healthcare provider may not have access to information about the actual location and orientation of the components of the surgical robot system 203 in the tableside area 233. The tableside area 233 shows an example of the environment surrounding the robotic arm 210 of the surgical robot system 203. For example, location and orientation information that may not be indicated in image 1103 may include the location and orientation of the robotic arm 210, various links and / or joints of the robotic arm 210, instrument 212A, 212B information, patient platform location information, drape information, staff location information, tower location information, lighting location information, etc.

[0035] Although Figure 1 While this relates to surgical robot system 203, it should be understood that the above description regarding the lack of visibility of the robotic arm 210 when the user of console 240 engages with display device 242 also applies to other types of robot systems. For example, the robotic arm 210 may be located in a warehouse or factory, and the console 240 may be positioned in a separate area or room away from the robotic arm 210. Similar situations may arise where the user of console 240 may be unaware of the connections between different user input devices on console 240 and the corresponding controlled robotic arms 210 of the robot system.

[0036] C. Introduction to the model used for display in the console This disclosure provides a technical solution to the aforementioned technical problems related to robotic systems and platforms by displaying the positioning of robotic arm 210, instruments 212A, 212B (hereinafter sometimes referred to as "instruments 212") attached to each robotic arm in robotic arm 210, and system hand deviance indications (i.e., indications that instruments 212A, 212B actively engage with and are controlled by their respective HIDs 226, 228) in an instrument-to-arm mapping model displayed at a console. In one embodiment, the instrument-to-arm mapping model may include a graphical representation of robotic arm 210 and the current position and orientation of each robotic arm in robotic arm 210 relative to the patient platform. The instrument-to-arm mapping model may also include instrument data describing instruments 212A, 212B attached to each robotic arm in robotic arm 210. The instrument data of instrument 212A or 212B may be displayed in the instrument-to-arm mapping model as associated with that robotic arm 210.

[0037] In some implementations, a stadium view model of the surgical robotic system 203 may also be displayed at the console 240, where the stadium view model provides a more holistic view of the surgical robotic system 203. The stadium view model can be a graphical representation or rendering of the area surrounding objects of the robotic system, such as a patient. For example, the stadium view model may display a rendering of the patient, the patient platform on which the patient rests, and the robotic arm 210 of the surgical robotic system 203. The stadium view model may also display other external individuals, personnel, or components within the operating room. In some cases, the stadium view model may display the current positions of the patient, the patient platform, and the robotic arm 210, and these positions may be updated as these positions change over time. Users may interact with the console 240 (e.g., touchscreen 232) to display different views of the stadium view model.

[0038] In some implementations, a state model of the surgical robot system may also be displayed at console 240. The state model may display the current position and orientation of each link and joint on each robotic arm 210, and indicate the state of each link and / or joint on each robotic arm 210 (e.g., a certain color / shade may indicate that a link on the robotic arm 210 is locked or unlocked, a certain color / shade may indicate that the robotic arm 210 is engaged with instrument 212A or 212B and docked with the patient, etc.).

[0039] The instrument-to-arm mapping model, stadium view model, and state model may each be a two-dimensional (2D) or three-dimensional (3D) rendering of a different part of the robotic system (i.e., not an actual camera view) and may include other icons, text, or graphics as described herein. The instrument-to-arm mapping model, stadium view model, and state model may be displayed at display device 242 and / or touchscreen 232. When the instrument-to-arm mapping model, stadium view model, or state model is displayed at display device 242, the model may be displayed in picture-in-picture (PIP) format and positioned in a corner of display device 242 so as to overlap only a small portion of the subject image primarily displayed at display device 242. When the instrument-to-arm mapping model, stadium view model, or state model is displayed at touchscreen 232, the model may cover any portion of the display of touchscreen 232 (i.e., the display size is not limited when displayed at touchscreen 232, as touchscreen 232 may not display an image of the object).

[0040] In this way, the embodiments disclosed herein offer several advantages to both the medical provider and the patient when operating the surgical robotic system. For example, by providing the medical provider with an instrument-to-arm mapping model, a stadium view model, or a state model at console 240, the medical provider can set up the surgical robotic system 203 for surgery in a much more efficient and effective manner (e.g., the medical provider can use the model to quickly obtain the information needed to set up the robotic arm). Furthermore, the medical provider can use the instrument-to-arm mapping model, stadium view model, or state model to rule out collisions between the robotic arm 210 and instruments 212A, 212B and take appropriate actions to prevent harm to the patient and ensure safe surgery. Finally, the medical provider can also use the instrument-to-arm mapping model, stadium view model, or state model to understand how the robotic arm 210 moves when adjusting the patient platform intraoperatively, which can be used for decision-making during surgery and to understand how the robotic arm grasps instruments 212A, 212B even during surgery.

[0041] D. Instrument-to-arm mapping model Turn now Figures 2A-2D Various examples of the instrument-to-arm mapping model 1200 and setting menus 1250 and 1275 are shown, which can be displayed in response to the selection of an indicator in the instrument-to-arm mapping model 1200. See now for details. Figure 2AThe present disclosure illustrates an instrument-to-arm mapping model 1200 according to various embodiments thereof. The instrument-to-arm mapping model 1200 provides a technical solution to the aforementioned technical problems by displaying additional information at a console 240, allowing a user to access the information without leaving the console 240 or waiting to receive it from a staff member near the physical robotic arm 210. In one embodiment, the instrument-to-arm mapping model 1200 may include a graphical representation (or rendering) of each robotic arm 210 of the system. In one embodiment, the instrument-to-arm mapping model 1200 may include a graphical representation of each of the joints and / or links of each robotic arm 210 of the system. The instrument-to-arm mapping model 1200 may indicate the current position and orientation of each robotic arm 210 (and each link / joint along the robotic arm 210), which may be updated as the position and orientation of each robotic arm 210 changes during setup or surgery.

[0042] like Figure 2A As shown, the instrument-to-arm mapping model 1200 includes graphical representations of four robotic arms 1230A, 1230B, 1230C, and 1230D (also referred to herein as "robotic arms 1230A, 1230B, 1230C, and 1230D"). Although four robotic arms 1230A-1230D are shown in this example, it should be understood that any number of robotic arms can be represented. The graphical representations of these four robotic arms 1230A-1230D can be rendered to represent the actual physical structure of the four robotic arms 210 deployed by a subject performing a task or surgery on them. The graphical representations of these four robotic arms 1230A-1230D can be rendered in real time using a rendering application, or can be obtained from a library pre-loaded with graphical representations of robotic arms 1230A-1230D.

[0043] Robotic arms 1230A-1230D can also depict one or more joints and / or links positioned along the corresponding robotic arm 210 (e.g., including their rendering). Graphical representations of joints, links, and other components on the robotic arm 210 can be rendered in real time using a rendering application, or can be obtained from a library pre-loaded with graphical representations of joints, links, and other components.

[0044] In some implementations, the position and orientation of each robotic arm 1230A-1230D in the instrument-to-arm mapping model 1200 can reflect the current position and orientation of each robotic arm 210. For example, links and / or joints along each robotic arm 210 may include one or more processors or encoders that can acquire (e.g., calculate) position data within the workspace and / or relative to the subject or platform. This data can be transmitted along the links and / or joints of each robotic arm 210 to a processor located at or coupled to the console 240. The processor can visualize the robotic arms 1230A-1230D within the instrument-to-arm mapping model 1200 to reflect the current and accurate position and orientation of the robotic arm 210 based on the received position data. The position data can be collected in real time (i.e., continuously throughout the use of the robotic arm 210) or based on a predefined schedule (e.g., every millisecond (ms), every 2 ms, etc.). In this way, the processor can continuously receive updates describing the position and orientation of each robotic arm in the robotic arm 210 relative to the subject or platform (e.g., at the tableside). The processor can then use this update to correspondingly update the position and orientation of each robotic arm in the robotic arms 1230A-1230D in the instrument-to-arm mapping model 1200.

[0045] For example, the position and orientation of each robotic arm in the robotic arms 1230A-1230D can be rendered in the instrument-to-arm mapping model 1200 relative to a graphical representation of the patient platform 1234. The graphical representation of the patient platform 1234 can be rendered to represent the actual physical structure of the patient platform of the surgical robotic system 203, on which the patient can be secured during surgery.

[0046] In some implementations, the instrument-to-arm mapping model 1200 may also display instrument data associated with an instrument 212A or 212B attached to each robotic arm 210. The instrument data can be represented in the instrument-to-arm mapping model 1200 as follows: Figure 2AThe instrument indicators 1203A-1203D (or icons) are shown. Each instrument indicator in 1203A-1203D may be shown to be associated with a specific robotic arm 1230A-1230D. For example, instrument indicators 1203A-1203D may be located near robotic arms 1230A-1230D, to which instrument 212A or 212B described by instrument indicators 1203A-1203D is coupled. Instrument indicators 1203A-1203D may also, or otherwise, indicate that an instrument 212A or 212B described by instrument indicators 1203A-1203D is coupled to robotic arms 1230A-1230D via, for example, a call line. The call line can be a line connecting instrument indicators 1203A-1203D to robot arms 1230A-1230D, indicating that the instrument 212A or 212B described by instrument indicators 1203A-1203D is connected to the robot arm 210 represented by robot arms 1230A-1230D. In one embodiment, even if the corresponding robot arm 1230A-1230D changes its position within the robot arm 1230A-1230D, the instrument indicators 1203A-1203D can remain stationary (i.e., at the same position within the instrument-to-arm mapping model 1200). In this case, the call line can be extended or adjusted to ensure that the call line connects the dynamically changing positions of the fixed instrument indicators 1203A-1203D and the robot arms 1230A-1230D.

[0047] like Figure 2AAs shown, each instrument indicator 1203A-1203D may include data, icons, images (e.g., camera images or pre-stored digitally rendered models), text, identifiers, and / or other data used to describe and identify a particular instrument 212A, 212B. Instrument indicators 1203A-1203D may include an identifier of the robotic arm 210 to which the described instrument 212A, 212B is attached, an identifier of instrument 212A or 212B, an image of instrument 212A or 212B, and / or any other data related to instrument 212A, 212B, or the robotic arm 210 to which instrument 212A or 212B is attached. The identifier of the robotic arm 210 may be represented as one or more alphanumeric values ​​identifying the robotic arm 210, and the identifier may also physically exist on the robotic arm 210 itself. The identifier of instrument 212A, 212B may be represented as text describing the name of instrument 212A or 212B. The image of instrument 212A or 212B can be a rendered graphical representation of instrument 212A or 212B, or an image of instrument 212A or 212B obtained from a camera. The image can be rendered in real time using a rendering application, or it can be obtained from a library loaded with rendered graphical representations of various instruments 212A, 212B. As disclosed herein, instrument 212A or 212B can be a medical device, camera device, gripper, welding tool, drilling tool, laser, sensor, fastening tool, or any other type of tool, device, or instrument that can be detachably attached to robotic arm 210.

[0048] Finally, one or more of the instrument indicators 1203A-1203D may also include HID indicators 1215A, 1215B, and 1215D, which can indicate the hand position and engagement state of each of the instruments 212A and 212B described by the respective instrument indicators 1203A-1203D. When the described instrument 212A or 212B is controlled by the left HID 226 or the right HID 228, the HID indicators 1215A-1215B and 1215D may be included only in the instrument indicators 1203A-1203B and 1203D. Figure 2A As shown, only instrument indicators 1203A-1203B and 1203D include HID indicators 1215A-1215B and 1215D, and this may be because the robotic arm 210 controlling instrument 212A or 212B described by instrument indicator 1203C may not necessarily be assigned to only one of HIDs 226 and 228. For example, when instrument 212A, 212B described by instrument indicator 1203C is a camera device, HIDs 226 and 228 may have to operate and move together in a single plane to control the movement of the camera device (i.e., a single HID 226, 228 may not be used to control the movement of the camera device).

[0049] In some implementations, HID indicators 1215A-1215B, 1215D may be close to or within the corresponding instrument indicators 1203A-1203B, 1203D. In such cases... Figure 2A In the example shown, HID indicators 1215A-1215B, 1215D are shown positioned within the corresponding instrument indicators 1203A-1203B, 1203D. However, in other embodiments, HID indicators 1215A-1215B, 1215D may be positioned outside the corresponding instrument indicators 1203A-1203B, 1203D, but close to, overlapping with, or in contact with the edges of the corresponding instrument indicators 1203A-1203B, 1203D. In one embodiment, HID indicators 1215A-1215B, 1215D may indicate whether the described instruments 212A, 212B are configured to be controlled by the left HID 226 or the right HID 228. For example, HID indicators 1215A-1215B, 1215D may include text indicating whether devices 212A, 212B are configured to be controlled by the left HID 226 (indicated by the text "L") or by the right HID 228 (indicated by the text "R"). For example, additional icons or graphic representations of hands may also be positioned within HID indicators 1215A-1215B, 1215D to easily signal to the user the data type indicated by HID indicators 1215A-1215B, 1215D. In one example, if device 212A is actively controlled by the left HID 226 and device 212B is actively controlled by the right HID 228, then HID indicator 1215A of device 212A may display an icon of the left hand, and HID indicator 1215B of device 212B may display an icon of the right hand.

[0050] In some cases, HIDs 226 and 228 can be programmed to control multiple robotic arms 210, and thus multiple instruments 212A and 212B, but can only engage with one instrument 212A, 212B and / or robotic arm 210 at a time. In this way, HIDs 226 and 228 can actively control one active instrument 212A or 212B, but can be configured to control one or more other non-active instruments 212A or 212B. HIDs 226 and 228 can switch between active and non-active instruments 212A or 212B based on user input received at a user input device on the console 240 (e.g., a pedal at the foot actuator assembly).

[0051] To this end, HID indicators 1215A-1215B, 1215D can indicate whether devices 212A, 212B are actively engaged with or not actively engaged with HID 226, 228, but are still configured to be controlled by specific HID 226, 228. HID indicators 1215A-1215B, 1215D may include, for example, a shaded or colored border around the outer edge of the HID indicators 1215A-1215B, 1215D, which can indicate whether devices 212A, 212B are actively engaged. For example, when HID 226, 228 are actively engaged with devices 212A, 212B, the border can be activated or highlighted in a specific color (e.g., blue). When HID 226, 228 are disengaged from devices 212A, 212B, the border can be deactivated or darkened to a specific color (e.g., dark gray). In one example, if device 212 is currently inactive but can be controlled by left HID 226, then HID indicators 1215A-1215B, 1215D may include the text "L" with a circular arrow around the text to indicate that device 212 can be used for control exchange by left HID 226. Similarly, if device 212 is currently inactive but can be controlled by right HID 228, then HID indicators 1215A-1215B, 1215D may include the text "R" with a circular arrow around the text to indicate that device 212 can be used for control exchange by right HID 226.

[0052] It should be understood that instrument indicators 1203A-1203D may include other information, images, and / or icons that are not necessarily shown or described herein. Furthermore, a simplified version of the instrument-to-arm mapping model 1200 may be available. For example, a simplified version of the instrument-to-arm mapping model 1200 may include only a virtual representation of each instrument 212 to indicate the instrument type, without text describing the name of the instrument 212. Although only four robotic arms 1230A-1230D and instrument indicators 1203A-1203D are shown in the instrument-to-arm mapping model 1200, any number of robotic arms 1230A-1230D and instrument indicators 1203A-1203D may be included in the instrument-to-arm mapping model 1200, depending on the number of robotic arms 210 deployed by the surgical robotic system.

[0053] In some implementations, device indicators 1203A-1203D can be selected by the user to trigger the display of one or more adjustable settings corresponding to the device 212 to be displayed on display device 242. For example, each device indicator 1203A-1203D can be an icon that a healthcare provider can select using an input device at console 240 to open a menu related to the settings of devices 212A, 212B. For example, the device-to-arm mapping model 1200 can be displayed on touchscreen 232, and the healthcare provider can select device indicators 1203A-1203D via the touchscreen interface of touchscreen 232. By selecting device indicators 1203A-1203D, the healthcare provider can attempt to adjust the settings of the relevant devices 212A, 212B.

[0054] Turn now Figure 2B The diagram illustrates a setup menu 1250 for instruments 212A, 212B according to various embodiments of the present disclosure. The setup menu 1250 can be displayed after a selection of instrument indicators 1203A-1203D is received at the console 240. Figure 2B In the example shown, the settings menu 1250 can be overlaid on the instrument-to-arm mapping model 1200. In another embodiment, the settings menu 1250 can be displayed as a separate window from the instrument-to-arm mapping model 1200.

[0055] The setup menu 1250 may include identification data 1252, which may include identifiers of the robotic arms 210 to which devices 212A and 212B are attached. Identification data 1252 may also include identifiers for devices 212A and 212B. Identification data 1252 may be in the form of icons, images, text, or other types of indicators. The setup menu 1250 may also include hand assignment data 1251, which indicates which hand of the user is available to control devices 212A and 212B, or which HIDs 226 and 228 are configured to control devices 212A and 212B. Hand assignment data 1251 may be text indicating hand assignment for devices 212A and 212B. For example, hand assignment data 1251 may include the text "left," which may indicate that the user's left hand is available to control device 212A, and / or that the left HID 226 is configured to control device 212A.

[0056] In one embodiment, the settings menu 1250 may further include icons 1253 depicting hand assignments for devices 212A, 212B. In one embodiment, icon 1253 may also indicate finger placement for the assigned hand at specific HIDs 226, 228. In one embodiment, icon 1253 may depict optimal finger positioning for HIDs 226, 228 to the user. For example, icon 1253 may show the left hand positioned around the gripper, which is digitally represented as the left HID 226. Icon 1253 may be rendered in real-time using a rendering application, or it may be obtained from a library loaded with graphical representations of different types of HIDs 226, 228 and user interactions with HIDs 226, 228.

[0057] In one implementation, the settings menu 1250 may also include an edit icon 1256. The healthcare provider may select the edit icon 1256 to further adjust the settings displayed in the settings menu 1250. For example, in response to receiving a selection of the edit icon 1256, another more detailed settings menu may be displayed, where hand assignment settings or other settings associated with the device 212A, 212B or the corresponding robotic arm 210 can be adjusted. It should be understood that different types of devices 212A, 212B may be associated with different types of settings menus 1250.

[0058] See now Figure 2C Another example of a settings menu 1275 according to various embodiments of the present disclosure is shown. Settings menu 1275 can be displayed after a selection of instrument indicators 1203A-1203D is received at console 240. Figure 2C In the example shown, the settings menu 1275 can be overlaid on the instrument-to-arm mapping model 1200. In another embodiment, the settings menu 1275 can be displayed as a separate window from the instrument-to-arm mapping model 1200.

[0059] The settings menu 1275 may include identification data 1252, which may include, for example, identifiers of the robot arm 210 to which devices 212A and 212B are attached. Identification data 1252 may also include identifiers for devices 212A and 212B. For example, the identifiers for devices 212A and 212B may include text defining the names of devices 212A and 212B. Identification data 1252 may also include additional settings for devices 212A and 212B, such as angle and direction icons indicating the angles and directions of devices 212A and 212B.

[0060] The settings menu 1275 may also include one or more settings windows 1280A, 1280B, each corresponding to a different setting associated with the instruments 212A, 212B described in the settings menu 1275. For example, when instruments 212A, 212B are camera devices, settings window 1280A may indicate settings related to lights on the camera device, and settings window 1280B may indicate image settings for the camera device. Although only two settings windows 1280A, 1280B are shown in the settings menu 1275, it should be understood that the settings menu 1275 may include any number of settings windows 1280A, 1280B, each corresponding to a different setting for instruments 212A, 212B.

[0061] Within setting windows 1280A and 1280B, one or more user interface elements 1285A and 1285B may exist. User interface elements 1285A and 1285B may indicate the current setting related to the setting indicated in the respective setting window 1280A or 1280B. For example, setting window 1280A related to a light on a camera device may include multiple settings, such as settings for turning the light on / off, settings for adjusting the brightness of the light, etc. User interface element 1285A may correspond to each of these settings and can be interacted with by a user via an input device (e.g., touchscreen 232) on console 240 to adjust the corresponding setting. For example, interactive elements may be toggle buttons, sliders, checkboxes, radio buttons, labels, icons, drag-and-drop elements, navigation bars, etc. User interface elements 1285A and 1285B may also include icons or text indicating the current setting.

[0062] See now Figure 2D Examples of setup windows 1280C-1280D displayed in setup menu 1275 according to various embodiments of the present disclosure are shown. Setup windows 1280C-1280D can be displayed after a selection is received at console 240 for an instrument indicator 1203A-1203D associated with the endoscope device or camera. Setup windows 1280C-1280D can each display text, images, icons, and / or other user interface elements, each of which can be used to display and / or adjust settings of the endoscope device attached to the robotic arm 210.

[0063] like Figure 2DAs shown, the example setup window 1280C includes text 1291 describing the type of setup indicated in the setup window 1280C. The setup window 1280C may also include a visual representation 1292 of the endoscope device and corresponding text 1293 indicating the angle of the endoscope device. The setup window 1280C may also include a user interface element 1294 that, when selected or interacted with in a certain way, can adjust the orientation of the endoscope device attached to the robotic arm 210.

[0064] The graphic representation 1292 of the endoscopic device can be an image or rendered icon representing the endoscopic device, which can depict the angle and orientation of the endoscopic device. In, for example... Figure 2D In the example shown, graphic representation 1292 depicts the endoscope device as tilted downwards at approximately 30°, text 1293 describes the endoscope device as tilted at 30°, and user interface element 1294 indicates that the endoscope device is tilted downwards. As another illustrative example, when the endoscope device attached to the robotic arm 210 is angled at approximately 30°, graphic representation 1292 may depict the endoscope device as angled at approximately 30°, text 1293 may describe the endoscope device as angled at 30°, and user interface element 1294 may indicate that the endoscope device is angled.

[0065] The graphic representation 1292 of the endoscope device can be varied based on the type of endoscope device attached to the robotic arm 210. Similarly, the angle of the endoscope device depicted in graphic representation 1292 can be varied based on the actual angle of the endoscope device. The orientation of the endoscope device depicted in graphic representation 1292 can also be varied based on the actual orientation of the endoscope device. Figure 2D In the example shown, user interface element 1294 is a toggle user interface element where a user can select an "Up" or "Down" button on the user interface element to adjust the orientation of the endoscope device to the selected up / down orientation. When the orientation of the endoscope device is adjusted based on this user input, the orientation of the endoscope device depicted in the graphical representation 1292 can also be updated accordingly. It should be understood that user interface element 1294 can be any of the following: Figure 2D Any type of user interface element other than the toggled user interface element shown.

[0066] The similarity between the setup menu 1280D and the setup window 1280C lies in that the setup window 1280D includes text 1291 describing the type of setup indicated in the setup window 1280D, a graphical representation 1292 of the endoscope device, and text 1293 indicating the angle of the endoscope device. Specifically, the setup window 1280D depicts the setup of a zero-degree endoscope device, which may have a straight distal end without an angle. For this purpose, the graphical representation 1292 of the endoscope device in the setup menu 1280D is depicted as having a straight distal end (i.e., 0°). In this way, the graphical representation 1292 of the endoscope device can be changed based on the actual characteristics and setup of the endoscope device attached to the robotic arm 210 and possibly inside the patient.

[0067] E. Stadium View Model Turn now Figure 3 The image shows a stadium view model 1300 of a robotic system according to various embodiments of the present disclosure. Although... Figure 3 The robotic system shown is surgical robotic system 203, but it should be understood that stadium view model 1300 can be generated for other types of robotic systems across a wide range of industries. As described above, stadium view model 1300 provides a more holistic view of the patient, patient platform, and robotic arm 210. For example, stadium view model 1300 can be a graphical representation or rendering of an environment that includes not only the entire tableside but also the patient and / or any other external individuals, staff, or components in the operating room.

[0068] Examples of stadium view models 1300 include graphical representations of the patient, patient platform, robotic arm 210, patient table base, and / or various other structural aspects of the surgical robotic system 203. The stadium view model 1300 may also include different view icons 1310A-1310D, each corresponding to a different view (or perspective) of the stadium view model 1300 that can be displayed at console 240. Different views of the stadium view model 1300 can be displayed at console 240 when a healthcare provider selects a different view icon 1310A-1310D. Each view can depict the environment surrounding the surgical robotic system from different perspectives (e.g., bird's-eye view, high-perspective view, low-perspective view, side view, etc.).

[0069] Figure 3A first view of a stadium view model 1300 according to various embodiments of the present disclosure is shown. The first view may be a first side view from a high angle, depicting a graphical representation of the entire patient, all robotic arms 210, the patient platform, and the system base. The first view of the stadium view model 1300 may be displayed together with view icons 1310A-1310D, wherein one of the view icons (e.g., view icon 1310B) corresponds to the first view. In one embodiment, a healthcare provider may select view icon 1310B by providing user input to a user input device on console 240 (e.g., a touchscreen interface on touchscreen 232). This selection then causes the first view of the stadium view model 1300 to be displayed at console 240. After selection, view icon 1310B may be highlighted with a different background color (e.g., blue) to indicate that the displayed viewpoint corresponds to the first view of the stadium view model 1300.

[0070] In one embodiment, the stadium view model 1300 may include a graphical representation (or rendering) of each robotic arm 210 of the system, and in some embodiments, it may include a graphical representation (or rendering) of each of the joints and / or links of each robotic arm 210 of the system. In one embodiment, the stadium view model 1300 may indicate the current position and orientation of each robotic arm 210, which may be updated as the position and orientation of each robotic arm 210 moves during setup or surgery. In this way, the stadium view model 1300 may depict the current position and orientation of the robotic arms 210 as deployed at the side of the table.

[0071] like Figure 3 As shown, the stadium view model 1300 includes a graphical representation of four robotic arms 1303A, 1303B, 1303C, and 1303D (also referred to herein as "robotic arms 1303A, 1303B, 1303C, and 1303D"). The graphical representation of these four robotic arms 1303A-1303D can be rendered to represent the actual physical structure of the four robotic arms 210 deployed at the tableside of the surgical robotic system 203. The graphical representation of these four robotic arms 1303A-1303D can be rendered in real-time using a rendering application, or it can be obtained from a library that can load graphical representations of robotic arms 1303A-1303D.

[0072] Robotic arms 1303A-1303D can also depict one or more joints and / or links positioned along the corresponding stage side of the robotic arm 210 (e.g., including their rendering). Graphical representations of the joints, links, and other components of the robotic arm 210 can be rendered in real time using a rendering application, or can be obtained from a library that can load graphical representations of joints, links, and other components.

[0073] The stadium view model 1300 may also include a graphical representation of the patient platform 1306 and pedestal 1313 of the surgical robot system 203. Based on an embodiment of the surgical robot system 203, the pedestal 1313 may include different components of the surgical robot system 203. It should be understood that the stadium view model 1300 may include a graphical representation of any of the components of the surgical robot system. The graphical representations of the patient platform 1306, pedestal 1313, and other components may be rendered in real time based on data received at the console 240, or may be loaded from a library that stores rendered objects corresponding to various objects or people in the operating room.

[0074] The stadium view model 1300 may also include a graphical representation of patient 1315. In some embodiments, for example, when the patient is actually positioned on a patient platform, the graphical representation of patient 1315 may be a rendering representing the patient's actual physical dimensions, shape, and anatomical structure. The rendering may be generated based on data received from various sensors positioned on the patient platform. For example, the graphical representation of patient 1315 and the graphical representation of patient platform 1306 may depict the patient's actual height and width relative to the height and width of the patient platform. In other embodiments, the graphical representation of patient 1315 may be a default rendering of a general human anatomy displayed as a graphical representation positioned on patient platform 1306. This default rendering of the human body may be stored in a library that may be preloaded with rendering objects corresponding to various objects or people.

[0075] In one implementation, the stadium view model 1300 may not depict any instrument 212A, 212B information. Instead, the stadium view model 1300 may focus on the location and status of various components of the surgical robotic system 203, the patient, and other personnel / objects in the operating room.

[0076] Each of the view icons 1310A-1310D may correspond to a different view of the stadium view model 1300. For example, icon 1310A may correspond to a second view of the stadium view model 1300, which may be a bird's-eye view of the environment surrounding the surgical robot system 203. The second view may display the components of the surgical robot system 203 (e.g., a graphical representation of the robotic arms 1303A-1303D, the patient platform 1306, and the base 1313) from a top-down perspective. The second view may also provide a graphical representation of the patient 1315 from a top-down perspective. In one embodiment, a healthcare provider may select view icon 1310A by providing user input to a user input device of console 240 (e.g., a touchscreen interface of touchscreen 232) so that the second view of the stadium view model 1300 is displayed at console 240. After selecting view icon 1310A, view icon 1310A may be highlighted with a different background color (e.g., blue).

[0077] For example, icon 1310C may correspond to a third view of stadium view model 1300, which may be from a lower perspective (e.g., a perspective from near the feet of patient platform 1306 or patient 1315). The third view may similarly display components of surgical robot system 203 from a lower perspective (e.g., graphical representations of robotic arms 1303A-1303D, patient platform 1306, and base 1313). The third view may also provide a graphical representation of patient 1315 from a lower perspective. In one embodiment, a healthcare provider may select view icon 1310C by providing user input to a user input device of console 240 (e.g., a touchscreen interface of touchscreen 232) so that the third view of stadium view model 1300 is displayed at console 240. After selecting view icon 1310C, it may be highlighted with a different background color (e.g., blue).

[0078] For example, icon 1310D can correspond to the fourth view of stadium view model 1300, which can be from a higher perspective (e.g., similar to...). Figure 3The fourth view (shown as a view, but from a different side of the patient) can similarly display the components of the surgical robot system 203 from a higher perspective (e.g., graphical representations of the robotic arms 1303A-1303D, patient platform 1306, and base 1313). The fourth view can also provide a graphical representation of the patient 1315 from a higher perspective. In one embodiment, the healthcare provider can select view icon 1310D by providing user input to a user input device on console 240 (e.g., a touchscreen interface on touchscreen 232) so that the fourth view of the stadium view model 1300 is displayed at console 240. After selecting view icon 1310D, view icon 1310D can be highlighted with a different background color (e.g., blue).

[0079] Therefore, the stadium view model 1300 includes multiple predefined views or perspectives of the environment surrounding the surgical robotic system 203, allowing medical providers to access and switch between predefined views as needed during surgery using view icons 1310A-1310D without leaving the immersion. While only four views of the stadium view model 1300 are discussed herein, it should be understood that the stadium view model 1300 may include any number of views depicting the surgical robotic system 203 (or any robotic system) from different perspectives.

[0080] In another embodiment, a single stadium view model 1300 may be presented, wherein the stadium view model 1300 is displayed together with, for example, a slider user interface element. The slider user interface element can be dragged left and right and / or up and down, which can correspondingly rotate the stadium view model 1300 left and right, up and down to present different views of the stadium view model 1300. As described above, regardless of the view being displayed, the stadium view model 1300 can accurately depict the current position and orientation of different objects and people in the operating room, including the patient, each robotic arm in the robotic arm 210, the patient platform, etc.

[0081] F. Target Screen Model Figure 4 A target screen model 1400 displayed at console 240 of a surgical robotic system 203 is shown according to various embodiments of the present disclosure. However, it should be understood that the target screen model 1400 can be generated for other types of robotic systems across a wide variety of industries. In one embodiment, the target screen model 1400 may include all components of the instrument-to-arm mapping model 1200, but may also include a graphical representation of the patient 1415. In such... Figure 4In the illustrated embodiment, the target screen model 1400 may not depict the entire patient, but rather only a graphical representation of the patient and the healthcare provider-related portion of the surgical robotic system (e.g., the portion of the patient for which only the robotic arm 210 can perform surgery). In other embodiments, the target screen model 1400 may depict the entire patient 1415.

[0082] The target screen model 1400 may include a graphical representation of the robotic arms 1403A-1403D, which may represent the actual position of the robotic arm 210 on the tableside. In some embodiments, the target screen model 1400 may depict the distal ends of the robotic arms 1403A-1403D, which may include unlocked and movable distal links and joints.

[0083] The target screen model 1400 may also include instrument indicators 1402A-1402D (similar to instrument indicators 1203A-1203D) which describe instruments 212A, 212B connected to the robotic arm 210 represented by robotic arms 1403A-1403D. The target screen model 1400 may also include HID indicators 1404A, 1404C, 1404D (similar to HID indicators 1215A-1215B, 1215D) which describe the hand bias or hand assignment between the robotic arms 1403A-1403D / instruments 212A, 212B described by instrument indicators 1402A-1402D and HID 226, 228.

[0084] The target screen model 1400 can be the default screen displayed on the display device 242 or touchscreen 232 of the console 240. For example, unless the healthcare provider interacts with the touchscreen 232 to change the display to a different screen or window (e.g., for displaying...). Figure 2A-2D Instrument-to-arm mapping model 1200 or Figure 3 (Stadium view model 1300), otherwise the target screen model 1400 can be set to always be displayed at touch screen 232 by default during the setup and execution of the operation.

[0085] G. State Model Turn now Figures 5A-5BVarious examples of state models 1500A-1500B displayed at the console 240 of a surgical robotic system 203 according to various embodiments of the present disclosure are shown. However, it should be understood that state models 1500A-1500B can be generated for other types of robotic systems across a wide variety of industries. State models 1500A-1500B may indicate the current position, orientation, and state of one or more joints and / or links along each robotic arm 210 deployed on the tableside of the surgical robotic system 203. In one embodiment, state models 1500A-1500B may be represented in a reduced-scale manner, where the patient, robotic arm 210, patient platform, and base are represented as a whole, similar to stadium view model 1300. In another embodiment, state models 1500A-1500B may be represented in an enlarged-scale manner, where only the unlocked portion of the robotic arm 210 is depicted and emphasized.

[0086] See now Figure 5A A first view 1501A depicts a state model 1500A and docking status indicators 1510A-1510D according to various embodiments of the present disclosure. State model 1500A includes a graphical rendering of robotic arms 1503A-1503D reflecting the current position and orientation of robotic arm 210 relative to the tableside. State model 1500A also includes a graphical representation of a patient platform 1506, representing the patient platform on the tableside. State model 1500A also includes a graphical representation of a patient 1515, depicting the patient in its entirety (although in some embodiments, depicting the patient 1515 in its entirety is not required). State model 1500A may include features not shown in... Figure 5A The illustrated state model 1500A shows a graphical representation of other people or objects in the operating room. Graphical representations of the robotic arms 1503A-1503D, patient platform 1506, patient 1515, and other objects / people can be rendered in real time using a rendering application, or can be obtained from a library that can load these graphical representations.

[0087] In one embodiment, the graphical representation of the robotic arms 1503A-1503D may further include a graphical representation of each joint and / or link along the robotic arms 1503A-1503D. Specifically, referring to the graphical representation of the robotic arm 1503D, the robotic arm 1503D includes graphical representations of a plurality of links 1520. The graphical representations of the links 1520 can be rendered individually to represent the physical structure of each link on the stage side.

[0088] In some implementations, the position and orientation of each robot arm in the state model 1500A 1503A-1503D can reflect the current position and / or orientation of each robot arm in the corresponding robot arm 210 at the stage side. Similarly, the position and / or orientation of each link in the links 1520 within the robot arms 1503A-1503D can reflect the current position and / or orientation of each link in the links 1520 along the robot arm 210 at the stage side.

[0089] In one implementation, the state model 1500A can reflect the state of each link along the robotic arm 210 by visual elements in a graphical representation of each link 1520 along each of the robotic arms 1503A-1503D, based on the state changes of the links. The state of each link 1520 can indicate whether a particular link on the robotic arm 210 is unlocked and allowed to move or locked and prohibited from moving. When a link is unlocked and allowed to move, the position and orientation of the link may be important to the medical provider operating the robotic arm 210. This may be because movement of the robotic arm 210 may adversely affect system setup or the execution of surgery. Conversely, when a link is locked and prohibited from moving, the position and orientation of the link may be less important to the medical provider operating the robotic arm 210. This may be because a locked robotic arm 210 may have little effect on system setup or the execution of surgery. Therefore, providing clear indication of the state of the links 1520 along the deployed robotic arm 210 can be significantly helpful to the medical provider.

[0090] State model 1500A illustrates the preoperative position of robotic arm 210. In the preoperative position, robotic arm 210 is deployed above the patient platform 1506, but relatively far from the patient platform (e.g., not positioned above the patient's anatomy for a specific surgery). In the preoperative position, all links and joints along robotic arm 210 can be unlocked and moved, as robotic arm 210 is in the process of being set up for surgery at this stage.

[0091] By changing the visual elements of each graphic representation in the graphic representation of each link 1520 of each robot arm 1503A-1503D, the state of each link in the links along the robot arm 210 can be indicated in the state model 1500A. For example, when the corresponding link at the stage side is unlocked and movable, the graphic representation of each of the links 1520 can be set to a first color, and when the corresponding link at the stage side is locked and otherwise prevented from moving, it can be set to a second color. As another example, when the corresponding link at the stage side is unlocked and movable, the graphic representation of each of the links 1520 can be set to a solid color, and when the corresponding link at the stage side is locked and otherwise prevented from moving, it can be set to a gradient color or a grayscale color. In either case, visual elements indicating movable links are brighter or easier to see than visual elements indicating immovable links, thereby highlighting relevant movable links that may affect the setup or execution of the surgery, while dimming links that may be irrelevant to the medical provider during the setup or execution of the surgery. In this way, the state model 1500A selectively highlights certain aspects of the surgical robot system 203 for the medical provider to focus on, while diminishing other irrelevant aspects of the surgical robot system 203 that would otherwise distract the medical provider during the setup and execution of the surgery.

[0092] In addition, the state model 1500A may further include docking state indicators 1510A-1510D. The docking state indicators 1510A-1510D can convey the state of the entire robot arm 210 at the stage side. When one or more predefined links on the robot arm 210 are locked in place, the robot arm 210 can, for example, dock with a predefined position. The docking state indicators 1510A-1510D can be represented in various different ways as associated with the robot arm 210. For example, the graphical representation of the robot arms 1503A-1503D can be positioned as the graphical representation closest to the robot arms 1503A-1503D. For example, the docking state indicators 1510A-1510D can be positioned from left to right or from right to left to correspond to the graphical representation of the robot arms 1503A-1503D positioned in a specific order from left to right or right to left. Figure 5AAs shown, the robotic arm 1503A is closest to the docking status indicator 1510B, and thus the docking status indicator 1510B describes whether the table-side robotic arm 210 represented by the robotic arm 1503A is docked, for example, to a predefined position. Similarly, the robotic arm 1503B is closest to the docking status indicator 1510A, and thus the docking status indicator 1510A describes whether the table-side robotic arm 210 represented by the robotic arm 1503B is docked. The robotic arm 1503C is closest to the docking status indicator 1510C, and thus the docking status indicator 1510C describes whether the table-side robotic arm 210 represented by the robotic arm 1503C is docked. Finally, the robotic arm 1503D is closest to the docking status indicator 1510D, and thus the docking status indicator 1510D describes whether the table-side robotic arm 210 represented by the robotic arm 1503D is docked.

[0093] The docking status indicators 1510A - 1510D may include identifiers of the corresponding robotic arms 210. The docking status indicators 1510A - 1510D may also include text indicating whether the corresponding table-side robotic arm 210 is docked. For example, the text may state "Docked" to indicate that the robotic arm 210 is docked, or state "Not Docked" to indicate that the robotic arm 210 is not docked.

[0094] Now referring to Figure 5B , a second view 1501B depicting the status model 1500B and the docking status indicators 1510A - 1510D in accordance with various embodiments of the present disclosure is shown. Different from the first view 1501A of the status model 1500A as Figure 5A shown, the second view 1501B of the status model 1500B as Figure 5B shown is magnified and does not include a graphical representation of the patient. Additionally, the positions of the robotic arms 1503A - 1503D are different (e.g., surgical positions). Such positions of the corresponding physical robotic arms 210 may be used, for example, for upper abdominal surgery performed on a patient.

[0095] In this case, the linkages 1520 on the robotic arms 1503A - 1503D may be set to maintain a first color (e.g., white), which may be a highlighted color indicating that the linkages on the corresponding table-side robotic arm 210 are unlocked and movable. Similarly, the docking status indicators 1510A - 1510D may also indicate that the table-side robotic arms 210 corresponding to the robotic arms 1503A - 1503D are all still not docked.

[0096] In some implementations, state models 1500A-1500B can be updated to reflect that some of the robotic arms 210 have been docked, while others remain un-docked. For example, state models 1500A-1500B can be updated to reflect that the table-side robotic arm 210 corresponding to the graphical representations of robotic arms 1503A, 1503C, and 1503D has been docked, while the table-side robotic arm 210 corresponding to the graphical representation of robotic arm 1503B has not been docked. This change in state can be reflected in state models 1500A-1500B in several different ways. For example, docking status indicators 1510A, 1510C, and 1510D can be updated to include the text "Docked" to indicate that the robotic arm 210 corresponding to the respective docking status indicator 1510A, 1510C, or 1510D has been docked. Furthermore, the visual elements of some links 1520 on robotic arms 1503A, 1503C, and 1503D can be adjusted to be darker (e.g., changed to gray, black, or shadow) in terms of color or shading. This change in the visual elements of link 1520 indicates that the corresponding tableside link 1520 of robotic arm 210 is locked and cannot be moved. Meanwhile, some links 1520 on other links 1503A can be highlighted in a brighter color (e.g., white) in state models 1500A-1500B. The brighter color of the link indicates that the corresponding tableside link of robotic arm 210 is unlocked and allowed to move (thus, important for the healthcare provider in terms of focus).

[0097] In some implementations, state models 1500A-1500B can be updated to reflect that some of the robotic arms 210 have been connected to the instrument 212, while others may not yet be connected. For example, state models 1500A-1500B can be updated to reflect that three of the table-side robotic arms 210 corresponding to the graphical representations of robotic arms 1503B, 1503C, and 1503D have been connected to instruments 212A and 212B, while the table-side robotic arm 210 corresponding to the graphical representation of robotic arm 1503A has been docked. These changes in the status and connected instruments 212A and 212B can be reflected in state models 1500A-1500B in several different ways. For example, when instruments 212A and 212B are connected to the corresponding robotic arms 210, the docking status indicators 1510B, 1510C, and 1510D may have been updated to include instrument indicators, such as those referenced above. Figure 2A-2DThe described instrument indicators are 1203A-1203D. Instrument indicators may include icons, images (e.g., camera images or pre-stored digitally rendered models), text, identifiers, and / or other data used to describe and identify specific instruments 212A, 212B connected to the corresponding stage-side robot arm 210. Simultaneously, the docking status indicator 1510A of the robot arm 1503A may be updated to include the text "Docked" to indicate that the corresponding stage-side robot arm 210 has been docked (however, instruments 212A, 212B may not yet be connected to the corresponding stage-side robot arm 210).

[0098] H. Display devices and touchscreens at the console Figure 6 Figure 1600 illustrates various embodiments of the display device-to-arm mapping model 1200, stadium view model 1300, target screen model 1400, and status models 1500A, 1500B, or modified versions of models 1200, 1300, 1400, 1500A, and / or 1500B, according to the present disclosure. As described above, the console 240 may include multiple displays. For example, one display may be located in a display device 242, which may be implemented as a head-mounted device. Another display may be located at a touchscreen 232, which may be positioned, for example, on the handle of the console 240.

[0099] In one implementation, one or more of the following models can be displayed on display device 242: instrument-to-arm mapping model 1200, stadium view model 1300, target screen model 1400, and status models 1500A and 1500B. Figure 6 As shown, one or more of the models 1200, 1300, 1400, 1500A, and 1500B disclosed herein can be displayed on display device 242 in picture-in-picture (PIP) format, wherein models 1200, 1300, 1400, 1500A, and 1500B are located in a small frame 1616, which is positioned at a corner 1618 of the display or image 1103. In this case, models 1200, 1300, 1400, 1500A, and 1500B can be rendered and overlaid on top of image 1103. Although Figure 6 The corner 1618 is shown in the lower right corner of image 1103, but models 1200, 1300, 1400, 1500A, and 1500B can be displayed in PIP format anywhere in image 1103 or in a corner.

[0100] In one implementation, the modified model 1617 can be displayed on display device 242 in PIP format. The modified model 1617 can be a simpler or coarser version of the instrument-to-arm mapping model 1200, stadium view model 1300, target screen model 1400, and / or status models 1500A, 1500B. Compared to other models 1200, 1300, 1400, 1500A and / or 1500B, the modified model 1617 displayed in PIP format can include less data (e.g., text, icons, images, etc.). Figure 6 In the example shown, the modified model 1617 may include a graphical representation 1610 of the robotic arm 210 and / or instruments 212A-212B. The modified model 1617 may include HID indicators 1613 depicting information about instruments 212 attached to the robotic arm 210. Each HID indicator 1613 may be adjacent to the graphical representation 610 of the robotic arm 210 and thus may represent information about instruments 212 attached to the robotic arm 210 represented by the adjacent graphical representation 610. HID indicators 1613 may indicate hand position, engagement state, and / or other data associated with instruments 212. Each HID indicator 1613 may include an icon, image, or text identifying the instrument 212 attached to the robotic arm 210. For example, when the represented instrument 212 is a endoscopic device or a camera, the HID indicator 1613 may include a camera icon. HID indicator 1613 may include an icon of either a left or right hand, depending on whether device 212 is actively controlled by left HID 226 or right HID 228. When device 212 is disengaged from HID 226, 228 but can be exchanged by the corresponding left HID 226 or right HID 228, HID indicator 1613 may include the text “L” or “H”, with a circular arrow surrounding the text. The outline of HID indicator 1613 may also indicate whether device 212 is actively engaged or disengaged from HID 226, 228 based on, for example, color or brightness. In some cases, modified model 1617 may also include a graphical representation of the patient, such that modified model 1617 depicts the position of each of device 212 and / or robotic arm 210 relative to the patient's location.

[0101] In some implementations, the modified model 1617 may depict specific perspective views or cropped areas of the stadium view model 1300 based on components of the surgical robotic system most relevant to the healthcare provider during the setup or execution of the surgery. For example, if only one robotic arm 210 is active with the instrument 212 and another robotic arm 210 actively operates the camera, the stadium view model 1300 may only show the two robotic arms 210 operating the camera, as well as the position and orientation of the individual instruments (e.g., the stadium view model 1300 may be cropped to exclude any rendering of inactive robotic arms 210 that would unnecessarily consume space at the display device 242).

[0102] When models 1200, 1300, 1400, 1500A, 1500B, and 1617 are displayed in PIP format, models 1200, 1300, 1400, 1500A, 1500B, and 1617 can be selectively displayed on display device 242 in response to user input received at a user input device on console 240. For example, console 240 may include multiple different user input devices (e.g., buttons, switches, touch-sensitive surfaces, gimbals, toggle switches, pedals, etc.) positioned, for example, on HID 226, 228, armrests, and / or foot actuator assemblies of console 240. A healthcare provider may provide user input or a combination of user input at one or more input devices on console 240 to trigger the display of models 1200, 1300, 1400, 1500A, 1500B, and 1617 in PIP format on display device 242.

[0103] For example, when accessing the position of various robotic arms 210 relative to the patient platform is desired, a healthcare provider may want to view stadium view model 1300 or modified model 1617, but the healthcare provider does not want to leave the immersion. In this case, the healthcare provider can provide one or more user inputs to console 240, which can trigger a PIP display of stadium view model 1300 or modified model 1617 at display device 242. In some cases, when user input is provided to console 240, models 1200, 1300, 1400, 1500, 1500A, 1500B, and 1617 may be displayed at display device 242 only in PIP format (i.e., the healthcare provider may need to repeatedly press the foot pedal at the foot actuator assembly and / or press the buttons at HID 226, 228 to make models 1200, 1300, 1400, 1500A, 1500B, and 1617 displayed in PIP format). In this scenario, when the healthcare provider stops providing user input, models 1200, 1300, 1400, 1500A, 1500B, and 1617 can interrupt the display at display device 242.

[0104] Similarly, when a healthcare provider wants to understand the instruments 212 attached to one or more robotic arms in robotic arm 210, they may want to view the instrument-to-arm mapping model 1200, but do not want to leave the immersive environment. In this case, the healthcare provider can provide one or more user inputs to console 240, which can trigger a PIP display of the instrument-to-arm mapping model 1200 at display device 242.

[0105] In one implementation, the target screen model 1400 may be the default screen displayed at the touchscreen 232 of the console 240. For example, the target screen model 1400 may be configured to always be displayed at the touchscreen 232 by default during the setup and execution of the procedure, unless the healthcare provider interacts with the touchscreen 232 (e.g., selects an icon on the touchscreen 232 interface) to display a specific model 1200, 1300, or 1500A, 1500B, 1617.

[0106] For example, during surgical setup, before the healthcare provider begins performing an incision or examination on the patient using the robot, the provider may need to set the initial position of the robotic arm 210 and instruct staff to attach specific instruments 212 or cameras to one or more of the robotic arms within the robotic arm 210. Thus, the healthcare provider can view, for example, status models 1500A, 1500B displayed on touchscreen 232 to set the positioning and locking / unlocking of each robotic arm in the robotic arm 210 in the immersive display device 242 before beginning surgery on the patient.

[0107] As another example, a medical provider may be performing surgery on a patient and immersing themselves in display device 242 while noticing that one of the instruments 212 has restricted movement, which can be shown in image 1103 displayed at display device 242. However, the cause of the restricted movement may be unlikely to be shown in image 1103, as very limited information is available at display device 242. To address this, the medical provider may temporarily withdraw from immersion (in some embodiments, locking the robotic arm 210 for safety purposes) and consult touchscreen 232 to diagnose the cause of this restricted movement. As mentioned above, touchscreen 232 may be set by default to display target screen model 1400. In some cases, target screen model 1400 may indicate that two robotic arms 210 are relatively close together and may collide. However, the medical provider may use one or more user input devices to provide one or more user inputs at console 240 to display stadium view model 1300 or a modified model 1617, which provides a zoomed-out view of the position and orientation of all deployed robotic arms 210. The healthcare provider can use the scaled-down stadium view model 1300, the modified model 1617, and / or the target screen model 1400 to rule out the cause of this restricted movement and can take appropriate action (i.e., correct the positioning of the robotic arm 210 by robot or manual correction, or request staff closer to the table to correct the positioning of the robotic arm 210).

[0108] In some cases, the robotic system may include a tower that is separate from the console 240 and the robotic arm 210. The tower may provide support for controls, electronics, fluid dynamics, optics, sensors, and / or power to the robotic arm 210 and / or the console 240. In some embodiments, the tower includes a display device. The display device at the tower may display one or more of the following: an instrument-to-arm mapping model 1200, a stadium view model 1300, a target screen model 1400, status models 1500A, 1500B, and / or a modified model 1617. The different models 1200, 1300, 1400, 1500A, 1500B, and / or 1617 may be displayed periodically throughout the procedure, for example, based on user input at the console or based on the stage of the procedure.

[0109] I. Exemplary Methods Figure 7 A flowchart illustrating method 1700 performed by a robotic system is provided. Specifically, method 1700 may be executed by a processor in console 240 or coupled to console 240.

[0110] At step 1703, method 1700 includes displaying an instrument-to-arm mapping model 1200 at a console 240 of the surgical robot system. This instrument-to-arm mapping model includes a graphical representation of the robotic arm 210 relative to the patient platform of the surgical robot system 203. The graphical representation of the robotic arm may be rendered as follows: Figure 2A Robotic arms 1230A-1230D, Figure 3 Robotic arms 1303A-1303D Figure 4 Robotic arms 1403A-1403D, or Figures 5A-5B The robotic arm 1503A-1503D.

[0111] At step 1706, method 1700 includes indicating, in the instrument-to-arm mapping model 1200, instrument data describing instruments 212A, 212B associated with a first robotic arm 210 in the robotic arm 210, wherein the first robotic arm 210 is coupled to instruments 212A, 212B. The instrument data can be displayed in the instrument-to-arm mapping model 1200 as... Figure 2A The instrument indicator shown is 1203A-1203D.

[0112] At step 1709, method 1700 includes displaying HID indicators 1215A-1215B, 1215D along with instrument data in the instrument-to-arm mapping model 1200. HID indicators 1215A-1215B, 1215D indicate whether a first HID 226 or a second HID 228 of the console 240 is configured to control instruments 212A, 212B coupled to the first robotic arm 210.

[0113] Figure 8 A flowchart illustrating method 1800 performed by surgical robot system 200 or 400 is provided. Specifically, method 1800 may be executed by a processor in console 240 or coupled to console 240.

[0114] At step 1803, method 1800 includes displaying an instrument-to-arm mapping model 1200 at a console 240 of the surgical robotic system. This instrument-to-arm mapping model includes a graphical representation of a plurality of robotic arms 210 of the surgical robotic system 203. The instrument-to-arm mapping model 1200 depicts the position of each of the robotic arms 210 relative to the patient platform 1234 of the surgical robotic system 203. The graphical representation of the robotic arms may be rendered as follows: Figure 2A Robotic arms 1230A-1230D, Figure 3 Robotic arms 1303A-1303D Figure 4 Robotic arms 1403A-1403D, or Figures 5A-5B The robotic arm 1503A-1503D.

[0115] At step 1806, method 1800 includes indicating, in instrument-to-arm mapping model 1200, instrument data describing instruments 212A, 212B associated with a first robotic arm 210 in robotic arm 210, wherein the first robotic arm 210 is coupled to instruments 212A, 212B. The instrument data can be displayed in instrument-to-arm mapping model 1200 as follows: Figure 2A The instrument indicator shown is 1203A-1203D.

[0116] J. Implementation Systems and Terminology .

[0117] Figure 9 This is a schematic diagram illustrating the electronic components of a surgical robot system according to some embodiments.

[0118] Surgical robotic systems, such as surgical robotic system 203, include one or more processors 380, which store information for performing any of the methods described herein (e.g., regarding...). Figure 1-9 The computer-readable storage medium 382 (e.g., a computer storage device such as random access memory, read-only memory, static random access memory, and non-volatile memory, as well as other storage devices such as hard disk drives, optical discs, magnetic tape recording, or any combination thereof) communicates with instructions for the described operations. One or more processors 380 also communicate with an input / output controller 384 (via a system bus or any suitable circuitry). The input / output controller 384 receives sensor data from one or more sensors 388-1, 388-2, etc., and relays the sensor data to one or more processors 380. The input / output controller 384 also receives instructions and / or data from one or more processors 380 and relays the instructions and / or data to one or more actuators, such as first motors 387-1 and 387-2, etc. In some embodiments, the input / output controller 384 is coupled to one or more actuator controllers 386 and provides instructions and / or data to at least a subset of the one or more actuator controllers 386, which in turn provides control signals to selected actuators. In some implementations, one or more actuator controllers 386 are integrated with an input / output controller 384, and the input / output controller 384 provides control signals directly to one or more motors 387-1 and 387-2, etc. (in the absence of a separate actuator controller). Although Figure 9It is shown that there is an actuator controller 386 (e.g., an actuator controller for the entire surgical robot system; in some embodiments, additional actuator controllers may be used (e.g., one actuator controller per actuator, etc.). In some embodiments, one or more processors 380 communicate with one or more displays 381 for displaying information as described herein.

[0119] The following examples illustrate various non-exhaustive ways in which the teachings herein can be combined or applied. It should be understood that the following examples are not intended to limit the scope of any claims that may be provided at any time in this patent application or a subsequent filing thereof. No disclaimer is intended. The following examples are provided merely for illustrative purposes. It is contemplated that the various teachings herein may be arranged and applied in a variety of other ways. It is also contemplated that some variations may omit certain features mentioned in the following examples. Therefore, none of the aspects or features mentioned below should be considered definitive unless otherwise expressly indicated by the inventor or a successor with an interest in the inventor at a later date. If any claim set forth in this patent application or a subsequent filing related to this patent application includes additional features beyond those mentioned below, such additional features should not be presumed to have been added for any reason related to patentability.

[0120] Example Combination 1: A surgical robotic system may include: a plurality of robotic arms, the plurality of robotic arms including a first robotic arm, wherein the first robotic arm is coupled to an instrument; and a console communicatively coupled to the robotic arms, the console including: a display device; and a processor coupled to the display device and configured to: display at the display device an instrument-to-arm mapping model that may include a graphical representation of the robotic arms, wherein the instrument-to-arm mapping model depicts the position of each of the robotic arms; and indicate in the instrument-to-arm mapping model instrument data associated with the first robotic arm describing the instrument.

[0121] Example Combination 2: The surgical robot system according to Example Combination 1, wherein, in order to indicate the instrument data associated with the first robotic arm, the display is further configured to display a call line that connects the instrument data to the first robotic arm.

[0122] Example Combination 3: A surgical robotic system according to any one of Example Combination 1 or Example Combination 2, wherein the graphical representation of the robotic arms may include a rendering of each of the robotic arms.

[0123] Example Combination 4: A surgical robotic system according to any one of Example Combinations 1 to 3, wherein the rendering of each robotic arm includes the rendering of one or more joints and links of the robotic arm.

[0124] Example Combination 5: A surgical robot system according to any one of Example Combinations 1 to 4, wherein the instrument data may include at least one of an identifier of the first robotic arm, an image depicting the instrument, text describing the instrument, an image indicating the orientation of the instrument, or text describing operations that can be performed by the instrument.

[0125] Example Combination 6: A surgical robot system according to any one of Example Combinations 1 to 5, wherein the instrument-to-arm mapping model may further include a tactile interface device (HID) indicator displayed together with the instrument data, wherein the HID indicator indicates the HID of the console engaged to control the instrument.

[0126] Example Combination 7: A surgical robot system according to any one of Example Combinations 1 to 6, wherein the instrument-to-arm mapping model may further include a haptic interface device (HID) indicator displayed together with the instrument data, wherein the HID indicator indicates whether the HID of the console is configured to control the instrument but is not currently engaged to control the instrument.

[0127] Example Combination 8: A surgical robot system according to any one of Example Combinations 1 to 7, wherein the robotic arm may further include a second robotic arm, wherein the second robotic arm is coupled to a camera, wherein the instrument data may include endoscope data, and wherein the range data may include at least one of the angle of the camera or the orientation of the camera.

[0128] Example combination 9: A surgical robot system according to any one of example combinations 1 to 8, wherein the processor is further configured to display an image of a patient at the display device, wherein the instrument-to-arm mapping model is superimposed on a portion of the image of the patient.

[0129] Example combination 10: A surgical robot system according to any one of example combinations 1 to 9, wherein the instrument data is indicated by icons displayed on the display device.

[0130] Example Combination 11: A surgical robot system according to any one of Example Combinations 1 to 10, wherein the display device is located in a head-mounted device of the console or on the armrest of the console.

[0131] Example Combination 12: A method performed by a surgical robotic system, the method comprising: displaying an instrument-to-arm mapping model at a console of the surgical robotic system, the instrument-to-arm mapping model comprising a graphical representation of a plurality of robotic arms of the surgical robotic system, wherein the instrument-to-arm mapping model depicts the position of each of the robotic arms relative to a patient platform of the surgical robotic system; and indicating instrument data describing an instrument and associated with a first robotic arm of the robotic arms, wherein the first robotic arm is coupled to the instrument, in the instrument-to-arm mapping model.

[0132] Example combination 13: The method according to example combination 12, wherein indicating the instrument data associated with the first robotic arm may include displaying a call line that connects the instrument data to the first robotic arm.

[0133] Example Combination 14: The method according to any one of Example Combination 12 or Example Combination 13, wherein the graphical representation of the robot arm may include rendering of each of the robot arms, and wherein the method may further include updating the rendered position of each of the robot arms based on the actual position of the robot arms.

[0134] Example Combination 15: The method according to any one of Example Combinations 12 to 14, wherein the instrument data may include at least one of an identifier of the first robotic arm, an image depicting the instrument, text describing the instrument, an image indicating the orientation of the instrument, or text describing operations that can be performed by the instrument, wherein the instrument-to-arm mapping model may further include a haptic interface device (HID) indicator displayed together with the instrument data, and wherein the HID indicator indicates an HID of the console engaged to control the instrument.

[0135] Example combination 16: The method according to any one of example combinations 12 to 15 may further include: receiving one or more user inputs received at one or more user input devices on the console by the processor of the console, and displaying the instrument-to-arm mapping model on the console in response to receiving the one or more user inputs.

[0136] Example combination 17: The method according to any one of example combinations 12 to 16, wherein the graphical representation of the robot arm depicts at least one of the positions, angles, or orientations of one or more joints and links of each robot arm.

[0137] Example combination 18: The method according to any one of example combinations 12 to 17, wherein the instrument data is indicated by an icon displayed at the console, wherein the method may further include: receiving a selection of the icon by a processor of the console; and in response to receiving the selection of the icon, displaying a settings menu at the console to adjust one or more settings of the instrument.

[0138] Example combination 19: The method according to any one of example combinations 12 to 18, wherein the instrument-to-arm mapping model is displayed at a display device on the console, wherein the display device is located in a head-mounted device of the console or on the armrest of the console.

[0139] Example Combination 20: A non-transitory computer-readable medium storing instructions that, when executed by a processor of a surgical robotic system that may include multiple robotic arms, cause the processor to: display an instrument-to-arm mapping model at a console of the surgical robotic system, the instrument-to-arm mapping model including a graphical representation of the robotic arms relative to a patient platform of the surgical robotic system; in the instrument-to-arm mapping model, indicate instrument data describing an instrument and associated with a first robotic arm of the robotic arms, wherein the first robotic arm is coupled to the instrument; and display a haptic interface device (HID) indicator in the instrument-to-arm mapping model along with the instrument data, wherein the HID indicator indicates whether a first HID or a second HID of the console is configured to control the instrument.

[0140] Example combination 21: The non-transitory computer-readable medium according to example combination 20, wherein, in order to indicate the instrument data associated with the first robotic arm, the display is further configured to display a call line that connects the instrument data to the first robotic arm.

[0141] Example Combination 22: A non-transitory computer-readable medium according to any one of Example Combination 20 or Example Combination 21, wherein the graphical representation of the robot arm may include a rendering of each of the robot arms, wherein the rendering of each of the robot arms includes the rendering of one or more joints and links of the robot arm.

[0142] Example Combination 23: A non-transitory computer-readable medium according to any one of Example Combinations 20 to 22, wherein the instrument data may be at least one of an identifier of the first robotic arm, an image depicting the instrument, text describing the instrument, an image indicating the orientation of the instrument, or text describing operations that can be performed by the instrument, wherein the instrument-to-arm mapping model may further include a haptic interface device (HID) indicator displayed together with the instrument data, and wherein the HID indicator indicates an HID of the console configured to control the instrument.

[0143] Example combination 24: A non-transitory computer-readable medium according to any one of example combinations 20 to 23, wherein the processor is further configured to display an image of a patient at the console, wherein the instrument-to-arm mapping model is superimposed on a portion of the image of the patient.

[0144] Example combination 25: A non-transitory computer-readable medium according to any one of example combinations 20 to 24, wherein the console may include a display device, wherein the display device is positioned in a head-mounted device of the console or on the armrest of the console.

[0145] Example Combination 26: A surgical robotic system may include: a patient platform; a plurality of robotic arms, the plurality of robotic arms including a first robotic arm, wherein the first robotic arm is coupled to an instrument; and a console communicatively coupled to the robotic arm, the console including: a first haptic interface device (HID) and a second HID; and a display configured to: display an instrument-to-arm mapping model, the display instrument-to-arm mapping model including a graphical representation of the robotic arm relative to the patient platform of the surgical robotic system; indicate instrument data in the instrument-to-arm mapping model describing the instrument and associated with the first robotic arm; and display a haptic interface device (HID) indicator in the instrument-to-arm mapping model along with the instrument data, wherein the HID indicator indicates whether the first HID or the second HID of the console is configured to control the instrument.

[0146] Example Combination 27: A method performed by a surgical robotic system, the surgical robotic system including a plurality of robotic arms, wherein the method may include: displaying an instrument-to-arm mapping model at a console of the surgical robotic system, the instrument-to-arm mapping model including a graphical representation of the robotic arms relative to a patient platform of the surgical robotic system; indicating instrument data describing an instrument and associated with a first robotic arm among the robotic arms, wherein the first robotic arm is coupled to the instrument; and displaying a haptic interface device (HID) indicator in the instrument-to-arm mapping model along with the instrument data, wherein the HID indicator indicates whether a first HID or a second HID of the console is configured to control the instrument.

[0147] It should be noted that, as used herein, other variations of the term "connection" or the word "linkage" can indicate an indirect or direct connection. For example, if a first component is "connected" to a second component, the first component may be indirectly connected to the second component or directly connected to the second component via another component.

[0148] The function of determining whether an instrument is within or outside the surgical field of view provided by a camera or endoscope and rendering one or more indicators representing the position or orientation of one or more medical instruments described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" means any available medium accessible by a computer or processor. By way of example, and not limitation, such a medium may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disk read-only memory (CD-ROM) or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. It should be noted that a computer-readable medium may be tangible and non-transitory. As used herein, the term "code" may mean software, instructions, code, or data executable by a computing device or processor.

[0149] The methods disclosed herein include one or more steps or actions for implementing the described methods. The method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims unless the correct operation of the described method requires a specific order of steps or actions.

[0150] As used herein, the term "multiple" means two or more. For example, multiple components indicates two or more components. The term "determine" encompasses a variety of actions, and therefore, "determine" can include calculation, operation, processing, derivation, investigation, lookup (e.g., searching in a table, database, or another data structure), ascertainment, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0151] Unless otherwise explicitly stated, the phrase “based on” does not mean “based on only”. In other words, the phrase “based on” describes both “based on only” and “based on at least”.

[0152] As used herein, the term “exemplary” means “serving as an example, instance or illustration” and does not necessarily indicate any preference or superiority of the example relative to any other configuration or specific implementation.

[0153] As used herein, the term “and / or” covers any combination of the listed elements. For example, “A, B and / or C” includes the following sets of elements: A only, B only, C only, A and B without C, A and C without B, B and C without A, and combinations of all three elements A, B and C.

[0154] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. For example, it should be understood that those skilled in the art will be able to employ several corresponding alternative and equivalent structural details, such as equivalent methods of fastening, mounting, connecting or engaging tool components, equivalent mechanisms for generating specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, this disclosure is not intended to be limited to the embodiments shown herein, but is given the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A surgical robot system, comprising: Multiple robotic arms, the multiple robotic arms including a first robotic arm, wherein the first robotic arm is coupled to a device; and A control console, communicatively connected to the robotic arm, the control console comprising: Display devices; and A processor, connected to the display device and configured to: A device-to-arm mapping model, including a graphical representation of the robotic arms, is displayed on the display device, wherein the device-to-arm mapping model depicts the position of each robotic arm in the robotic arm; and The instrument-to-arm mapping model indicates instrument data describing the instrument associated with the first robotic arm.

2. The surgical robot system according to claim 1, wherein, To indicate the instrument data associated with the first robotic arm, the display is further configured to display a call line that links the instrument data to the first robotic arm.

3. The surgical robot system according to any one of claims 1-2, wherein, The graphical representation of the robotic arm includes a rendering of each of the robotic arms, wherein the processor is further configured to update the rendered position of each of the robotic arms based on the actual position of the robotic arm.

4. The surgical robot system according to any one of claims 1-3, wherein, The rendering of each robotic arm in the robotic arm includes the rendering of at least one of the positions, angles, or orientations of one or more joints and links of the robotic arm.

5. The surgical robot system according to any one of claims 1-4, wherein, The instrument data includes at least one of the following: an identifier of the first robotic arm, an image depicting the instrument, text describing the instrument, an image indicating the orientation of the instrument, or text describing operations that can be performed by the instrument.

6. The surgical robotic system according to any one of claims 1-5, wherein, The device-to-arm mapping model also includes a haptic interface device (HID) indicator displayed along with the device data, wherein the HID indicator indicates the HID of the console engaged to control the device.

7. The surgical robot system according to any one of claims 1-6, wherein, The device-to-arm mapping model also includes a haptic interface device (HID) indicator displayed along with the device data, wherein the HID indicator indicates whether the console's HID is configured to control the device but is not currently engaged to control the device.

8. The surgical robotic system according to any one of claims 1-7, wherein, The robotic arm further includes a second robotic arm, wherein the second robotic arm is coupled to a camera, wherein the instrument data includes endoscope data, wherein the endoscope data includes at least one of the camera angle or the camera orientation.

9. The surgical robot system according to any one of claims 1-8, wherein, The processor is further configured to display an image of the patient at the display device, wherein the instrument-to-arm mapping model is superimposed on a portion of the patient's image.

10. The surgical robot system according to any one of claims 1-9, wherein, The instrument data is indicated by icons displayed on the display device.

11. The surgical robot system according to any one of claims 1-10, wherein, The display device is located in the head-mounted device of the console or on the armrest of the console.

12. The surgical robot system according to any one of claims 1-11, wherein, The instrument-to-arm mapping model depicts the position of each robotic arm in the robotic arm relative to the patient platform of the surgical robotic system.

13. The surgical robotic system according to any one of claims 1-12, wherein, The processor is further configured to: Receive one or more user inputs received at one or more user input devices on the console, and In response to receiving the one or more user inputs, the instrument-to-arm mapping model is displayed at the console.

14. The surgical robot system according to any one of claims 1-13, wherein, The instrument data is indicated by icons displayed at the console, wherein the processor is further configured to: Receive selection of the icon; and In response to receiving a selection of the icon, a settings menu is displayed on the console to adjust one or more settings of the device.

15. The surgical robotic system according to any one of claims 1-14, wherein, The processor is further configured to: The device-to-arm mapping model displays a haptic interface device (HID) indicator along with the device data, wherein the HID indicator indicates whether the console's first HID or second HID is configured to control the device.

Images