Enhanced visibility of the robot when the surgeon is focused on the viewer

Abstract

A surgical robotic system includes a plurality of robotic arms including an active robotic arm and an inactive robotic arm, and a console communicatively coupled to the robotic arms. The console includes a first haptic interface device (HID) and a second HID, and a display configured to display an interactive menu including active zone elements and inactive zone elements, indicate data associated with a first instrument coupled to the active robotic arm in the active zone elements based on whether the active robotic arm is assigned to be controlled by the first HID or the second HID, and indicate data associated with a second instrument coupled to the inactive robotic arm in the inactive zone elements, wherein the inactive robotic arm is disengaged from the first HID and the second HID.

Description

priority

[0001] This patent application claims priority to U.S. Provisional Patent Application No. 63 / 596,499, filed November 6, 2023, entitled “Enhanced Visible of Robot While Surgeon Immersed in Viewer,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The systems and methods disclosed herein relate to devices and methods for indicating information about instruments used to perform procedures, and more specifically to robotic systems for displaying information about instruments. Background Technology

[0003] Robotic systems can be used to perform a wide variety of tasks and procedures. They are applicable to various industries, such as manufacturing, automotive, healthcare, and construction. For example, in the healthcare industry, robotic surgical systems have been used to perform a large number of medical procedures, including minimally invasive surgery (e.g., laparoscopic surgery) and non-invasive surgery (e.g., endoscopic surgery). Generally, a robotic system may include a robotic arm and a control console. The robotic arm is configured to control the movement of tools or instruments attached to it, and the user can control the movement of the robotic arm and / or tools through the control console. Summary of the Invention

[0004] In one embodiment, a surgical robotic system is disclosed. The surgical robotic system includes: a plurality of robotic arms, including active and inactive robotic arms; and a console communicatively coupled to the robotic arms. The console includes a first haptic interface device (HID) and a second HID, and a display configured to display an interactive menu including active and inactive area elements, indicating data associated with a first instrument coupled to the active robotic arm in the active area elements, and indicating data associated with a second instrument coupled to the inactive robotic arm in the inactive area elements, wherein the inactive robotic arm is disengaged from both the first and second HIDs.

[0005] In another embodiment, a method performed by a surgical robotic system is disclosed. The method includes: displaying an interactive menu at a console of the surgical robotic system, comprising active and inactive area elements, wherein the active area elements include a left segment, a center segment, and a right segment; indicating, in the left segment of the active area elements, a first instrument coupled to a first robotic arm of the surgical robotic system and controlled by a left haptic interface device (HID) of the console; indicating, in the right segment of the active area elements, a second instrument coupled to a second robotic arm of the surgical robotic system and controlled by a right HID of the console; indicating, in the center segment of the active area elements, data associated with a camera of a third robotic arm coupled to the surgical robotic system; and indicating, in the inactive area elements, a third instrument coupled to a fourth robotic arm, wherein, upon receiving an instruction to switch control of the right HID from the second instrument to the third instrument, the right HID is engaged to control the fourth robotic arm.

[0006] In yet another embodiment, a surgical robot system is disclosed. The surgical robot system includes a plurality of robotic arms and a console communicatively coupled to the robotic arms. The plurality of robotic arms includes a first robotic arm coupled to a first instrument, a second robotic arm coupled to a second instrument, and a third robotic arm coupled to a third instrument. The console includes: a first haptic interface device (HID) configured to control the first robotic arm; a second HID configured to control either the first or second robotic arm; a display configured to display an interactive menu including active and inactive area elements, wherein the active area elements include a first segment, a central segment, and a second segment, wherein the first segment indicates that the first instrument is controlled by the first HID, wherein the second segment indicates that the second instrument is controlled by the second HID, and wherein the inactive area elements indicate that the third instrument is disengaged from both the first and second HIDs; and a processor coupled to the display and configured to receive an instruction to switch control of the second HID from the second instrument to the third instrument, wherein in response to the instruction, the display is configured to indicate the third instrument in the second segment and the second instrument in the inactive area elements.

[0007] In yet another embodiment, a method performed by a surgical robotic system is disclosed. The method includes: displaying an interactive menu including active and inactive area elements at a console of the surgical robotic system; indicating in the active area elements data associated with one or more instruments respectively coupled to the one or more active robotic arms of the surgical robotic system, wherein the one or more active robotic arms are actively controlled by at least one of the first HID or the second HID, based on whether one or more active robotic arms are assigned to be controlled by a first haptic interface device (HID) or a second HID; and indicating in the inactive area elements data associated with instruments coupled to one or more inactive robotic arms of the surgical robotic system, wherein the one or more inactive robotic arms are disengaged from the first HID and the second HID.

[0008] In yet another embodiment, a non-transitory computer-readable medium is disclosed. This non-transitory computer-readable medium stores instructions that, when executed by a processor of a surgical robot system, cause the processor to: display an interactive menu including active and inactive area elements at a console of the surgical robot system; indicate in the active area elements data associated with one or more instruments respectively coupled to one or more active robotic arms of the surgical robot system, based on whether one or more active robotic arms are assigned to be controlled by a first haptic interface device (HID) or a second HID of the console, wherein the one or more active robotic arms are actively controlled by at least one of the first HID or the second HID; and indicate in the inactive area elements data associated with instruments coupled to one or more inactive robotic arms of the surgical robot system, wherein the one or more inactive robotic arms are disengaged from the first HID and the second HID.

[0009] It should be noted that the various examples described above can be combined with any other examples presented herein. The features and advantages described in the specification are not exhaustive, and in particular, many additional features and advantages will be apparent to those skilled in the art, taking into account the drawings, specification, and claims. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and guidance purposes and may not be intended to depict or limit the subject matter of the invention. Attached Figure Description

[0010] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects, wherein similar reference numerals denote similar elements.

[0011] Figure 1 An example is shown of a display that shows an interactive menu at the console of a robot system according to various embodiments of the present disclosure.

[0012] Figure 2A and Figure 2B Examples of various embodiments according to this disclosure are provided. Figure 1 An interactive menu.

[0013] Figure 3 Examples of various embodiments according to this disclosure are provided. Figure 1 An interactive menu.

[0014] Figure 4 Examples of various embodiments of this disclosure that dynamically change based on user input received from a console are illustrated. Figures 1 to 3 An interactive menu.

[0015] Figure 5A and Figure 5B Examples of various embodiments of this disclosure that dynamically change based on user input received from a console are illustrated. Figures 1 to 3 An interactive menu.

[0016] Figure 6A and Figure 6B Examples of various embodiments of this disclosure that dynamically change based on user input received from a console are illustrated. Figures 1 to 3 An interactive menu.

[0017] Figure 7 Examples of various embodiments of this disclosure that dynamically change based on user input received from a console are illustrated. Figures 1 to 3 An interactive menu.

[0018] Figure 8A , Figure 8B and Figure 8C Examples of various embodiments of this disclosure that dynamically change based on user input received from a console are illustrated. Figures 1 to 3 An interactive menu.

[0019] Figure 9A and Figure 9B Examples of various embodiments of this disclosure are provided. Figures 1 to 3 An example of a status indicator displayed in an interactive menu.

[0020] Figure 10A , Figure 10B , Figure 10C and Figure 10D Examples of interactive menus including the status indicator of FIG9 are illustrated according to various embodiments of the present disclosure.

[0021] Figure 11 Examples of various embodiments of this disclosure are provided. Figures 1 to 3 The display of the interactive menu's console indicator.

[0022] Figure 12 This is a flowchart illustrating a first method according to an embodiment of the present disclosure.

[0023] Figure 13 This is a flowchart illustrating a second method according to an embodiment of the present disclosure.

[0024] Figure 14 This is a flowchart illustrating a third method according to an embodiment of the present disclosure.

[0025] Figure 15 This is a schematic diagram illustrating the electronic components of a surgical robot system according to some implementation schemes. Detailed Implementation

[0026] A. Robotic System

[0027] A robotic system may include various components that work together to perform a specific task. While the specific design and functionality of a robotic system can vary based on industry and application, some examples of robotic systems may include one or more robotic arms communicatively coupled to a control console. The control console can be used to manipulate the robotic arms and instruments detachably attached to them. The robotic arm may include one or more joints and links interconnecting the joints. The robotic arm may also include end effectors (e.g., grippers, tools, instruments, etc.) positioned at or detachably attached to the robotic arm, such that the end effectors interact with the environment or a subject.

[0028] A robot system's control console can essentially serve as a control center or interface, allowing an operator or user to interact with and control the robot. The console acts as a command and feedback point for the robot system, providing users with mechanisms to input instructions, monitor the status of various components of the robot system, and receive information about the robot system's components and environment. For example, the console may include a display device configured to display images captured by the robot system's cameras, data describing the robot system's components, the settings and status of the robot system's components, data related to the control of the robot system, and other data describing the robot system. The console may also include one or more input devices configured to control the robot arm / end-effector of the system. Input devices may include, for example, buttons, switches, touch-sensitive surfaces, gimbals, haptic interface devices (HID), foot pedals, etc., located in various areas of the console (e.g., near the user's hand or foot). The user of the console can actuate or interact with the input devices to control the robot arm / end-effector, manipulate information displayed on the console, adjust console settings, etc. It should be understood that, in addition to the robotic arm and control console, a robotic system may include many other components, such as processing hardware, sensors, power supplies, safety systems, programming interfaces, and memory.

[0029] For illustrative purposes, this document may describe various implementations in relation to surgical robotic systems used in the medical industry. However, it should be understood that robotic systems can be applied to many other industries, and the robotic systems described herein are not limited to surgical robotic systems.

[0030] To this end, an exemplary surgical robotic system may include multiple robotic arms whose movement can be controlled by a HID located on a console of the surgical robotic system. The robotic arms can be positioned in a manner convenient for performing procedures on a patient, and the healthcare provider can manipulate the HID to control the movement of end effectors (e.g., which may include medical instruments, cameras, etc.). The end effectors may interact with or be positioned within the patient's body, and therefore, the HID can be used to control the movement of the end effectors within or on the patient during the procedure. At least one of the instruments may include a camera that can be used to capture real-time images of the patient's anatomy during the procedure. The images may also show the movement of the end effectors relative to the patient's anatomy. A display device on the console can display the images in real time, thereby providing visual feedback to the healthcare provider during procedure execution, which can be used to make informed decisions during the procedure.

[0031] B. Display devices for robot systems

[0032] In some cases, the console's display device can be implemented as a head-mounted device, where the structure of the display device essentially surrounds the area around the user's eyes to create a sense of immersion in the images and data displayed by the device. When the user is immersed, they can insert their head into the display device in a way that blocks out ambient light. In this way, the user is able to focus on images (e.g., patient anatomy) and the movement of instruments, which can be shown on the display device, while performing various tasks (e.g., patient procedures). The user can perform these tasks without being distracted by any movement or events occurring outside of the immersion. For example, immersion at the display device reduces the likelihood of a healthcare provider being distracted during procedures that could otherwise lead to harm or injury to the patient.

[0033] In some cases, immersion at the display device limits the amount of information a user can access while performing a task using a robotic arm. For example, the display device might be limited to showing images of the patient received by a camera that defines the patient's position. Sometimes, it might also display basic details about the instruments being used by the robotic system. In this way, the information displayed at the display device is typically minimized to further reduce the risk of distracting the user from the primary focus of the procedure.

[0034] However, detailed information about the instruments can be crucial for informed decision-making during the procedural process. For example, the information displayed on the display device may not indicate the system's hand-handling in an easily understandable format, and the format may be space-conservative (i.e., minimizing the display size on the display device). The term "hand-handling" can refer to the allocation of control of an instrument (and therefore, the associated robotic arm to which the instrument is attached) to a specific input device at the console. For example, the hand-handling of a robotic system might indicate that a first instrument attached to a first robotic arm is controlled by the left HID (i.e., the input device) of the console, and a second instrument attached to a second robotic arm is controlled by the right HID of the console. Furthermore, the information displayed on the display device may not indicate the difference between instruments actively engaged by the input device and instruments that can be attached to the robotic arm but not actively engaged by the input device. When an instrument is assigned to be attached (e.g., locked in a detachably attached manner) to a robotic arm, the instrument can be attached to the robotic arm but not actively engaged by the input device. For example, the robotic system may be pre-configured such that certain robotic arms and therefore attached instruments are attached via certain user input devices. A device can be actively engaged by the input device only when the action performed at the input device actually causes the movement or manipulation of the robotic arm and / or the attached device.

[0035] For example, a surgeon can perform a procedure on a patient by considering how to manipulate the instruments shown on the display device using either the left-hand HID operated by the surgeon's left hand or the right-hand HID operated by the surgeon's right hand. In other words, the healthcare provider's hand can substantially drive and manipulate the instruments by controlling the HID. Therefore, the hand bias of the console is particularly important for the healthcare provider during procedure execution.

[0036] However, the menus displayed to the healthcare provider during the procedure may not show the system's eccentricity in an easily understandable format, and the format may also be space-conservative (i.e., minimizing the display size at display device 242). Furthermore, these menus may not indicate the difference between instruments actively engaged by the HID and instruments that can be coupled to the robotic arm but not actively engaged by the HID. An instrument can be actively engaged by the HID when it is coupled (e.g., locked in a detachably attached manner) to the robotic arm, and when an action performed at the HID (or at any actuator / button on the HID) causes movement or manipulation of the robotic arm and / or the instrument. In contrast, an instrument may not be actively engaged by the HID when it is coupled to the robotic arm but is not currently controllable by the HID.

[0037] Failure to indicate the hand bias of a robotic system can be problematic for various reasons. For example, failing to indicate the hand bias of a robotic system to a healthcare provider can cause confusion and may lead to delays during procedure execution. Healthcare providers may need additional time before procedure to manually identify instruments controlled by each HID, or to manually test the HID to identify instruments that move as the HID moves, which can be shown on the display device. This is especially true in surgical robotic systems where the positioning of the robotic arms may not be indicated in a linear and sequential manner. When, for example, the robotic arm translates around the operating table, the robotic arm may not be indicated in a linear and sequential manner, such that the order in which the robotic arm moves from one side (i.e., the left) to the other side (i.e., the right) is not always sequential.

[0038] Failure to indicate the hand bias of a robot system can also lead to unnecessary consumption of processing and communication resources during setup and procedure execution. For example, if the user of the console is unaware of the machine's HID control, they may have to reconfigure the console settings to manually specify the machine to be controlled by a particular HID. If the user already knows the hand assignment for the machine, the processing and communication load involved in reconfiguring the HID settings can be avoided.

[0039] C. Introduction to the interactive menus displayed in the console

[0040] This disclosure provides a technical solution to the aforementioned technical problems related to robotic systems and platforms by providing an interactive menu to be displayed on a display device. The interactive menu can indicate the system's hand position (i.e., indicate the instrument actively engaged with and controlled by the corresponding HID). In some embodiments, the interactive menu may include an activity area element comprising indicators associated with one or more instruments actively engaged by the HID. The activity area element may also include data associated with a camera device attached to one of the robotic arms. The interactive menu can be overlaid on an image displayed on the display device. The activity area element may also include segments or indicators to the right and left of the endoscope data to visually indicate which instrument is active in each of the healthcare provider's hands. In this way, the activity area element can combine the indications of two active instruments and the camera into a single unified cluster, which can be spatially conservatively positioned on the display.

[0041] In some implementations, the interactive menu may also include an inactive area element indicating data about an inactive device that may be coupled to a robotic arm and assigned to be controlled by the HID, but may not be actively engaged (e.g., controlled) by the HID, as the HID may be controlling another active device at that time. The inactive area element may be separate from the active area element, but is located proximal to either hand assigned the inactive device or the inactive robotic arm. The separation and positioning of the inactive area element emphasizes to the healthcare provider that the inactive device is inactive (e.g., may not be under HID control). This also indicates to the healthcare provider which HID the inactive device is assigned to (and therefore the corresponding hand of the healthcare provider).

[0042] D. Interactive Menu

[0043] Figure 1 Figure 1100 illustrates a console 240 according to various embodiments of the present disclosure, which displays an image 1103 with an interactive menu 1110 above the image. Figure 1 The interactive menu 1110 also illustrates the hand association between HID 226, 228 and the instruments 212A-212B attached to the robotic arms 210A-210B (i.e., it shows the hand bias of HID 226, 228 to the various instruments 212A-212B). Figure 1 The console 240 is shown as including a display device 242, a left HID 226 (also referred to herein as "first HID 226"), a right HID 228 (also referred to herein as "second HID 228"), and a foot pedal assembly 244. However, it should be understood that the console 240 may include... Figure 1 Other components not shown or described elsewhere.

[0044] exist Figure 1 In the example shown, the healthcare provider may sit at console 240 with his or her head immersed in display device 242. The healthcare provider's left hand can operate left HID 226, and the healthcare provider's right hand can operate right HID 228. Each of HIDs 226 and 228 may be assigned to control multiple robotic arms 210A-210B, but may only be allowed to actively control one robotic arm 210A-210B at a time, and thus one instrument 212A-212B coupled to that robotic arm 210A-210B at a time. The robotic arms 210A-210B and instruments 212A-212B that are actively engaged with and controlled by HIDs 226 and 228 may be referred to herein as "active robotic arm 210" and "active instrument 212".

[0045] Display device 242 can display image 1103. Image 1103 may be an image of the patient's anatomy captured by a camera (such as, for example, an endoscopic camera or a laparoscopic camera). In one case, the camera may be coupled to one of the robotic arms 210A-210B and capture image 1103 as a real-time stream while the healthcare provider performs procedures on the patient using the surgical robotic system. In another case, another camera may have already captured image 1103 before performing procedures on the patient using the surgical robotic system.

[0046] Image 1103 can not only show a portion of the patient's anatomy, but in some cases, it can also show one or more active instruments 212A-212B actively engaged by HID 226 and 228. Figure 1 In the example shown, image 1103 displays a first instrument 212A and a second instrument 212B, wherein the two instruments 212A-212B may be different or may be the same. Figure 1 In the example shown, the first robotic arm 210A is shown as being coupled to the first instrument 212A, while the second robotic arm 210B is shown as being coupled to the second instrument 212B. Figure 1 Image 1103 is shown as a portion of robotic arms 210A-210B for illustrative purposes only, and it should be understood that robotic arms 210A-210B may sometimes not be shown in image 1103. In these cases, only the end effectors of instruments 212A-212B may be shown in image 1103.

[0047] Interactive menu 1110 can be overlaid on top of image 1103. For example... Figure 1 As shown, the interactive menu 1110 can be positioned at the bottom edge of image 1103. The size of the interactive menu 1110 can be a small percentage of the size of image 1103. However, the size of the interactive menu 1110 can be adjusted via user interaction, for example, at console 240, to increase or decrease its size. In some embodiments, the size of the interactive menu 1110 may be less than or equal to a maximum proportion or percentage of the size of image 1103 to ensure that the interactive menu 1110 does not distract the healthcare provider from procedural control of the devices 212A-212B shown in image 1103.

[0048] The interactive menu 1110 may include an active area element 1111 and an inactive area element 1112. The active area element 1111 may indicate information about active devices 212A-212B that are actively controlled by HIDs 226 and 228. The inactive area element 1112 may indicate information about inactive device 212, which may be coupled to the robotic arm 210 but is not actively controlled by HIDs 226 and 228.

[0049] like Figure 1 As shown, the active area element 1111 may include a left segment 1115 (sometimes referred to herein as "first segment 1115"), a central segment 1117, and a right segment 1120 (sometimes referred to herein as "second segment 1120"). The left segment 1115 may include data describing the active device 212A controlled by the left HID 226. The left HID 226 may be assigned to and actively control a first robotic arm 210A coupled to the first active device 212A. The right segment 1120 may include data describing the active device 212B controlled by the right HID 228. The right HID 228 may be assigned to and actively control a second robotic arm 210A coupled to the second active device 212B.

[0050] like Figure 1 As shown, interactive menu 1110 indicates the hand association between HIDs 226, 228 and segments 1115 and 1120. Specifically, the left segment 1115 indicating device 212A is controlled by the left HID 226 and operated by the user's left hand. Similarly, the right segment 1120 indicating device 212B is controlled by the right HID 228 and operated by the user's right hand. Although Figure 1 The image shows device 212A positioned on the left side of image 1103 and device 212B positioned on the right side of image 1103. However, the positioning of devices 212A-212B within image 1103 may be independent of interactive menu 1110. For example, even when device 212A is positioned on the right side of image 1103 and device 212B is positioned on the left side of image 1103, the positioning of data within active area element 1111 may remain the same. In this case, interactive menu 1110 may still indicate that HID 226 is available for controlling device 212A and HID 228 is available for controlling device 212B. In the following text, the terms "devices 212A-212B" may sometimes be referred to as "device 212", and the terms "robot arms 210A-210B" may sometimes be referred to as "robot arms 210".

[0051] Inactive area element 1112 can indicate data associated with inactive device 212. For example... Figure 1As shown, inactive element 1112 is located outside of active element 1111, but near left segment 1115, thus indicating that inactive device 212 is assigned to left HID 226. Visual factors of inactive element 1112 can also indicate that inactive device 212 is not currently controlled by left HID 226. For example, visual factors could be the brightness level or color of inactive element 1112 or at least a portion of the boundary of inactive element 1112. The brightness level of inactive element 1112 may be less than the brightness level of active element 1111. The color of inactive element 1112 can be blended into the background of image 1103 to dim the information in the inactive element, while the color of active element 1111 can contrast with the background of image 1103 to emphasize the information in active element 1111. In this way, the interactive menu 1110 includes two main components, namely the active area element 1111 and the inactive area element 1112, which together illustrate the system’s eccentricity to the user in a compact and easy-to-digest format.

[0052] Figure 2A and Figure 2B This is a diagram illustrating an interactive menu 1110 according to various embodiments of this disclosure. See details. Figure 2A The interactive menu 1110 may include an active area element 1111 and an inactive area element 1112. The active area element 1111 may include a left segment 1115, a center segment 1117, and a right segment 1120.

[0053] The visual elements of active area element 1111 may indicate that the instruments 212A-212B described in active area element 1111 actively engage with HID 226, 228. For example, the visual elements of active area element 1111 may be displayed more prominently than the visual elements of inactive area element 1112 to emphasize that the instrument 212 indicated in active area element 1111 is active, while the instrument 212 indicated in inactive area element 1112 is inactive. For this purpose, active area element 1111 may be displayed with, for example, greater brightness or a different color than inactive area element 1112. For example, the color used to display different components, component boundaries, and text in active area element 1111 and / or the background color behind the text and icons in active area element 1111 may have a higher brightness level or higher contrast than the colors in inactive area element 1112. The size / thickness of the text in active area element 1111 may be larger than the size / thickness of the text in inactive area element 1112. In this way, the inactive element 1112 is darkened or dimmed relative to the active element 1111, so that the inactive element 1112 does not distract the user from the data indicated in the image 1103 or the active element 1111.

[0054] like Figure 2AAs shown, inactive area element 1112 is located near the left segment 1115 of active area element 1111. This signals to the user that the device 212 described in inactive area element 1112 can be controlled by the robotic arm 210 assigned to be operated by left HID 226. As further described herein, the user can actuate the user input device at console 240 to switch control of left HID 226 from device 212 indicated in left segment 1115 to device 212 indicated in inactive area element 1112.

[0055] Each of the segments 1115, 1117, and 1120 of the activity area element 1111 may include instrument indicators 1203A, 1203B, and 1203C, respectively. Instrument indicator 1203A in the left segment 1115 may indicate data regarding a first instrument 212A attached to the first robotic arm 210A, which can be controlled by a first HID 226. When the left segment 1115 is displayed to the left of the central segment 1117, the user can determine that the first instrument 212A indicated in the left segment 1115 can be controlled by the first HID 226, because the first HID 226 can be operated by the user's left hand. Instrument indicator 1203B in the right segment 1120 may indicate data regarding a second instrument 212B attached to the second robotic arm 210B, which can be controlled by a second HID 228. When the right segment 1120 is displayed to the right of the central segment 1117, the user can determine that the second instrument 212B indicated in the right segment 1120 can be controlled by the second HID 228, because the second HID 228 can be operated by the user's right hand. The instrument indicator 1203C of the central segment 1117 can indicate endoscope data describing the camera attached to the third robotic arm 210. The instrument indicator 1203D of the inactive area element 1112 can indicate data about the third instrument 212 attached to the fourth robotic arm 210, which can be the inactive robotic arm 210 (i.e., not actively controlled by any of the HIDs 226, 228, and can be, for example, placed next to the system and / or locked in place to prevent accidental manual movement of the inactive robotic arm 210).

[0056] Now for specific reference Figure 2BEach instrument indicator in instrument indicators 1203A-1203C may include text 1206A-1206B and / or icons 1209A-1209D, 1220, indicating data about the active instruments 212A-212B attached to the active robotic arms 210A-210B. For example, instrument indicator 1203A describing the first instrument 212A attached to the first robotic arm 210A may include, for example, an identifier of the first robotic arm 210A, an identifier of the first instrument 212A, icons 1209A-1209D indicating one or more operations that can be performed using the first instrument 212A, and / or any other information describing the first instrument 212A and / or the first robotic arm 210A. The identifier of the first robotic arm 210A may be represented as text 1206, which includes one or more alphanumeric values ​​identifying the first robotic arm 210A. The identifier of the first instrument 212A may be represented as text 1206 describing the name of the first instrument 212A. This designation can be easily and immediately understood by the user as the first instrument 212A, especially when the first instrument 212A is also shown in image 1103.

[0057] Icons 1209A-D may represent information describing one or more operations to be performed or currently performed using instruments 212A-212B. Each icon in 1209A-1209D may include a dynamic boundary 1213 and text 1216 (although...). Figure 2B Only icon 1209A is shown with reference numerals 1213 and 1216; however, it should be understood that icons 1209B, 1209C, and 1209D also include dynamic boundaries 1213 and text 1216. Text 1216 may describe the type of operation indicated by the corresponding icons 1209A-1209D. Each dynamic boundary 1213 may display certain visual features or attributes based on the operation to be performed or currently performed by the corresponding instruments 212A-212B. Figure 2BAs shown, the dynamic boundary 1213 at icons 1209A-1209B can be located on an edge of each icon in icons 1209A-1209B, near or adjacent to the central segment 1117. Similarly, the dynamic boundary 1213 at icons 1209C-1209D can also be located on the edges of icons 1209C-1209D, near or adjacent to the central segment 1117. However, it should be understood that other edges of icons 1209A-1209D, or multiple edges of icons 1209A-1209D, may also include similar dynamic boundaries 1213. Icons 1209A-1209D can also be located on the edges of instrument indicators 1203A-1203D, near or adjacent to the central segment 1117. In this way, the dynamic boundary 1213 and icons 1209A-1209D are positioned at the center of the active area element 1111, so that the information depicted by icons 1209A-1209D is displayed at the display device 242 in a centered but not distracting manner.

[0058] For example, each dynamic boundary in dynamic boundary 1213 can be set to a specific color, shadow, transparency, pattern, or other type of visual attribute to indicate the type of operation represented by icons 1209A-1209D, the steps of performing the operation (e.g., set to perform, actually performed, not performed, etc.), or other actions associated with the operation. The attributes of dynamic boundary 1213 can be changed based on the state of the operation (e.g., whether the operation is currently being performed using instruments 212A-212B or not). Figure 2B As shown, as a way to indicate the type of operation to the user, each dynamic boundary in dynamic boundary 1213 can be different for different types of operations. In some cases, the instruments 212A-212D described by instrument indicator 1203B may not be able to perform more than one operation. In this case, icon 1209C can be left blank.

[0059] Each icon in icons 1209A-1209D may include text 1216 in a predefined color and / or the background color of icons 1209A-1209D may be a predefined color, either of which may indicate the state of operation. For example, the background color may change to a first color when a specific operation is about to be activated or performed (e.g., when the user's foot is hovering over the input device at the foot pedal assembly 244), the background color may flash brightly when the position of the first instrument 212A is positioned to perform an operation, and the background color may change to a second color when the operation is actually performed, etc. Icons 1209A-1209D may also include an image representing instrument 212A or an operation, wherein the image may be provided by a camera or may be a rendered image.

[0060] As an illustrative example, a healthcare provider can activate the operation indicated by icon 1209A via an input device (e.g., a button or pedal) at a foot pedal assembly 244. For example, foot pedal assembly 244 may include a first input device that, when actuated (e.g., pressed), can activate a cutting operation using a first instrument 212A identified in instrument indicator 1203A. Similarly, foot pedal assembly 244 may include a second input device that, when actuated (e.g., pressed), can activate a coagulation operation using the first instrument 212A identified in instrument indicator 1203A. In this case, icon 1209A may be associated with a cutting operation, and icon 1209B may be associated with a coagulation operation. For example, when the input device is released, the first instrument 212A may terminate the corresponding operation. The background color and / or attributes of the dynamic boundary 1213 of the icon 1209A indicating an operation may change relative to the actuation of the foot pedal assembly 244 and / or the state of the operation performed on the patient at the location where the first device 212A is positioned. For example, the dynamic boundary 1213 and / or the background color of the icon 1209A may change to a first color when the healthcare provider hovers over the input device at the foot pedal assembly 244 for activating the operation, may change again when the healthcare provider first actuates the input device for performing the operation, and may remain flashing when the healthcare provider maintains actuation of the input device for a period of time to continue performing the operation. In this way, the icons 1209A-1209D indicating operations may include several different ways to signal information to the user in a compact format that is easy and clearly understandable to the user when the user is performing an operation and viewing the interactive menu 1110.

[0061] The instrument indicator 1203B describing the second instrument 212B attached to the second robotic arm 210B may similarly include, for example, an identifier of the second robotic arm 210B, an identifier of the second instrument 212B, an indication of one or more operations that can be performed by the second instrument 212B, and / or any other information describing the second instrument 212B or the second robotic arm 210B. For this purpose, the instrument indicator 1203B may also include text 1206 describing the second instrument 212B attached to the second robotic arm 210B and icons 1209C-1209D indicating and describing the operational states that can be performed by the second instrument 212B.

[0062] The instrument indicator 1203C describing the third instrument 212 (such as, for example, a camera) may also include an identifier for the third robotic arm 210, an identifier for the third instrument 212, instructions for one or more operations that can be performed using the third instrument 212, and / or any other information describing the third instrument 212 or the third robotic arm 210, which may be in the form of text 1206. The instrument indicator 1203C may also include an icon 1220 describing the setup and status of the third instrument 212 coupled to the robotic arm 210. Figure 2B In the example shown, the third device 212 can be a camera or a peep endoscope.

[0063] For example, when HIDs 226 and 228 are switched to camera control mode, HIDs 226 and 228 can be locked together in a single plane and move together to control the movement of the third robotic arm 210, and thus control the movement of the camera. In this case, the instrument indicator 1203C may include endoscope data describing the camera used by the robotic system. The endoscope data may include, for example, an identifier for the third robotic arm 210 and data describing the camera. The endoscope data may include an icon 1220 representing the camera. For example, icon 1220 may be represented as a circle with two wings, where the wings may indicate the orientation of the camera relative to a horizontal line (e.g., as indicated by "H" in icon 1213). Icon 1220 may rotate depending on the position of the camera relative to the horizontal line. Icon 1220 may also indicate the type of frame at the tip of the camera, for example, by indicating the tilt angle of the frame located at the tip of the endoscope or laparoscopic device (e.g., as indicated by the text 30° in icon 1213). Icon 1220 may also include an arrow indicating the direction of the camera used to delineate the patient's area during the procedure.

[0064] The instrument indicator 1203D included in the inactive area element 1112 describes an inactive instrument 212 coupled to the fourth robotic arm 210, which is not actively engaged with or controlled by HIDs 226, 228. The instrument indicator 1203D may include, for example, an identifier for the fourth robotic arm 210, an identifier for the fourth instrument 212, and / or any other information describing the fourth instrument 212 or the fourth robotic arm 210. For this purpose, the instrument indicator 1203D may also include text 1206 describing the fourth instrument 212 coupled to the fourth robotic arm 210.

[0065] In one implementation, inactive area element 1112 and segments 1115, 1117, and 1120 may each include dynamic boundaries that may be highlighted in various colors and / or shades to indicate whether the corresponding device 212 is engaged with or can be controlled by HIDs 226 and 228. The dynamic boundaries may be positioned around one or more edges of inactive area element 1112 or segments 1115, 1117, and 1120. For example, when left HID 226 is coupled to and engaged with first device 212A, making left HID 226 available for controlling first device 212A, the dynamic boundary of left segment 1115 may be blue. In contrast, when left HID 226 is disengaged, the dynamic boundary of left segment 1115 may be black or gray and may not be used for controlling first device 212A.

[0066] Although the interactive menu 1110 shown in Figure 2 includes only one inactive area element 1112 separated from the active area element 1111, it should be understood that the interactive menu 1110 may include, for example, another inactive area element 1112 located near and separated from the right segment 1120. Although Figure 12 Various components and text included in the interactive menu 1110 are shown, but it should be understood that the interactive menu 1110 may include several other types of data in the form of icons, colors, text and / or other indicators that may be used to describe the instrument 212, the robotic arm 210 and / or operations.

[0067] Figure 3 Example image 1103 and interactive menu 1110 are illustrated according to various embodiments of the present disclosure. Image 1103 and interactive menu 1110 can be displayed at display device 242 of console 240.

[0068] Interactive menu 1110 can be overlaid on image 1103, such as Figure 3 As shown. Alternatively, the interactive menu 1110 can be displayed as a separate window from the image 1103, allowing the user to actuate the input device at the console 240 to switch between the image 1103 and the interactive menu 1110.

[0069] like Figure 3 As shown, the interactive menu 1110 can be positioned at the bottom edge 1303 of the image 1103. The interactive menu 1110 can also be positioned near any other edge of the image 1103 in other ways, as long as the interactive menu 1110 is superimposed on the image 1103 in a non-distracting manner.

[0070] The size of the interactive menu 1110 can be a percentage of the size of the image 1103. However, the size of the interactive menu 1110 can be adjusted via user input at the console 240 to increase or decrease its size. In some embodiments, the size of the interactive menu 1110 may be less than or equal to a preset scale or percentage of the size of the image 1103 to ensure that the interactive menu 1110 does not distract the user from the task performed by the instrument 212 displayed in the image 1103.

[0071] Figure 3 Examples of eye movements of a medical provider when viewing image 1103 and interactive menu 1110 according to various embodiments of this disclosure are also illustrated. While display device 242 may display the entire image 1103, the user can primarily and most easily view area 1403 when their head is immersed in display device 242. In this way, area 1403 may represent an approximation of the user's general field of vision when their head is immersed in display device 242. Simultaneously, the area within circle 1406 may represent an approximation of the user's focal area or visual center when their head is immersed in display device 242. For example, the area within circle 1406 may present the patient's anatomy, where procedures are being or will be performed by active instruments 212A and 212B.

[0072] like Figure 3 As shown, the components of interactive menu 1110 or the information indicated in interactive menu 1110 are not distributed across the entire image 1103 or screen, but are presented compactly in one location. In this way, users can quickly view, read, and digest the information in interactive menu 1110 without straining their eyes or being distracted from the area within circle 1406 or area 1403.

[0073] An example of the healthcare provider's eye movement while performing procedures on a patient and viewing image 1103 and interactive menu 1110 is presented. Figure 3 Arrow 1409 in the image. The positioning of the interactive menu 1110 at the bottom edge 1303 of the image 1103 or screen allows the interactive menu 1110 to be positioned within area 1403, or within the general field of vision of the healthcare provider when they are immersed in the display device 242. Alternatively, the interactive menu 1110 may be positioned near circle 1406, so that viewing and interacting with the interactive menu 1110 may not necessarily distract the healthcare provider from the main area of ​​the procedure.

[0074] E. Interactive menu updates based on user input

[0075] The following Figures 4 to 8CThis describes one or more user input devices that, when actuated or released, can cause certain updates to be displayed at interactive menu 1110. The following is about... Figures 4 to 8C The example user input device described is a pedal on the foot pedal assembly 244. However, it should be understood that... Figures 4 to 8C The interactive menu 1110 shown can be updated in response to user interaction with any other type of user input device, such as a touchscreen-enabled display, switches / buttons on HID 226, 228, switches / buttons on the wrist rest of console 240, etc. Therefore, for illustrative purposes only, Figures 4 to 8C The example uses foot pedal assembly 244. However, user input devices that can cause updates to interactive menu 1110 should not be limited to the pedals in foot pedal assembly 244.

[0076] However, Figure 4 This is a diagram illustrating various modes of foot pedal assembly 244, according to various embodiments of the present disclosure, which can be used to operate the apparatus 212 and camera controlled by a robotic system. (See diagram for details.) Figure 4 As shown, the foot pedal assembly 244 includes multiple input devices 1501-1509, which may be buttons, pedals, switches, touch-sensitive surfaces, etc. Each input device 1501-1509, when actuated, can display a mode, perform an operation, or adjust the display at the display device 242. Although Figure 4 Only nine input devices 1501-1509 are shown in the diagram, but it should be understood that the foot pedal assembly 244 may include any number of input devices 1501-1509.

[0077] exist Figure 4 In the illustrated embodiment, the foot pedal assembly 244 may be part of the console 240 of the surgical robot system. In this case, the foot pedal assembly 244 may include, for example, a clutch input device 1501, a camera input device 1502, a menu input device 1503, an upper left input device 1504, a lower left input device 1505, an upper right input device 1506, a lower right input device 1507, a left instrument switch input device 1508, and a right instrument switch input device 1509.

[0078] When actuated, the clutch input device 1501 disengages HID 226 and 228 from the corresponding robot arm 210. When actuated, the camera input device 1502 locks HID 226 and 228 together, ensuring that they can only move together with the same motion. This can be used to control the movement of the camera attached to the robot arm 210.

[0079] When activated, menu input device 1503 can instruct display device 242 to display interactive menu 1110 or remove interactive menu 1110 from display. This can be done by overlaying interactive menu 1110 onto image 1103 or by presenting interactive menu 1110 as a separate display from image 1103. In the latter case, activation of menu input device 1503 can alternate between displaying image 1103 and displaying interactive menu 1110.

[0080] The upper left input device 1504 and the lower left input device 1505 may each correspond to different operations that can be performed by the first active instrument 212A controlled by the left HID 226. Referring back to Figure 2, the interactive menu 1110 may include icons 1209A-1209D, each icon representing an operation that can be performed by the corresponding instrument 212. Each operation of the first active instrument 212A controlled by the left HID 226 can be initiated and / or activated using either the upper left input device 1504 or the lower left input device 1505.

[0081] Similarly, the upper right input device 1506 and the lower right input device 1507 may each correspond to different operations that can be performed by the second active instrument 212B controlled by the right HID 228. Each operation of the second active instrument 212B controlled by the right HID 228 can be initiated and / or activated using either the upper right input device 1506 or the lower right input device 1507.

[0082] When actuated, the left instrument switch input device 1508 instructs the console 240 to move the instrument indicator 1203D describing the inactive instrument 212 (configured to be controlled by the left HID 226) from the inactive area element 1112 to the left segment 1115, and to move the instrument indicator 1203A describing instrument 212A from the left segment 1115 to the inactive area element 1112. In this case, the inactive area element 1112 can be positioned near the left side of the left segment 1115. When actuated, the right instrument switch input device 1509 instructs the console 240 to move the instrument indicators 1203A-1203D describing the inactive instrument 212 (configured to be controlled by the right HID 228) from the inactive area element 1112 to the right segment 1120, and to move the instrument indicator 1203B describing instrument 212B from the right segment 1120 to the inactive area element 1112. In the latter case, the inactive element 1112 can be located near the right side of the right segment 1120.

[0083] Figure 5A and Figure 5B Various embodiments of the interactive menu 1110 that dynamically changes in response to actuation of a user input device at the foot pedal assembly 244 are illustrated according to various embodiments of the present disclosure. Figure 5Aand Figure 5B The actuated user input device is the clutch input device 1501. Specifically, Figure 5A An embodiment of the interactive menu 1110A is shown when the clutch input device 1501 of the foot pedal assembly 244 is not actuated, and Figure 5B An embodiment of the interactive menu 1110B is shown when the clutch input device 1501 of the foot pedal assembly 244 is actuated.

[0084] Figures 5A to 5B The interactive menus 1110A-1110B shown each include dynamic boundaries 5503, 5506, and 5509. Dynamic boundary 5503 at least partially surrounds the left segment 1115 (or the edge of the left segment 1115 that is not adjacent to the center segment 1117). Dynamic boundary 5506 at least partially surrounds the right segment 1120 (or the edge of the right segment 1120 that is not adjacent to the center segment 1117). Dynamic boundary 5509 at least partially surrounds the center segment 1117 (or the top and bottom edges of the center segment 1117 that are not adjacent to the left and right segments 1115 and 1120).

[0085] Figure 5A An example of the interactive menu 1110A is shown when the clutch input device 1501 is not actuated. Figure 5A As shown, the dynamic boundary 5503 of the left segment 1115 and the dynamic boundary 5506 of the right segment 1120 remain active and highlighted, thereby indicating that the instruments 212A-212B indicated in the left segment 1115 and the right segment 1120 are engaged for control by HID 226, 228. For example, dynamic boundaries 5503 and 5506 in Figure 5A The dynamic boundary 5509 of the central section 1117 is indicated as active and highlighted by depicting boundaries 5503 and 5506 in white without any shading. Simultaneously, the dynamic boundary 5509 of the central section 1117 is deactivated, indicating that the instrument 212 described in the central section 1117 is not controlled by HID 226, 228. For example, the dynamic boundary 5509 is indicated as deactivated by depicting boundary 5509 in gray with shading. Furthermore, the dynamic boundary 1213 of icons 1209A-D and icons 1209A-1209D in the corresponding sections 1115 and 1120 is also shown as active and highlighted in the interactive menu 1110A, indicating that the operation represented by icons 1209A-D can be performed using instruments 212A-212B. Similarly, the dynamic boundary 1213 of icons 1209A-1209D and icons 1209A-1209D is also indicated as active and highlighted in the interactive menu 1110A, indicating that the operation represented by icons 1209A-D can be performed using instruments 212A-212B. Figure 5A The middle is indicated as being active and is highlighted by depicting the white icons 1209A-1209D and the dynamic border 1213 without any shading.

[0086] Figure 5B An example of the interactive menu 1110B when the clutch input device 1501 is actuated is shown. Figure 5B As shown, the dynamic boundary 5503 of the left segment 1115 and the dynamic boundary 5506 of the right segment 1120 are deactivated, thus indicating that the instruments 212A-212B indicated in the left segment 1115 and the right segment 1120 are not controlled by HID 226, 228. The dynamic boundary 5509 of the central segment 1117 remains deactivated, similarly indicating that the instrument 212 described in the central segment 1117 is also not engaged for control by HID 226, 228. By depicting boundaries 5503, 5506, and 5509 as gray with shading (i.e., coloring boundaries 5503, 5506, and 5509 in gray or another darker color), the dynamic boundaries 5503, 5506, and 5509 are... Figure 5B It was instructed to be deactivated.

[0087] Furthermore, both left segment 1115 and right segment 1120 include clutch icon 1503 instead of icons 1209A-1209D. In other words, the area of ​​each instrument indicator in instrument indicators 1203A-1203B, which previously included two icons 1209A-D, can instead include a single clutch icon 1503. The size of clutch icon 1503 can be the same as the size of the two icons 1209A-1209B or 1209C-1209D. Figure 5B As shown, the clutch icon 1503 may include a rendered icon with an opposing arrow and / or text stating the term "clutch". In this way, it can be easily and immediately determined that the devices 212A-212B described in the left section 1115 and the right section 1120 are not only disengaged from HID 226, 228, but also engaged at the bedside.

[0088] Although the activation and deactivation of segments 1115, 1117, and 1120, dynamic boundaries 1203, 5503, 5506, and 5509, and icons 1209A-D are related to... Figures 5A to 5B It is described as being displayed in white or dark gray, but it should be understood that the activation and deactivation of segments 1115, 1117 and 1120, dynamic boundaries 1203, 5503, 5506, 5509 and icons 1209A-1209D can be indicated in a variety of other different ways (e.g., using color, flashing, shadows, etc.), which should not be limited in any other way in this document.

[0089] Figure 6A and Figure 6BVarious embodiments of the interactive menu 1110 that dynamically changes in response to actuation of a user input device at the foot pedal assembly 244 are illustrated according to various embodiments of the present disclosure. Figure 6A and Figure 6B The actuated user input device is the camera input device 1502. Specifically, Figure 6A An embodiment of the interactive menu 1110A is shown when the camera input device 1502 of the foot pedal assembly 244 is not actuated, and Figure 6B An embodiment of the interactive menu 1110B is shown when the camera input device 1502 of the foot pedal assembly 244 is actuated.

[0090] like Figure 6A As shown in the interactive menu 1110A, when the camera input device 1502 is not activated, visual elements of the dynamic boundary 5503 around at least a portion of the left segment 1115 and the dynamic boundary 5506 around at least a portion of the right segment 1120 remain active (e.g., presented in color or brightness, for example, emphasizing the left segment 1115 and the right segment 1120 to the user). Figure 6A As shown, the dynamic boundary 5503 of the left segment 1115 is activated and remains highlighted in a brighter color (e.g., shown in white instead of a gray shading), and similarly, the dynamic boundary 5506 of the right segment 1120 is activated and remains highlighted, thereby indicating that the instrument 212 identified in the left segment 1115 and right segment 1120 is actively controlled by HID 226, 228. Furthermore, the corresponding segments 1115 and 1120 may also display instrument indicators 1203A and 1203B (e.g., text 1206 or icons 1209A-1209D) indicating operations that can be performed by the active instrument 212. Figure 6A As shown, icons 1209A-1209B are also activated and highlighted (e.g., shown in white instead of a gray shade), making the data included in icons 1209A-1209B easily visible to the user. Simultaneously, the dynamic boundary 5509 of the central segment 1117 is deactivated and displayed in a darker color (e.g., a gray shade), thus indicating the presence of data in the display area. Figure 6A When the interactive menu 1110A is active, the device 212 (e.g., camera) is not actively controlled by HID 226, 228.

[0091] In comparison, such as Figure 6B As shown in the interactive menu 1110B, when the camera input device 1502 is activated, visual elements (e.g., grayscale) around at least a portion of the dynamic boundaries 5503 and 5506 of the left segment 1115 and right segment 1120 can be deactivated. Figure 6BAs shown, both the dynamic boundary 5503 of the left segment 1115 and the dynamic boundary 5506 of the right segment 1120 are shaded, thus indicating the display. Figure 6B When the interactive menu 1110B is accessed, the device 212 identified in the left segment 1115 and right segment 1120 is not currently actively controlled by HIDs 226 and 228. As described above, presenting the dynamic boundaries 5503 and 5506 in this manner can help signal to the user that the left HID 226 and right HID 228 may have disengaged and therefore may not be used to control the device 212 indicated in the left segment 1115 and right segment 1120, respectively. Simultaneously, the dynamic boundary 5509 of the central segment 1117 can be changed to be activated and highlighted (e.g., shown in white), thereby indicating the device 212 displayed. Figure 6B When the interactive menu 1110B is accessed, the instrument 212 (e.g., a camera) is actually being actively controlled by HIDs 226, 228. Presenting the dynamic boundary 5509 of the central section 1117 in this manner helps signal to the user that the left HID 226 and right HID 228 are now engaged with the instrument 212 (e.g., a camera) identified and described in the central section 1117. While the activation and deactivation of the dynamic boundaries 5503, 5506, and 5509 are related to... Figures 6A to 6B It is described as being displayed in white or dark gray, but it should be understood that the activation and deactivation of dynamic boundaries 5503, 5506 and 5509 can be indicated in dynamic boundaries 5503, 5506 and 5509 in a variety of other different ways (e.g., using color, blinking, shadows, etc.), which should not be limited in any other way herein.

[0092] Similarly, when camera input device 1502 is actuated, icons 1209A-D in left segment 1115 and right segment 1120 can be changed to indicate that the operation may no longer be performed because the instruments 212A-212B indicated in left segment 1115 and right segment 1120 have disengaged from HID 226, 228. Figure 6B As shown, icons 1209A-1209D can be shaded with a darker color (such as gray) to dim the data displayed in icons 1209A-1209D. Similarly, the dynamic boundary 1213 on icons 1209A-1209D can be deactivated by coloring the dynamic boundary 1213 a similar dark color, which may or may not be the same shade as the rest of icons 1209A-1209D. The previously presented operations that can be performed by instrument 212 can be replaced by the disengagement indication of instrument 212, and thus, HIDs 226, 228 disengage from the robotic arm 210 attached to instrument 212. The disengagement indication of instrument 212 can be represented using text 1206 and / or icon 1209, where icon 1209 can be an image or rendering.

[0093] Although the activation and deactivation of dynamic boundaries 5503, 5506, 5509, 1213 and icons 1209A-1209D are related to Figures 6A to 6B It is described as being displayed in white or dark gray, but it should be understood that the activation and deactivation of dynamic boundaries 5503, 5506 and 5509, 1213 and icons 1209A-1209D can be indicated in a variety of other different ways (e.g., using color, flashing, shadow, etc.), which should not be limited in any other way in this document.

[0094] Figure 7 Examples of various embodiments of the present disclosure are shown, including an interactive menu 1110 when the upper right input device 1506 of the foot pedal assembly 244 is actuated. For example... Figure 7 As shown, the dynamic boundary 5509 surrounding the central segment 1117 is deactivated (gray shading), while the dynamic boundary 5503 surrounding the left segment 1115 and the dynamic boundary 5506 surrounding the right segment 1120 are activated (depicted in white). The shading of dynamic boundaries 5503, 5506, and 5509 indicates the device 212 actively engaged for control by HIDs 226 and 228. That is, the device 212 indicated in the left segment 1115 and right segment 1120 with activated dynamic boundaries 5503 and 5506 is actively controlled by HIDs 226 and 228. Meanwhile, the device 212 indicated in the central segment 1117 with deactivated dynamic boundary 5509 is not actively controlled by HIDs 226 and 228.

[0095] As described above, the right segment 1120 may include one or more icons 1209C-1209D associated with one or more operations performed by the instrument 212 indicated in the right segment 1120. The icons 1209C-1209D displayed in the right segment 1120 may correspond to the upper right input device 1506, such that actuation of the upper right input device 1506 corresponds to the execution of the operation indicated by the icons 1209C-1209D. When the upper right input device 1506 is actuated, the operation indicated by the icons 1209C-1209D in the right segment 1120 can be performed using the instrument 212 indicated in the right segment 1120. While an operation is being performed (i.e., when the upper right input device 1506 is actuated), the interactive menu 1110 can drastically change the visual elements of the right segment 1120. For example, the colors and shading of the right section 1120 and / or the interactive menu 1110 may even begin to flash to clearly indicate to the user that an operation is being performed using the instrument 212. In some cases, an audible alarm may also be output by the console 240 when the upper right input device 1506 is activated. For example, such an alarm may be very helpful to healthcare providers to ensure that operations are performed at the most accurate and appropriate time to prevent injury to the patient and other serious problems. In this way, the interactive menu 1110 can provide an alarm to the healthcare provider in response to an operation performed using the specific active instrument 212.

[0096] In one implementation, the visual elements of the dynamic boundary 5506 surrounding at least a portion of the right segment 1120 may be activated to varying degrees relative to the color, brightness, or other visual features indicated by other dynamic boundaries of the interactive menu 1110 (i.e., indicated by even brighter colors, such as, for example, fluorescent yellow). When the upper right input device 1506 is actuated, icons 1209C-1209D corresponding to the upper right input device 1506 may display a predefined background color, for example, further highlighting icon 1209 corresponding to the upper right input device 1506 to indicate that the operation indicated by icon 1209 is being performed. For example, icon 1209 may display a yellow background color when the upper right input device 1506 is actuated. In some cases, when the upper right input device 1506 is actuated, the background color 1803 of the entire right segment 1120 may be set to a predefined background color 1803. For example, when the upper right input device 1506 is activated, the right segment 1120 can be changed to a transparent yellow background color 1803, and the transparency can provide readability for the text 1206 included in the right segment 1120.

[0097] In one implementation, the upper right input device 1506 may correspond to the top operation icon 1209C, and the lower right input device may correspond to the bottom operation icon 1209D. In this example, in response to the actuation of the upper right input device 1506, the bottom operation icon 1209D may disappear completely, except that the background color is applied to the entire area of ​​the right segment 1120 and its boundary 5506. That is, another operation is no longer available, and therefore it may no longer appear on the user interface. Similarly, in response to the actuation of the lower right input device, the top operation icon 1209C may disappear completely or no longer appear on the user interface.

[0098] Figures 8A to 8C Various implementations of interactive menus 1110A-1110C based on whether the left instrument switch input device 1508 is actuated are illustrated. For example... Figures 8A to 8C As shown, the dynamic boundary 5509 surrounding the central segment 1117 is deactivated (gray shading), while the dynamic boundary 5503 surrounding the left segment 1115 and the dynamic boundary 5506 surrounding the right segment 1120 are activated (depicted in white). The shading of dynamic boundaries 5503, 5506, and 5509 indicates the device 212 actively engaged for control by HIDs 226 and 228. That is, the device 212 indicated in the left segment 1115 and right segment 1120 with activated dynamic boundaries 5503 and 5506 is actively controlled by HIDs 226 and 228. Meanwhile, the device 212 indicated in the central segment 1117 with deactivated dynamic boundary 5509 is not actively controlled by HIDs 226 and 228.

[0099] Figures 8A to 8C An example is shown where, via actuation of the left instrument switch input device 1508, the console 240 can receive instrument control switch commands. The instrument control switching command can be an instruction to switch control of HID 226, 228 from the currently active instrument 212 (indicated in the left segment 1115 or right segment 1120) to the inactive instrument 212 (indicated in the inactive area element 1112). Figures 8A to 8C In this process, the instrument control switch can trigger the console 240 to disengage the left HID 226 from the control connection to the robotic arm 210 of the instrument 212A indicated in the left section 1115, and to engage the left HID 226 with the robotic arm 210 connected to the instrument 212D indicated in the inactive area element 1112. Specifically, Figure 8A An embodiment of the interactive menu 1110A is shown when the left instrument switch input device 1508 of the foot pedal assembly 244 is not actuated. Figure 8B An embodiment of the interactive menu 1110B is shown when or after the left instrument switch input device 1508 is actuated, and Figure 8CAn embodiment of the interactive menu 1110C is shown after the left instrument switch input device 1508 is actuated and released.

[0100] Figure 8A The first state of the interactive menu 1110A is shown when the left segment 1115 indicates the active device 212A and the inactive element 1112 indicates the inactive device 212D. In this first state (which may be a first moment), the active device 212A is dockable to the active robotic arm 210, and the inactive device 212D is dockable to the inactive robotic arm 210. In this example, the left HID 226 can be assigned to control both the active and inactive robotic arms 210. The assignment can be preset by the system as the default assignment, or it can be manipulated by the user using a user interface, for example, on the handrest of the console 240.

[0101] Figure 8B The second state of the interactive menu 1110B is shown when the left instrument switch input device 1508 is actuated or shortly thereafter. In the second state (which may be a second moment after the first moment), the interactive menu 1110B can remove the instrument indicator 1203A describing instrument 212A from the left section 1115 and remove the instrument indicator 1203D describing inactive instrument 212D from the inactive area element 1112.

[0102] Figure 8C The third state of the interactive menu 1110C is shown after the left instrument switch input device 1508 is actuated (and in some cases, released) and after the console 240 receives an instrument control switch command. In this third state (which may be a third moment after the second moment), instrument 212D can become active instrument 212, and instrument 212A can become inactive instrument 212A. Then, the left segment 1115 can indicate instrument 212D and include an instrument indicator 1203D describing instrument 212D. Then, the inactive segment element 1112 can indicate instrument 212A and include an instrument indicator 1203A. In this state, the left HID 226 can engage with the active robotic arm 210 to control instrument 212D.

[0103] It should be understood that different tool types can have different operations. For example, energy tools may have cutting and condensing functions, while non-energy tools may not have these operations. In other examples, a tool may have only one available operation, two operations, or no available operations. In these examples, the operation icons 1209A-1209D mentioned above may appear or disappear as the tool is swapped into or out of the left segment 1115 or right segment 1120. Furthermore, even for tools with one or more available operations, operation icons 1209A-1209D are not invoked in response to the tool being deactivated and shown in the inactive element 1112. That is, in one example, operation icons 1209A-1209D are invoked only when the tool is selected and shown in the left segment 1115 or right segment 1120. In one example, if a tool has only one available operation and it is in the left segment 1115 or right segment 1120, it may only have the invoked operation icon 1209A.

[0104] F. Status Indicator

[0105] In some implementations, the interactive menu 1110 may include status indicators that indicate the current status of the robot arm 210 and / or instrument 212 associated with the left segment 1115, the center segment 1117, the right segment 1120, and / or the inactive area element 1112. The status indicators may be displayed in the interactive menu 1110 according to their priority.

[0106] Figure 9A A diagram 2000 depicts various examples of status indicators 2001-2006 based on status priority 2007, illustrating various embodiments of the present disclosure. While Figure 9 shows six different types of status indicators 2001-2006 as described, the interactive menu 1110 may include any number of status indicators 2001-2006. In one embodiment, status indicators 2001-2006 may be adjusted to higher or lower priorities, for example, by the user based on user input received at console 240. For example, status indicators 2001-2006 may be icons, and diagram 2000 may be displayed on a display of console 240 (e.g., display device 242 or touchscreen device). In this case, the user can drag and drop status indicators 2001-2006 into diagram 2000 at different priorities.

[0107] Each of the status indicators 2001-2006 may include text and / or an icon that, when presented in the interactive menu 1110, indicates the status of the device 212 indicated in segments 1115, 1117, or 1120 near status indicators 2001-2007. For example, status indicator 2001 may include text and / or an icon indicating clamping at the bedside, which, when presented in the interactive menu 1110, indicates that the robotic arm 210 attached to the device 212 is engaged or disengaged at the bedside (e.g., via a patient support platform). Status indicator 2002 may include the text “Loading…”, which, when presented in the interactive menu 1110, indicates that a device loading process on the robotic arm 210 has been initiated. Status indicator 2003 may include the text “Insert” and / or include an icon with a downward arrow. When presented in interactive menu 1110, status indicator 2003 indicates that instrument 212 is being inserted into robotic arm 210. Status indicator 2004 may include the text "Insert Limit" and / or include an icon with a downward arrow with a horizontal line. When presented in interactive menu 1110, status indicator 2004 indicates that a particular instrument 212 has reached its insertion limit (i.e., it cannot be inserted further into the patient). Status indicator 2005 may include the text "Retraction Limit" and / or include an icon with an upward arrow with a horizontal line. When presented in interactive menu 1110, status indicator 2005 indicates that a particular instrument 212 has reached its retraction limit (i.e., it cannot be retracted further). Status indicator 2006 may include the text "Match Grip," which, when presented in interactive menu 1110, indicates that instrument 212 may require manual alignment and mating with the grip of robotic arm 210. The status indicator 2006 may be a call to action of a medical provider immersed in the display device 242 or present in the room with the surgical robot system, to perform matching actions before the HID 226, 228 is able to control the robotic arm 210 and the instrument 212.

[0108] In some implementations, status indicators 2001-2006 may be ordered according to status priority 2007, because only one status indicator 2001-2006 may be presented in the interactive menu 1110 at a time for each segment 1115, 1117, 1120 and for inactive element 1112. In this way, when the robot arm 210 or instrument 212 associated with segment 1115, 1117, or 1120 or inactive element 1112 has multiple state changes making multiple status indicators 2001-2006 applicable, only one status indicator 2001-2006 may be presented for that segment 1115, 1117, or 1120 or inactive element 1112. The displayed status indicators 2001-2006 may be determined based on the status priority 2007 shown in FIG9. For example, when the robotic arm 210 or instrument 212 associated with the left segment 1115 has two state updates corresponding to state indicators 2001 and 2004, the left segment 1115 may only present one state indicator 2001 (e.g., as an icon, text, or other indicator above the left segment 1115). This may be because state indicator 2001 has a higher state priority 2007 than state indicator 2004.

[0109] Figure 9B Examples of status indicators 2001-2006 are depicted, which can be displayed near segments 1115, 1117, and 1120 in active area element 1111. For example... Figure 9BAs shown, status indicator 2001 may be included in sections 1115, 1117, and 1120 to indicate that instruments 212A-212B represented by sections 1115, 1117, and 1120 are engaged or disengaged at the bedside. Status indicator 2002 may be included in sections 1115, 1117, and 1120 to indicate that instruments 212A-212B represented by sections 1115, 1117, and 1120 are being loaded onto the robotic arm 210. Status indicator 2003 may be included in sections 1115, 1117, and 1120 to indicate that instruments 212A-212B represented by sections 1115, 1117, and 1120 are being inserted into the robotic arm 210. Status indicator 2004 may be included in sections 1115, 1117, and 1120 to indicate that instruments 212A-212B represented by sections 1115, 1117, and 1120 have reached their insertion limit. Status indicator 2005 may be included in sections 1115, 1117, and 1120 to indicate that instruments 212A-212B represented by sections 1115, 1117, and 1120 have reached their retraction limit. Status indicator 2006 may be included with sections 1115, 1117, and 1120 to instruct a healthcare provider immersed in display device 242 or present in the room with the surgical robot system to perform a matching action before HID 226, 228 can control the robotic arm 210 and instruments 212.

[0110] Figures 10A to 10D Examples of the positioning of one or more status indicators 2001-2006 in an interactive menu 1110 according to various embodiments of the present disclosure are illustrated. Figure 10A The example shows status indicators 2001-2006 located near the top of right segment 1120. Figure 10B The example shows status indicators 2001-2006 located near the top of the left segment 1115. Figure 10C Examples are shown for status indicators 2001-2006 located near the center segment 1117, and... Figure 10D Examples of status indicators 2001-2006 are shown located near the left segment 1115, the center segment 1117, the right segment 1120, and the inactive element 1112. Figure 10D It is illustrated that one or more of the left segment 1115, the center segment 1117, the right segment 1120, and the inactive element 1112 may each have associated status indicators 2001-2006. Figures 10A to 10DOnly one status indicator 2001-2006 can be displayed at a time above each of the left segment 1115, center segment 1117, right segment 1120, and inactive element 1112. Furthermore, each of the left segment 1115, center segment 1117, right segment 1120, and inactive element 1112 can each display one status indicator 2001-2006 at a time, as shown below. Figure 10D As shown.

[0111] like Figures 10A to 10D As shown, status indicators 2001-2006 can be positioned near the left segment 1115, the center segment 1117, the right segment 1120, and / or the inactive area element 1112 of the device 212 to which status indicators 2001-2006 are applied. In other words, status indicators 2001-2006 can indicate the status of device 212, and status indicators 2001-2006 can be positioned in the interactive menu 1110 near the segments 1115, 1117, 1120, and / or the inactive area element 1112 of the device 212.

[0112] Although Figures 10A to 10D Status indicators 2001-2006 are shown as positioned above the left segment 1115, the center segment 1117, the right segment 1120, and / or the inactive element 1112. However, it should be understood that status indicators 2001-2006 can be positioned anywhere near the left segment 1115, the center segment 1117, the right segment 1120, and / or the inactive element 1112. For example, status indicators 2001-2006 can be positioned below the left segment 1115, the center segment 1117, the right segment 1120, and / or the inactive element 1112; status indicators 2001-2006 can be positioned near the left segment 1115, the center segment 1117, the right segment 1120, and / or the inactive element 1112; and so on.

[0113] G. Stage indicator

[0114] In embodiments where the surgical robotic system includes a patient platform or table on which the patient lies during the procedure, the display device 242 may additionally display table indicators with an interactive menu 1110. Figure 11 An example of a stage indicator 3000 displayed on display device 242 above interactive menu 1110 is shown according to various embodiments of the present disclosure. Similar to interactive menu 1110, stage indicator 3000 may be superimposed on image 1103 displayed on display device 242.

[0115] In one implementation, the table indicator 3000 may typically display information related to the types of adjustments that can be made to the orientation and / or height of the patient table. Figure 11 In the example shown, the table indicator 3000 includes three components 3004, 3006, and 3009, each corresponding to a different type of adjustment that can be made to the patient table. For example, component 3004 may describe tilt adjustment of the patient table. Tilt adjustment may refer to adjusting the angle of the patient table in the vertical direction along the length of the table. For example, the patient table may be tilted in a head-up (head raised) orientation or a feet-up (feet raised) orientation. Tilt adjustment may also refer to adjusting the angle of the patient table in the horizontal direction along the width of the table. For example, the patient table may be tilted in a left-raised orientation or a right-raised orientation. Component 3006 may describe trend adjustment of the patient table. Trend adjustment may refer to longitudinal or lateral movement of the patient table. For example, trend adjustment may involve sliding the patient table back and forth in a linear direction along the length and / or width of the patient. Component 3009 may describe height adjustment of the patient table. For example, height adjustment may involve sliding the patient table up and down in a linear vertical direction.

[0116] Although Figure 11 The table indicator 3000 shown includes only three components 3004, 3006, and 3009; however, it should be understood that the table indicator 3000 may include any number of components, each corresponding to a different type of adjustment that can be made to the patient table. In some embodiments, two components 3004 may be present, describing different tilt adjustments. For example, there may be a first component 3004 describing tilt adjustments along the length of the table (i.e., head elevation or foot elevation), and a second component 3004 describing tilt adjustments along the width of the table (i.e., left-side elevation or right-side elevation).

[0117] Each of components 3004, 3006, and 3009 may display text, icons, images, and / or other user interface components that identify the type of adjustment and / or describe the settings associated with the adjustment. Component 3004, describing the tilt adjustment, may include a graphical representation 3012 of a patient table, for example, depicting the patient table and a patient positioned on it. The graphical representation 3012 of the patient table may be tilted to match the tilt of the actual patient table (i.e., proportional to it). The graphical representation 3012 of the patient table may be oriented in the length or width direction of the patient to indicate the direction of the tilt indicated in component 3004. For example, when the tilt adjustment is made along the length of the table, the graphical representation 3012 of the patient table may be oriented to depict the length of the patient. When the tilt adjustment is made along the width of the table, the graphical representation 3012 of the patient table may be oriented to depict the width of the patient. The graphical representation 3012 of the patient table may also include an arched, bi-lateral arrow positioned horizontally above the patient table, indicating the direction of the tilt adjustment described in component 3004. Component 3004 may also include text describing a setting 3015 associated with tilt adjustment. For example, the text describing setting 3015 may specify the angle (in degrees) of the patient table tilt.

[0118] Component 3006 may similarly include a graphical representation 3018 of the patient table, wherein the trend (i.e., relative translation) of the graphical representation 3015 of the patient table is proportional to the trend of the actual patient table. The graphical representation 3018 of the patient table may be oriented along the length or width of the patient to indicate the direction of the trend indicated in component 3006. The graphical representation 3018 of the patient table may also include a bidirectional arrow positioned horizontally above the patient table, indicating the direction of the trend adjustment described in component 3006. Component 3006 may also include text describing a setting 3021 associated with the trend adjustment. For example, the text describing setting 3021 may describe the distance the patient table has been translated (or trend-moved) longitudinally or laterally relative to a starting / default position.

[0119] Component 3009 may similarly include a graphical representation 3024 of the patient table, wherein the height adjustment of the graphical representation 3015 of the patient table is proportional to the relative height of the actual patient table. The graphical representation 3023 of the patient table may be oriented along the length or width of the patient. The graphical representation 3024 of the patient table may also include a bidirectional arrow positioned vertically on the side of the patient table, indicating the direction of the height adjustment described in component 3009. Component 3009 may also include text describing a setting 3027 associated with the height adjustment. For example, the height describing setting 2037 may describe the height of the patient table's vertical translation (or trend movement) relative to the starting / default position.

[0120] Although Figure 11The console indicator 3000 is shown as being positioned directly above the interactive menu 1110, but it should be understood that the console indicator 3000 can be positioned anywhere on the display device 242. Figure 11 The size of the table indicator 3000 is depicted as significantly smaller than the size of the interactive menu 1110. However, it should be understood that in different embodiments, the size of the table indicator 3000 may vary to be the same size as, comparable in size to, or larger than the interactive menu 1110. The positioning and size of the table indicator 3000 may vary, as long as the positioning and size of the table indicator 3000 do not obstruct the view of the patient's anatomy interacting with instruments 212A-212B as displayed on display device 242.

[0121] In one implementation, a table indicator 3000 may be displayed either always or only in response to user input received at one or more input devices on the robotic system, wherein the input devices may be located at console 240 or another device for controlling the robotic system. For example, this device may be a remote controller 3002, which includes, for example, one or more input devices 3003, some of which may be used to adjust the patient table as described herein. For example, one or more first input devices 3003 at remote controller 3002 may control tilt adjustment in the vertical direction, one or more second input devices 3003 at remote controller 3002 may control tilt adjustment in the horizontal direction, one or more third input devices 3003 at remote controller 3002 may control trend adjustment, and one or more fourth input devices 3003 at remote controller 3002 may control height adjustment. The user may accordingly select input device 3002 or otherwise interact with the input devices to control adjustments to the patient table.

[0122] In the implementation scheme, the table indicator 3000 is displayed only when the input device 3003 is used to adjust the patient table settings. When the actual patient table settings are also adjusted, the information in components 3004, 3006, and / or 3009 can be adjusted in real time. For example, when the input device 3003 is selected to change the patient table tilt from a head-up orientation to a foot-up orientation, the graphical representation 3012 of the patient table can be similarly updated to depict the foot-up orientation. Similarly, the text describing the settings 3015 (which records the table angles) can also be updated accordingly.

[0123] In one implementation, the display of the station indicator 3000 on the display device 242 may cease when the input device 3003 is no longer manipulated, selected, or otherwise interacted with to adjust the patient station settings. In another implementation, the station indicator 3000 may be displayed for a predetermined period of time after the patient station settings have been adjusted or after the input device 3003 is no longer used to adjust the station settings. The predetermined period of time may be, for example, between 2 and 5 seconds, or it may be 3 seconds. In this way, the station indicator 3000 may only be displayed when the patient station settings may be relevant to setting up a robotic system or a healthcare provider performing procedures on the patient. This, in turn, prevents the healthcare provider from being distracted by the station indicator 3000 for unnecessarily long periods.

[0124] H. Exemplary Methods .

[0125] Figure 12 A flowchart illustrating method 2200 performed by a surgical robotic system is provided. Specifically, method 2200 may be performed by a processor in or coupled to console 240.

[0126] At step 2203, method 2200 includes displaying an interactive menu 1110 at the console 240 of the surgical robot system, comprising an active area element 1111 and an inactive area element 1112. In one embodiment, the active area element 1111 includes a left segment 1115, a right segment 1120, and a central segment 1117.

[0127] At step 2206, method 2200 includes indicating a first instrument 212, coupled to a first robotic arm 210 connected to a surgical robot system and controlled by the left HID 226 of console 240, in the left segment 1115 of active area element 1111. At step 2209, method 2200 includes indicating a second instrument 212, coupled to a second robotic arm 210 connected to a surgical robot system and controlled by the right HID 228 of console 240, in the right segment 1120 of active area element 1111. At step 2212, method 2200 includes indicating data associated with a camera of a third robotic arm 210 connected to the surgical robot system, in the central segment 1117 of active area element 1111. At step 2215, method 2200 includes indicating a third instrument 212, coupled to a fourth robotic arm 210, in inactive area element 1112. In one implementation, when an instruction is received to switch control of the right HID 228 from the second instrument 212 to the third instrument 212, the right HID 228 can be engaged to control the fourth robotic arm 210.

[0128] Figure 13A flowchart illustrating method 2300 performed by a surgical robotic system is provided. Specifically, method 2300 may be performed by a processor in or coupled to console 240.

[0129] At step 2303, method 2300 includes displaying an interactive menu 1110, including active area elements 1111 and inactive area elements 1112, at a console 240 of the surgical robot system. At step 2306, method 2300 includes indicating, in the active area element 1111, data associated with one or more instruments 212 respectively coupled to the one or more active robotic arms 210 of the surgical robot system, based on whether one or more active robotic arms 210 are assigned to be controlled by a first HID 226 or a second HID 228 of the console 240. In one embodiment, the one or more active robotic arms 210 are actively controlled by at least one of the first HID 226 or the second HID 228. At step 2309, method 2300 includes indicating, in the inactive area element 1112, data associated with instruments 212 coupled to the one or more inactive robotic arms 210 of the surgical robot system. The one or more inactive robotic arms 210 are disengaged from the first HID 226 and the second HID 228.

[0130] Figure 14 A flowchart illustrating method 2400 performed by a surgical robotic system is provided. Specifically, method 2400 may be executed by a processor in console 240 or coupled to console 240.

[0131] At step 2403, method 2400 includes displaying an interactive menu 1110 comprising an active area element 1111 and an inactive area element 1112, wherein the active area element may include a first segment 1115, a central segment 1117, and a second segment 1120. The first segment 1115 may indicate that a first device 212 is controlled by a first HID 226, and the second segment 1120 indicates that a second device 212 is controlled by a second HID 228. The inactive area element 1112 may indicate that a third device 212 is disengaged from the first HID 226 and the second HID 228. At step 2406, method 2400 includes receiving an instruction to switch control of the second HID 228 from the second device 212 to the third device 212. At step 2409, method 2400 includes, in response to receiving the instruction, indicating the third device 212 in the second segment 1120 and indicating the second device 212 in the inactive area element 1112.

[0132] I. Implementation system and terminology.

[0133] Figure 15This is a schematic diagram illustrating the electronic components of a surgical robot system according to some implementation schemes.

[0134] The surgical robotic system includes one or more processors 380, which store information for performing any of the methods described herein (e.g., regarding...). Figure 1 Figure 2 Figure 3 , Figure 4 Figure 5 Figure 6A , Figure 6B , Figure 7 , Figure 8A , Figure 8B , Figure 8C Figure 9 Figure 10A , Figure 10B , Figure 10C , Figure 10D , Figure 11 , Figure 12 and Figure 13 The computer-readable storage medium 382 (e.g., a computer memory device such as random access memory, read-only memory, static random access memory, and non-volatile memory, as well as other storage devices such as hard disk drives, optical disks, magnetic tape recording, or any combination thereof) communicates with instructions for the described operations. One or more processors 380 also communicate with an input / output controller 384 (via a system bus or any suitable circuitry). The input / output controller 384 receives sensor data from one or more sensors 388-1, 388-2, etc., and relays the sensor data to one or more processors 380. The input / output controller 384 also receives instructions and / or data from one or more processors 380 and relays the instructions and / or data to one or more actuators, such as first motors 387-1 and 387-2, etc. In some embodiments, the input / output controller 384 is coupled to one or more actuator controllers 386 and provides instructions and / or data to at least a subset of the one or more actuator controllers 386, which in turn provides control signals to the selected actuators. In some implementations, one or more actuator controllers 386 are integrated with an input / output controller 384, and the input / output controller 384 provides control signals directly to one or more motors 387-1 and 387-2, etc. (in the absence of a separate actuator controller). Although Figure 15 It is shown that there is an actuator controller 386 (e.g., an actuator controller for the entire surgical robot system; in some embodiments, additional actuator controllers may be used (e.g., one actuator controller per actuator, etc.). In some embodiments, one or more processors 380 communicate with one or more displays 381 for displaying information as described herein.

[0135] It should be noted that, as used herein, other variations of the term "connection" or the word "linkage" can indicate an indirect or direct connection. For example, if a first component is "connected" to a second component, the first component may be indirectly connected to the second component or directly connected to the second component via another component.

[0136] The function of determining whether an instrument is within or outside the surgical field of view provided by a camera or endoscope and rendering one or more indicators representing the position or orientation of one or more medical instruments described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" means any available medium accessible by a computer or processor. By way of example, and not limitation, such a medium may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disk read-only memory (CD-ROM) or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. It should be noted that a computer-readable medium may be tangible and non-transitory. As used herein, the term "code" may mean software, instructions, code, or data executable by a computing device or processor.

[0137] The methods disclosed herein include one or more steps or actions for implementing the described methods. The method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims unless the correct operation of the described method requires a specific order of steps or actions.

[0138] As used herein, the term "multiple" means two or more. For example, multiple components indicates two or more components. The term "determine" encompasses a variety of actions, and therefore, "determine" can include calculation, operation, processing, derivation, investigation, lookup (e.g., searching in a table, database, or another data structure), ascertainment, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0139] Unless otherwise explicitly stated, the phrase “based on” does not mean “based on only”. In other words, the phrase “based on” describes both “based on only” and “based on at least”.

[0140] As used herein, the term “exemplary” means “serving as an example, instance or illustration” and does not necessarily indicate any preference or superiority of the example relative to any other configuration or specific implementation.

[0141] As used herein, the term “and / or” covers any combination of the listed elements. For example, “A, B and / or C” includes the following sets of elements: A only, B only, C only, A and B without C, A and C without B, B and C without A, and combinations of all three elements A, B and C.

Claims

1. A surgical robot system, comprising: Multiple robotic arms, including active robotic arms and inactive robotic arms; A control console, communicatively connected to the robotic arm, the control console comprising: First HID and second HID; and The display is configured to: Displays an interactive menu that includes active and inactive area elements; Based on whether the active robotic arm is assigned to be controlled by the first HID or the second HID, data associated with the first instrument coupled to the active robotic arm is indicated in the active area element; and The inactive area element indicates data associated with a second instrument connected to the inactive robotic arm, wherein the inactive robotic arm is disengaged from the first HID and the second HID.

2. The surgical robot system according to claim 1, wherein, The plurality of robotic arms includes a first movable robotic arm, a second movable robotic arm, and a third movable robotic arm, wherein the active area element includes a first segment, a second segment, and a central segment, and wherein, in order to indicate the data associated with the first instrument and the data associated with the second instrument in the active area element, the display is further configured to: The first instrument, which is coupled to the first active robotic arm and controlled by the first HID, is indicated in the first segment of the active area element; The second segment of the active area element indicates a third instrument coupled to the second active robotic arm and controlled by the second HID; and The central segment of the active area element indicates data associated with the camera connected to the third active robotic arm.

3. The surgical robot system according to claim 2, wherein, The display is further configured to indicate, in the inactive area element, the second instrument coupled to the inactive robotic arm, wherein, when an instruction is received to switch control of the second HID from the third instrument to the second instrument, the inactive robotic arm is assigned to be controlled by the second HID, and wherein the inactive area element is located near the second segment of the active area element.

4. The surgical robot system according to claim 3, wherein, The console also includes a processor connected to the display, wherein the processor is configured to: Receive the instruction to switch control of the second HID from the third device to the second device; In response to receiving the instruction, the display is configured as follows: In the second segment of the active area element, the indication of the third instrument is replaced with the indication of the second instrument; and In the inactive area element, the indication of the second instrument is replaced with the indication of the third instrument.

5. The surgical robot system according to claim 2, wherein, The data associated with the camera includes at least one of the following: the orientation of the camera, the type of the camera, or the identifier of the third active robotic arm to which the camera is attached.

6. The surgical robot system according to claim 2, wherein, The data associated with the first device or the data associated with the second device includes at least one of the following: device identification, one or more operations to be performed using the device, background color indicating the status of the one or more operations, operation indicator color identifying the one or more operations, identifier of the robot arm to which the device is attached, or color indicating whether the device is engaged by the first HID or the second HID.

7. The surgical robotic system according to any one of claims 1 to 6, wherein, The active robotic arm is coupled to a camera configured to capture images, and the interactive menu is overlaid on the images.

8. The surgical robot system according to claim 7, wherein, The camera is an endoscope or a laparoscopic device.

9. The surgical robotic system according to any one of claims 1 to 8, wherein, The interactive menu is overlaid on a portion of the image, wherein the size of the interactive menu is less than a predefined percentage of the size of the image, and wherein the interactive menu is positioned near the edge of the image.

10. The surgical robotic system according to any one of claims 1 to 9, wherein, The inactive area element includes at least one of the following: the identifier of the second instrument, or the identifier of the inactive robotic arm to which the second instrument is attached.

11. The surgical robotic system according to any one of claims 1 to 10, wherein, The console includes a plurality of foot actuators, wherein each of the foot actuators, when actuated, activates at least one of the following: Instrument mode that uses the first instrument to perform the operation. The engagement / disengagement mode that causes the first instrument to disengage from the mobile robotic arm. A peephole mode that controls the movement of a third robotic arm connected to the camera. When a machine switching operation is manually performed at the active robotic arm, the machine switching mode of the active robotic arm is locked, or The control of the second HID is switched from the active instrument to the switching command of the second instrument.

12. The surgical robotic system according to any one of claims 1 to 10, wherein, The display is further configured to indicate one or more status indicators near the active or inactive area element according to status indicator priority, wherein the one or more status indicators indicate the current state of at least one of the robotic arm or instrument.

13. A method performed by a surgical robotic system, comprising: An interactive menu including active and inactive area elements is displayed at the console of the surgical robot system, wherein the active area elements include a left segment, a center segment, and a right segment; The left segment of the active area element indicates a first instrument that is coupled to the first robotic arm of the surgical robot system and controlled by the left haptic interface device (HID) of the console; The right segment of the active area element indicates a second instrument that is coupled to the second robotic arm of the surgical robot system and controlled by the right HID of the console; The central segment of the active area element indicates data associated with a camera of a third robotic arm coupled to the surgical robotic system; and The inactive area element indicates a third instrument connected to the fourth robotic arm, wherein, when an instruction is received to switch control of the right HID from the second instrument to the third instrument, the right HID is engaged to control the fourth robotic arm.

14. The method according to claim 13, wherein, The camera is an endoscope or a laparoscopic device.

15. The method according to any one of claims 13 to 14, wherein, The data associated with the camera includes at least one of the following: the orientation of the camera, the type of the camera, or the identifier of the fourth robotic arm to which the camera is attached.

16. The method according to any one of claims 13 to 15, wherein, The left segment also indicates at least one of the following: an identifier of the first instrument, one or more operations to be performed using the first instrument, a background color indicating the status of the one or more operations, an operation indicator color indicating the one or more operations, an identifier of the first robotic arm to which the first instrument is attached, or a color indicating whether the first instrument is engaged by the left HID.

17. The method according to any one of claims 13 to 16, wherein, The right segment also indicates at least one of the following: an identifier of the second instrument, one or more operations to be performed using the second instrument, a background color indicating the status of the one or more operations, an operation indicator color indicating the one or more operations, an identifier of the second robotic arm to which the second instrument is attached, or a color indicating whether the second instrument is engaged by the right HID.

18. The method according to any one of claims 13 to 17, wherein, The interactive menu is overlaid on a portion of the image acquired by the camera, wherein the size of the interactive menu is less than a predefined percentage of the size of the image, and wherein the interactive menu is positioned near the edge of the image.

19. The method according to any one of claims 13 to 18, further comprising: Receive the instruction to switch control of the right HID from the second device to the third device; After receiving the instruction, the method further includes: In the right segment of the active area element, the indication of the second instrument is replaced by the indication of the third instrument; as well as In the inactive area element, the indication of the third device is replaced with the indication of the second device.

20. The method according to any one of claims 13 to 19, wherein, The inactive area element includes at least one of the following: the identifier of the third instrument, or the identifier of the third robotic arm to which the third instrument is attached.

21. The method according to any one of claims 13 to 20, wherein, The console includes a plurality of foot actuators, wherein each of the foot actuators, when actuated, activates at least one of the following: A mechanism mode that uses one or more instruments, each connected to a robotic arm, to perform operations. A clutch mode that uses one or more devices respectively connected to the first robotic arm and the second robotic arm to perform clutch operations. A peephole mode that controls the movement of a robotic arm attached to a camera. When a machine switching is manually performed at one of the robotic arms, the machine switching mode of one or more active robotic arms is locked, or A switching instruction to switch the control of the right HID from the second device to the third device.

22. A surgical robotic system, comprising: Multiple robotic arms, the multiple robotic arms including: A first robotic arm, the first robotic arm being connected to a first instrument; A second robotic arm, the second robotic arm being connected to a second instrument; and A third robotic arm, the third robotic arm being connected to a third instrument; and A control console, communicatively connected to the robotic arm, the control console comprising: A first haptic interface device (HID) is configured to control the first robotic arm; A second HID, configured to control either the first or the second robotic arm; A display configured to show an interactive menu including active and inactive area elements, wherein the active area elements include a first segment, a central segment, and a second segment, wherein the first segment indicates that the first device is controlled by the first HID, wherein the second segment indicates that the second device is controlled by the second HID, and wherein the inactive area elements indicate that the third device is disengaged from the first and second HIDs; and A processor, coupled to the display and configured to receive an instruction to switch control of the second HID from the second device to the third device. In response to the instruction, the display is configured to indicate the third device in the second segment and the second device in the inactive area element.

23. A method performed by a surgical robotic system, comprising: An interactive menu, including active and inactive area elements, is displayed at the console of the surgical robot system. Based on whether one or more robotic arms are assigned to be controlled by a first haptic interface device (HID) or a second HID of the console, data associated in the active area element with one or more instruments respectively coupled to one or more robotic arms in the surgical robot system is indicated, wherein the one or more robotic arms are actively controlled by at least one of the first HID or the second HID; and The inactive area element indicates data associated with instruments connected to one or more inactive robotic arms in the surgical robot system, wherein the one or more inactive robotic arms are disengaged from the first HID and the second HID.

24. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a surgical robot system, cause the processor to: An interactive menu, including active and inactive area elements, is displayed at the console of the surgical robot system. Based on whether one or more robotic arms are assigned to be controlled by a first haptic interface device (HID) or a second HID of the console, data associated with one or more instruments of one or more robotic arms respectively coupled to the surgical robot system is indicated in the active area element, wherein... The one or more active robotic arms are actively controlled by at least one of the first HID or the second HID; as well as The inactive area element indicates data associated with instruments connected to one or more inactive robotic arms in the surgical robot system, wherein the one or more inactive robotic arms are disengaged from the first HID and the second HID.

Images