Handheld user interface device for surgical robots

Abstract

This document discloses a mobile interface device for controlling a robot-assisted surgical system. The mobile interface device provides the surgeon with a direct view of the surgical robot system and allows the surgeon to easily and intuitively select, control, or manipulate various target components of the surgical robot system. The mobile interface device captures real-time images of the surgical robot system to automatically identify the robotic arm appearing at the center of the captured real-time image as the target component selected by the surgeon. Based on the current pose or position of the target component, the mobile interface device generates a list of target poses and control options. The surgeon can select control options to select a target pose and manipulate the target component, thereby commanding the selected robotic arm to perform robot-assisted movement from the current pose to the target pose.

Description

Technical Field

[0001] This subject matter relates generally to robotics and surgical systems, and more specifically to screen-based interfaces for identifying and manipulating components, such as surgical robotic arms, or for identifying and manipulating other components, such as tables of surgical robotic systems for preparing or performing minimally invasive surgical procedures. Background Technology

[0002] Minimally invasive surgery (MIS), such as laparoscopic surgery, involves techniques designed to minimize tissue damage during surgical procedures. For example, laparoscopic surgery typically involves making multiple small incisions inside the patient (e.g., in the abdomen) and introducing one or more surgical instruments (e.g., end effectors and endoscopes) through these incisions. The introduced surgical instruments can then be used to perform the surgical procedure, with visualization aids provided by the endoscope.

[0003] Generally speaking, surgical instruments (MIS) offer multiple beneficial effects, such as reducing patient scarring, alleviating patient pain, shortening patient recovery time, and reducing medical costs associated with patient recovery. Recent technological advancements have allowed for the use of robotic systems to perform more MIS. These robotic systems include one or more robotic arms used to manipulate surgical instruments based on commands from a remote operator. The robotic arms can support various devices at their distal ends, such as surgical end effectors, imaging equipment, and cannulas for providing access to patient cavities and organs. In robotic MIS systems, it may be desirable to establish and maintain a high degree of positioning accuracy for surgical instruments supported by the robotic arms.

[0004] Existing robot-assisted surgical systems typically consist of a surgeon's console from which the surgeon selects and manipulates robotic arms and devices attached to them. The robot controller requests the surgeon to select a robotic arm by presenting identifiers such as numbers associated with multiple robotic arms or their mounting locations on the surgeon's console. Because there is a lack of natural spatial relationship between the robotic arm seen by the surgeon at the surgeon's console and the robotic arm directly observed, the surgeon may select the wrong robotic arm. Moving the wrong robotic arm can cause serious injury to the patient, other equipment, or bedside personnel. There is a need for an interface that allows surgeons to control the surgical robot system in a more intuitive way, thereby minimizing errors when performing robot-assisted surgeries. Summary of the Invention

[0005] This paper discloses a mobile interface device for controlling a robot-assisted surgical system, also known as a surgical robot system, which is a software-controlled electromechanical system designed for surgeons to perform minimally invasive surgical procedures. The mobile interface device provides the surgeon with a direct and real-time view of the surgical robot system and allows the surgeon to easily and intuitively select, control, or manipulate various target components of the surgical robot system. For example, the mobile interface device can automatically identify a robotic arm located at the center of the surgeon's attention or appearing at the center of an image captured by the interface device as a target component selected by the surgeon. Based on the current pose or position of the target component, the mobile interface device can generate a list of control options. The surgeon can select control options to manipulate the target component, such as commanding the selected robotic arm to perform robot-assisted movement to the desired pose.

[0006] An interface device for a surgical robotic system may include a camera, a processor, a display, and one or more control buttons. The camera captures real-time images or video of the surgical robotic system. The processor executes image processing software on the captured real-time video or images to identify target parts of the surgical robotic system to be controlled using the interface device and determine the initial or current pose of the target parts. The display shows the captured real-time images or video, including the identified target parts of the surgical robotic system. The display may show one or more possible target poses for the target parts. The interface device presents control options to allow the surgeon to select one of the possible target poses and control the target parts. The control buttons can be used to generate input commands to the surgical robotic system to move the target parts from the current pose to the selected target pose.

[0007] In one aspect, the processor can identify possible target poses of a target component by processing captured video or images, thereby identifying one or more objects to which the target component can be engaged. In another aspect, the interface device can receive information about the target pose from a surgical robotic system or via commands input to the interface device by a surgeon. In another aspect, the interface device may include one or more sensors. The sensors can measure the relative positions of various surface points of the surgical robotic system captured by a camera with respect to the interface device in three dimensions. The interface device can process the measurement results from the sensors to aid in identifying the target component, determining the current pose of the target component, or identifying possible target poses of the target component. Attached Figure Description

[0008] To provide a more complete understanding of the invention, the accompanying drawings are provided together with the following description of various aspects and embodiments of the subject matter. The drawings and embodiments are illustrative of the invention and are not intended to limit its scope. It should be understood that those skilled in the art can modify the drawings to generate drawings of other embodiments that will still fall within the scope of the invention.

[0009] Figure 1 This is a drawing view of an exemplary surgical robot system 1 in a surgical setting according to various aspects of the subject matter.

[0010] Figure 2A This is a front view of an interface device according to various aspects of the subject matter, showing a display in which the robotic arm of a surgical robotic system is visualized to enable the identification or selection / target selection of individual parts; and Figure 2B This is a rear view of the interface device according to various aspects of the subject matter, showing the camera, optional 3D camera, and control buttons (which may also be located on the front or side of the device).

[0011] Figure 3 An interface device for controlling a robotic arm to move from an initial pose to a final pose, according to various aspects of the subject matter, is shown.

[0012] Figure 4 This is a block diagram illustrating exemplary hardware components of an interface device according to various aspects of the subject matter.

[0013] Figure 5 This is a block diagram illustrating exemplary hardware components of a surgical robotic system according to various aspects of the subject matter.

[0014] Figure 6 This is a flowchart illustrating a method for controlling a surgical robot system using a mobile device, according to various aspects of the art in this subject matter. Detailed Implementation

[0015] Examples of various aspects and variations of this subject matter are described herein and illustrated in the accompanying drawings. The following description is not intended to limit the invention to these embodiments, but rather to enable those skilled in the art to make and use the invention.

[0016] This paper discloses an interface device that allows surgeons to intuitively control a surgical robotic system, a software-controlled electromechanical system designed to assist surgeons in performing minimally invasive surgical procedures. The interface device allows surgeons, who directly observe the operating room environment, to select, control, or manipulate various components of the surgical robotic system or attachments thereto during preoperative setup, during surgery, or during postoperative procedures. For example, a surgeon can use a portable interface device to select a robotic arm of the surgical robotic system by placing the robotic arm near the center of the camera's field of view. The interface device can automatically identify the robotic arm of interest and determine its initial or current pose by performing image processing on images of the surgical robotic system captured by the camera. Selecting and controlling components of the surgical robotic system through a direct view provided by the interface device, rather than an indirect view via a remote console, reduces the risk of surgeons mistakenly selecting and moving incorrect components of the surgical robotic system.

[0017] Based on the initial pose of the selected component, the interface device can present control options for that component. The surgeon can select control options to activate robot-assisted movement of the selected component along a planned trajectory to a target location. For example, the surgeon can move the robotic arm from a ready position to a pre-docking position to engage with endoscopic instruments or attachments designed for endoscopic manipulation of tissues, including grasping, cutting, blunt and sharp dissection, approximation, ligation, electrocautery, and suturing. As another example, the surgeon can adjust the angle of the operating table to elevate the patient's feet, thereby providing a clearer view of the target tissue or organ for endoscopic instruments.

[0018] Figure 1 This is a pictorial view of an exemplary surgical robotic system 1 in a surgical setting according to various aspects of the subject matter. The robotic system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic platform 5 (e.g., table, bed, etc.). The system 1 can be combined with any number of devices, tools, or accessories for performing surgery on a patient 6. For example, the system 1 may include one or more surgical tools 7 for performing surgery. The surgical tool 7 may be an end effector attached to the distal end of the surgical arm 4 for performing surgical procedures.

[0019] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during surgery. For example, a surgical tool 7 can be a tool for accessing, viewing, or manipulating the internal anatomy of the patient 6. In one embodiment, the surgical tool 7 is a gripper that can grasp the patient's tissues. The surgical tool 7 can be manually controlled by a bedside operator 8; or it can be robotically controlled via actuated movement of its attached surgical robotic arm 4. The robotic arm 4 is shown as a tabletop system, but in other configurations, the arm 4 can be mounted on a trolley, ceiling, or sidewall, or in another suitable structural support.

[0020] Generally, a remote operator 9 (such as a surgeon or another person) can use the user console 2 to remotely manipulate the arm 4 and / or the attached surgical instruments 7, for example, through remote operation. The user console 2 may be located in the same operating room as the rest of the system 1, such as... Figure 1 As shown. However, in other environments, the user console 2 may be located in an adjacent or nearby room, or it may be located in a remote location, such as in different buildings, cities, or countries. The user console 2 may include a seat 10, foot controls 13, one or more handheld user input devices (UIDs) 14, and at least one user display 15 configured to display a view, for example, of a surgical site within a patient 6. In the exemplary user console 2, a remote operator 9 sits in the seat 10 and views the user display 15 while manipulating the foot controls 13 and the handheld UID 14 to remotely control the arm 4 and surgical instruments 7 (which are mounted on the distal end of the arm 4).

[0021] In some variations, the bedside operator 8 can also operate the system 1 in a "bedside" mode, where the bedside operator 8 (the user) is now positioned to one side of the patient 6 and simultaneously manipulates robot-driven tools (end-effectors attached to arm 4), for example, holding a handheld UID 14 and a manual laparoscopic tool with one hand. For instance, the bedside operator's left hand can manipulate the handheld UID to control the robotic components, while the bedside operator's right hand can manipulate the manual laparoscopic tool. Therefore, in these variations, the bedside operator 8 can perform both robot-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0022] During the exemplary procedure (surgical operation), patient 6 is aseptically prepared and covered with a sterile drape for anesthesia. Initial access to the surgical site (to facilitate access to the surgical site) can be manually performed while the arm of robotic system 1 is in a retracted or withdrawn configuration. Once access is complete, initial positioning or preparation of robotic system 1 (including its arm 4) can be performed. For example, a remote operator 9 at user console 2 or a bedside operator 8 can use a handheld UID 14 to move arm 4 from the retracted configuration to a ready position above patient 6 during preoperative setup. Alternatively, a surgeon or bedside person at direct observation table 5 can operate the interface device disclosed herein to select arm 4 and move it to the ready position. The surgical operation then continues, wherein the remote operator 9 at user console 2 utilizes foot controls 13 and UID 14 to manipulate various end effectors and, possibly, imaging systems to perform the surgical procedure. Manual assistance can also be provided by a bedside person wearing sterile surgical gowns (e.g., bedside operator 8) at the operating table or operating table 5. This bedside person can perform tasks on one or more of the robotic arms 4, such as tissue retraction, manual repositioning, and tool changing. Non-sterilized personnel may also be present to assist the remote operator 9 at the user console 2. When the procedure or surgery is completed, system 1 and user console 2 can be configured or set to facilitate postoperative procedures, such as cleaning or sterilization, and inputting or printing medical records via user console 2.

[0023] In one embodiment, the remote operator 9 holds and moves UID 14 to provide input commands, thereby moving the robotic arm actuator 17 in the robotic system 1. UID 14 may be communicatively coupled to the rest of the robotic system 1, for example, via a console computer system 16. UID 14 may generate spatial state signals corresponding to the movement of UID 14, such as the position and orientation of the UID's handheld housing, and the spatial state signals may be input signals for controlling the movement of the robotic arm actuator 17. The robotic system 1 may use control signals derived from the spatial state signals to control the proportional movement of the actuator 17. In one embodiment, a console processor of the console computer system 16 receives the spatial state signals and generates corresponding control signals. Based on these control signals controlling how the actuator 17 is energized to move a segment or connector of the arm 4, the movement of a corresponding surgical tool attached to the arm may simulate the movement of UID 14. Similarly, the interaction between the remote operator 9 and UID 14 may generate, for example, a gripping control signal that causes the jaws of the gripper of the surgical tool 7 to close and grip the tissue of the patient 6.

[0024] The surgical robot system 1 may include a plurality of UIDs 14, wherein each UID generates a corresponding control signal controlling the actuators and surgical instruments (end-effectors) of the corresponding arm 4. For example, a remote operator 9 may move a first UID 14 to control the movement of an actuator 17 located in the left robotic arm, wherein the actuator responds by moving links, gears, etc. in the arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the movement of another actuator 17, which in turn moves other links, gears, etc. of the robot system 1. The robot system 1 may include a right arm 4 fixed to a bed or table 5 on the right side of the patient, and a left arm 4 located on the left side of the patient. The actuators 17 may include one or more motors, which are controlled such that they drive the joints of the arm 4 to rotate, for example, to change the orientation of the endoscope or gripper of the surgical instrument 7 attached to the arm relative to the patient. The movement of a plurality of actuators 17 in the same arm 4 may be controlled by spatial state signals generated from a particular UID 14. The UID 14 may also control the movement of the corresponding surgical instrument gripper. For example, each UID 14 can generate a corresponding gripping signal to control the movement of an actuator (e.g., a linear actuator) that opens or closes the jaws of the gripper at the distal end of the surgical tool 7 to grip tissue in the patient 6.

[0025] In one aspect, instead of simulating the movement of the arm 4 or the attached surgical instrument 7 via UID 14 in the hand of a remote operator 9, a surgeon directly observing the arm 4 or surgical instrument 7 can use the interface device disclosed herein to move the arm 4 or surgical instrument 7 to a desired position. For example, the surgeon can select one arm of the arm 4 by framing a target arm near the center of the field of view of the interface device's camera. The interface device can run image processing software on the image captured by the camera to automatically identify the target arm as the object to be controlled and determine its initial or current pose. For example, the interface device can identify an arm near the center of the image frame as the target arm and determine that the target arm is in a ready position. The interface device can highlight the target arm by displaying visual indicators on the screen, such as by outlining the target arm on the display screen with a green engagement indicator or by rendering an image of the target arm enhanced with virtual information. In another aspect, the surgeon can use a touchscreen to indicate the object to be controlled by touching the object, such as by touching the arm 4 displayed on the touchscreen.

[0026] In one aspect, the interface device can run image processing software to identify one or more objects or points near an object in an image frame as possible destination locations for the target arm. The interface device can identify or highlight possible destination locations on a display screen, display the name of the possible destination location on the display screen, and request confirmation from the surgeon regarding the destination location. For example, the interface device can identify a cannula near the target arm as a surgical instrument that the target arm may wish to engage by highlighting the cannula on the screen and requesting confirmation from the surgeon, thus identifying a possible destination location for the target arm. If the surgeon confirms, the interface device can calculate a trajectory for moving the target arm from a ready position to a pre-aggregation position near the cannula, such that the cannula can be attached to the distal end of the target arm. The interface device can display the calculated trajectory as one of the options for the surgeon to select. In another aspect, the surgeon can use a touchscreen to indicate the destination location of the target arm by, for example, pressing the cannula displayed on the touchscreen. In yet another aspect, the interface device can display multiple options for the surgeon to select, indicating a predefined position or pose as a possible destination location for the target arm and a planned trajectory to the predefined position. Once the surgeon selects a trajectory, position, or pose, the interface device transmits information to the surgical robot system 1, enabling the surgical robot system 1 to activate actuator 17 to drive gears, links, or joints of the target arm, thereby moving the target arm along the selected or programmed trajectory to the destination position or pose.

[0027] In some respects, communication between platform 5 and user console 2 or interface device can be achieved via control tower 3, which can translate user commands received from user console 2 (and more specifically from console computer system 16) or from interface device into robot control commands transmitted to arm 4 on robot platform 5. Control tower 3 can also transmit status and feedback from platform 5 back to user console 2 or interface device. The communication connection between robot platform 5, user console 2, and control tower 3 can be via wired and / or wireless links, using any suitable data communication protocol from a variety of data communication protocols. Any wired connection can optionally be built into the floor and / or walls or ceiling of the operating room. Robot system 1 can provide video output to one or more displays, including displays in the operating room and remote displays accessible via the Internet or other networks. Video output or feeds can also be encrypted to ensure privacy, and all or part of the video output can be stored on a server or electronic healthcare record system.

[0028] Before initiating surgery using the surgical robotic system 1, the surgical team can perform preoperative setup. During preoperative setup, the main components of the surgical robotic system (stage 5 and robotic arm 4, control tower 3 and user console 2) are positioned, connected, and powered in the operating room. Stage 5 and robotic arm 4 can be in a fully retracted configuration, with arm 4 positioned below stage 5 for storage and / or transport purposes. The surgical team can extend the arm from its retracted position for sterile draping. After draping, arm 4 can be partially retracted until needed. Several routine laparoscopic steps may need to be performed, including cannula placement and insufflation. For example, each cannula may be inserted into a small incision and through the body wall using a tampon. The cannulas and tampon allow optical access for visualization of tissue layers during insertion, minimizing the risk of injury during placement. Typically, an endoscope is placed first to provide handheld camera visualization for placement of other cannulas. After insufflation, manual instruments can be inserted through the cannulas, if needed, to perform any laparoscopic steps manually.

[0029] Next, the surgical team positions the robotic arms 4 above the patient and attaches each arm to its corresponding cannula. The surgical robotic system 1 can immediately and uniquely identify each tool (endoscope and surgical instrument) upon attachment and can display the tool type and arm position on the open or immersive display 15 at the user console 2 and on the touchscreen display on the control tower 3. The corresponding tool function is enabled and can be activated using the master UID 14 and foot switch 13. The patient-side assistant can attach and detach tools as needed throughout the procedure. The surgeon, seated at the user console 2, can begin surgery using tools controlled by two master UIDs 14 and foot switch 13. The system translates the surgeon's hand, wrist, and finger movements into precise, real-time movements of the surgical instruments via the master UID 14. Therefore, the system continuously monitors each surgical maneuver by the surgeon and pauses instrument movement if it cannot accurately reflect the surgeon's hand movements. If the endoscope moves from one arm to another during surgery, the system can adjust the master UID 14 to align the instruments and continue controlling and moving them. The foot switch 13 can be used to activate various system modes, such as endoscope control and various instrument functions, including monopolar and bipolar ablation, without removing the surgeon’s hand from the master UID 116.

[0030] Table 5 can be repositioned intraoperatively. For safety reasons, all instruments must be within the surgeon's field of vision and under their active control at the user console 2. Instruments not under the surgeon's active control must be removed, and the table legs must be locked. During table movement, the integrated robotic arm 4 can passively follow the table's movement. Audio and visual cues are available to guide the surgical team during table movement. Audio cues may include tone and voice prompts. Visual messages on displays at the user console 2 and control tower 3 inform the surgical team of the table movement status.

[0031] In one aspect, a surgeon directly observing table 5 can use the interface device disclosed herein to reposition table 5. For example, the surgeon may wish to elevate the upper torso of patient 6. The surgeon can select the section of table 5 that holds the patient's upper torso by framing the section near the center of the camera's field of view. The interface device can run image processing software on the camera image to identify the section as the object to be controlled. In another aspect, the surgeon can use a touchscreen to indicate the section to be controlled by touching it. The interface device can present multiple options indicating different degrees to which the section can be raised or lowered. In one aspect, the surgeon can specify an elevation angle. Once the surgeon selects or enters a new position for the section, the interface device transmits the information to the surgical robot system 1, enabling the surgical robot system 1 to drive mechanisms to raise the section to the desired position.

[0032] Figure 2A This is a front view 25 of an interface device according to various aspects of the subject matter, showing a display in which the robotic arm of a surgical robot system is visualized for the identification or selection / target selection of individual parts. The interface device includes a display screen 26 and a handle 27. The display screen 26 may display real-time images or videos captured by a camera on the back of the interface device. The display screen 26 may be a touchscreen to allow the user to input commands, make selections, or manipulate displayed objects. In one aspect, the user can select a target part of the surgical robot system for control by pointing the interface device at the target part, such that the target part appears near the center of the display screen 26. In one aspect, an image processing algorithm may process the images or videos captured by the camera to automatically identify parts of the surgical robot system appearing near the center of the display screen 26 as target parts. In one embodiment, a processor on the interface device may run the image processing algorithm. In another embodiment, the computer of the surgical robot system, such as… Figure 1 The control tower 3 can run image processing algorithms. The interface device can transmit captured images or videos to the surgical robot system for remote image processing operations and can receive identified target parts from the surgical robot system. In one embodiment, a user can identify a target part by touching the target part on the touch display screen 26.

[0033] In one aspect, the image processing algorithm can determine the initial pose or position of the target component. For example, a robotic arm may be folded in a retracted position for storage, extended in a draped position for sterile draping, bent over the patient in a ready position during preoperative setup, configured to be in a pre-docking position ready to attach to nearby surgical instruments, or in a docking position to attach to surgical instruments. The interface device may highlight the target component on display 26 for user confirmation, for example, by highlighting the target component with an engagement indicator. The user may confirm or reject the target component, in which case the interface device may attempt to identify another component. In one aspect, if the user moves the interface device to remove the target component from the camera's field of view before the user confirms the target component, the image processing algorithm may deselect the target component. In another aspect, if the user moves the interface device to remove the target component from the camera's field of view after the user confirms the target component and when the surgical robot system is activated to move the target component, the interface device may deselect the target component and freeze its movement to prevent the user from moving the target component when it is not in the user's focus of attention.

[0034] Based on the confirmed target component and its initial position or pose, the interface device can generate a list of options for the user to control the position, orientation, movement, function, etc., of the target component. For example, if the user confirms that the robot arm in the cover position is the target component, the interface device can present a list of options for moving the robot arm to a predefined ready position or pre-docking position. In one aspect, the option list may include a planned trajectory of the robot arm to the predefined position. The interface device can present a list of predefined target positions of the target component, names of predefined target positions (e.g., ready position, pre-docking position, etc.), or planned trajectories to predefined target positions as an overlay, such as as an augmented reality (AR) overlay on display screen 26. The user can select one of the predefined target positions as the desired target position of the target component. The interface device can calculate the trajectory for moving the target component from the initial position to the selected target position. In one aspect, when a desired target position is selected, the user can select one of the planned trajectories.

[0035] In one implementation, the image processing algorithm can identify another object in an image or video as a possible destination location for a target component. For example, if a robotic arm in a ready position is identified as the target component, the image processing algorithm can identify a surgical tool in the image or video as a possible target to which the robotic arm can be attached. The interface device can present the current position or pose of the surgical tool, or present the volume in the space immediately adjacent to the surgical tool as a possible pre-docking destination location for the robotic arm. If the user selects a presented pre-docking location, the interface device can calculate a trajectory for moving the robotic arm from the ready position to the pre-docking location. The interface device can calculate the trajectory such that the robotic arm avoids collisions or interference with the operations of other robotic arms. In one aspect, if the planned movement of the robotic arm requires moving other robotic arms, the list of options can include one or more planned trajectories for each of a plurality of robotic arms.

[0036] In one implementation, a user can indicate a destination location directly on a touchscreen by, for example, touching a displayed surgical tool. The interface device can highlight the selected surgical tool to request user confirmation of the selection. If confirmed, the interface device can calculate a trajectory for moving the robotic arm to the selected surgical tool and can present the trajectory as an option for user selection. In one implementation, a computer of the surgical robot system, such as a control tower, can perform the trajectory calculation instead of the interface device.

[0037] Once the user selects an option for moving or changing the position, orientation, or function of the robotic arm, the interface device transmits the selected trajectory and related information (if any) to the control tower 3 of the surgical robot system. The user may be prompted to activate the movement of the robotic arm by pressing a continuous activation button. When the user presses the continuous activation button, the interface device transmits an activation signal to the control tower 3. Upon receiving the activation signal, the control tower 3 generates a control signal to activate actuator 17, thereby driving the gears, links, or engagements of the robotic arm to move the robotic arm along the selected trajectory to its destination position. For safety reasons, the user may need to hold down the continuous activation button until the robotic arm completes its trajectory to the destination position. If the user releases the continuous activation button, the interface device stops transmitting the activation signal, causing the control tower 3 to stop the movement of the robotic arm. During movement, the display screen 26 may continue to highlight the robotic arm with an engagement indicator to visually indicate to the user that the robotic arm is moving. To increase safety, if the user moves the interface device so that the robotic arm is out of the camera's field of view, the interface device may cut off the transmission of the activation signal to stop the movement of the robotic arm, thus preventing injury to the patient from a distracted user.

[0038] See Figure 2ADisplay screen 26 shows exemplary images of the target robotic arm 28 and the second arm 29. An image processing algorithm identifies the target robotic arm 28, appearing in the center of the image, as the target component to be controlled by the surgical robotic system. The image processing algorithm determines that the target robotic arm 28 is in a ready position, positioned above the patient. Display screen 26 highlights the target robotic arm 28 by enclosing it with an engagement indicator 30. Based on the determination that the target robotic arm 28 is in the ready position, display screen 26 displays a list 31 of options for user control of the position, orientation, movement, function, etc., of the target robotic arm 28. For example, the option list 31 may include options for moving the target robotic arm 28 to a predefined pre-docking position, moving it to a pre-docking position near surgical instruments, moving it along a planned trajectory, etc. The user can select an option on display screen 26. Once selected, the engagement indicator 30 can change to a different color, such as green, to indicate that the target robotic arm 28 is ready to be activated. The user can press and hold a continuous activation button on the back of the interface device to activate the movement of the target robotic arm 28 along its trajectory to the selected destination position. When the target robotic arm 28 completes its trajectory and reaches its destination position, the movement stops and the engagement indicator 30 can be turned off to indicate that the target robotic arm 28 is deactivated and deselected.

[0039] Figure 2B This is a rear view 35 of an interface device according to various aspects of the subject matter, showing a camera 37, an optional three-dimensional (3-D) camera 38, and control buttons 36. Camera 37 can capture planar images or videos of a surgical robotic system or its components for display on a display screen 26 in front of the interface device. In one embodiment, the interface device can transmit the captured images or videos to the surgical robotic system for display on a remote screen. As discussed, image processing algorithms running on the interface device or surgical robotic system can process the images or videos to automatically identify components of the surgical robotic system appearing near the center of the image or video as target components to be controlled by the user. Additionally, the image processing algorithms can determine the initial pose or position of the target component. In one embodiment, the image processing algorithm can identify another object in the image or video as a possible destination location for the target component.

[0040] To assist image processing algorithms in identifying target components or other objects and their positions, the interface device may include sensors such as a 3-D camera 38 to determine the 3-D position of components of the surgical robotic system relative to the interface device. For example, the 3-D camera can measure the 3-D position of the interface device relative to surface points of components of the surgical robotic system. Implementations of the 3-D camera may include an infrared camera tracking system that emits a stereoscopic infrared projection pattern, an inertial measurement unit, a structured light camera, a micro-time-of-flight camera, a stereo camera, etc. In one implementation, a surgeon may use the 3-D camera of the interface device to scan the patient on the operating table during preoperative setup to assist the surgeon in planning surgical procedures using the surgical robotic system.

[0041] Control button 36 can be positioned on the two handles 27 to align with the control options displayed on the display screen 26. The user can use control button 36 to select a control option for operating the target component. Once a control option is selected, the user can use the control button to operate the target component to execute the selected option. In one aspect, control button 36 can be a continuously activated button, requiring continuous pressing to allow the user to activate the movement of the target component from its initial position to its destination position along a planned trajectory. When the user releases the control button, the movement of the target component can stop even if it has not yet completed its trajectory.

[0042] Figure 3 An interface device for controlling a robotic arm to move from an initial pose to a final pose, according to various aspects of the subject matter, is shown. Operating room environment 35 shows a patient 39 positioned on a table 36A of the surgical robotic system. A first robotic arm 37A and a second robotic arm 38 are positioned on one side of the table 36A. A cannula 40A has been placed on the patient 39 near the first robotic arm 37A.

[0043] The user interface device's camera, used to directly observe the operating room environment 35, captures real-time images or videos of the surgical robot system. Display screen 26 shows images of the first robotic arm 37B, the stage 36B, and the cannula 40B. The second arm 38 is not in the camera's field of view. The image processing algorithm identifies the first robotic arm 37B, appearing in the center of the image, as the target component of the surgical robot system to be controlled. The image processing algorithm determines that the first robotic arm 37B is in a ready position, positioned above the patient 39. Display screen 26 highlights the first robotic arm 37B by enclosing it with a first color, such as a yellow engagement indicator 30. Based on the determination that the first robotic arm 37B is in the ready position, display screen 26 displays a list 31 of options 31 for user control of the position, orientation, movement, functions, etc., of the first robotic arm 37B.

[0044] Image processing algorithms can further identify the cannula 40B as a possible destination location for the first robotic arm 37B. Therefore, option list 31 may include an option to move the first robotic arm 37B along a planned trajectory to a pre-dating position near the cannula 40B. The user can select this option on display screen 26. Once selected, engagement indicator 30 can change to a second color, such as green, to indicate that the first robotic arm 37B is ready to be activated. The user can press and hold a continuous activation button on the back of the interface device to activate the movement of the first robotic arm 37B along its trajectory to the pre-dating position near the cannula 40B. When the first robotic arm 37B completes its trajectory and reaches the pre-dating position, the movement stops and engagement indicator 30 can be deactivated to indicate that the first robotic arm 37B is deactivated and deselected.

[0045] The interface device can also be used to select other components of the surgical robotic system. For example, the interface device and image processing algorithms can identify the stage 36B appearing in the center of an image as a target component of the surgical robotic system to be controlled. In one embodiment, the interface device can be used to perform a 3-D scan of the patient on the stage during preoperative setup to assist the surgeon in planning surgery using the surgical robotic system. The interface device can then be used to control the robotic arm and stage to achieve optimal robotic arm and stage movements specific to the patient. The interface device can facilitate the user's selection of components of the surgical robotic system using virtual representations or augmented reality views. For example, the display screen 26 can display rendered images or virtual models of the components, such as camera views of the components enhanced with additional or virtual information.

[0046] The interface device can provide additional functionalities such as a web portal (e.g., a browser) and can display information such as case settings, surgeon preference cards, instrument lifespan, documents, snapshots from the endoscope for documentation, patient photos for documentation, system eIFU, patient data, and protocol guidelines. Other functionalities may include: teleconferencing using a microphone, speaker, and front-facing webcam; service calls using a microphone, speaker, and rear-facing webcam; user authentication using a fingerprint reader and / or NFC card reader; docking stations for connecting to surgical robotic systems, such as pairing a new interface device with the surgical robotic system by sliding the interface device into the docking station to enable the surgical robotic system to recognize the interface device; and range detection of the interface device to the control tower using Bluetooth and WiFi to monitor signal strength and trigger alarms when the interface device is too far away.

[0047] Figure 4This is a block diagram illustrating exemplary hardware components of an interface device 50 according to various aspects of the subject matter. The interface device 50 includes a camera 51, a sensor 52, a display 53, a user command interface 54, a processor 55, a memory 56, and a network interface 57. The camera 51 may be configured to capture planar images or video of a surgical robotic system. The sensor 52 may be a 3D camera configured to determine the 3D position of components of the surgical robotic system relative to the interface device. The images captured by the camera 51 and the 3D position measurements taken by the sensor 52 may be displayed on the display 53, which may be a touchscreen.

[0048] Processor 55 can be configured to run image processing algorithms to process images captured by camera 51 and measurements taken by sensor 52, thereby automatically identifying parts of the surgical robotic system appearing near the center of the image as target parts to be controlled by the user. Processor 55 can be configured to run an operating system to control the operation of interface device 50. Memory 56 can store image processing algorithms, operating system, program code, and other data used by processor 55. User command interface 54 may include continuously activated buttons used by the user to select control options, thereby operating the target part and executing the selected options to move the target part to its destination location.

[0049] Hardware components can communicate via a bus. The interface device can communicate with the surgical robot system via network interface 57 through external interface 58. External interface 58 can be a wireless or wired interface.

[0050] Figure 5 This is a block diagram illustrating exemplary hardware components of a surgical robot system 60 according to various aspects of the present subject matter. The exemplary surgical robot system 60 may include an interface device 50, a surgical robot 80, and a control tower 70. The interface device 50 has been integrated with… Figure 4 This will be discussed further and will not be repeated here. The surgical robotic system 60 may include other hardware components or additional hardware components; therefore, the diagram is provided as an example rather than a limitation on the system architecture.

[0051] Control tower 70 may be a mobile field care cart housing a touchscreen display, a computer controlling surgeons to manipulate instruments with robot assistance, a safety system, a graphical user interface (GUI), a light source, and a video and graphics computer. For example... Figure 5As shown, the control tower 70 may include a central computer 71 (which may include at least a visualization computer, a control computer, and an auxiliary computer), various displays 73 (which may include team displays and nurse displays), and a network interface 78 coupling the control tower 70 to both the interface device 50 and the surgical robot 80. The control tower 70 may also accommodate third-party devices such as an advanced light engine 72, an electrosurgical generator (ESU) device 74, and inhalers and CO2 canisters 75. The control tower 70 may provide additional features for user convenience, such as a nurse display touchscreen, soft power and E-hold buttons, a user-facing USB port for video and still images, and an electronic caster control interface. The auxiliary computer may also run real-time Linux, providing logging / monitoring and interaction with cloud-based web services.

[0052] The surgical robot 80 includes an articulated operating table 84 with multiple integrated arms 82 positioned above the target patient anatomy. A set of compatible tools 83 can be attached to / detached from the distal ends of the arms 82, enabling the surgeon to perform various surgical procedures. The surgical robot 80 may also include a control interface 85 for manually controlling the arms 82, the operating table 84, and the tools 83. The control interface 85 may include items such as, but not limited to, remote controls, buttons, panels, and touchscreens. Other accessories such as cannulas (cannulas, sealing cartridges, and tamponades) and drapes can also be manipulated to perform surgery using the system. In one embodiment, the multiple arms 82 may include four arms mounted on both sides of the operating table 84, with two arms on each side. For a particular surgical procedure, an arm mounted on one side of the operating table 84 can be positioned on the other side of the operating table 84 by stretching and crossing under the operating table 84 and the arm mounted on the other side, resulting in a total of three arms positioned on the same side of the operating table 84. The surgical instrument may also include a computer 81 and a network interface 88, which allows the surgical robot 80 to be positioned to communicate with the control tower 70.

[0053] Figure 6 This is a flowchart illustrating a method 90 for controlling a surgical robotic system using a mobile device according to various aspects of the art of this subject. The mobile device may be... Figure 2A , Figure 2B or Figure 4 Interface devices.

[0054] In box 91, the mobile device can capture real-time images of a portion of the surgical robotic system. The mobile device can use a camera to capture planar images or videos of the components of the surgical robotic system for display on the screen.

[0055] In box 93, a mobile device or user can identify a target component of the surgical robotic system from the captured real-time images. For example, a user can select a target component of the surgical robotic system for control by pointing the mobile device at the target component so that the target component appears near the center of the display screen. Image processing algorithms running on the mobile device or on the control tower can process the captured real-time images to identify the target component.

[0056] In box 95, the mobile device can display captured real-time images, target parts, and possible target poses of the target parts on the screen. For example, the mobile device can highlight a robotic arm as a target part on the screen by displaying visual indicators outlining the robotic arm's contours or rendering an image of the robotic arm overlay with enhanced or virtual information. On the other hand, the mobile device can run image processing algorithms on the captured real-time images to identify one or more objects that the robotic arm can engage as possible target poses of the robotic arm.

[0057] In box 97, the mobile device can receive input from the user to select one of the displayed target poses for the target component. For example, the user can indicate the pre-docking destination position of the robotic arm on a touchscreen. In other respects, the user can use the touchscreen to select options for moving or changing the position, orientation, or function of the robotic arm.

[0058] In box 98, the mobile device can generate input commands to drive the target component to a selected target pose. For example, the mobile device can transmit an activation signal to a control tower to cause the control tower to generate control signals, thereby activating actuators to drive gears, links, or joints of the robot arm along a calculated trajectory to the selected destination position. In one aspect, the mobile device or control tower can calculate a trajectory for moving the robot arm from an initial pose to a destination pose. The calculated trajectory allows the robot arm to avoid collisions or interference with the operations of other robot arms.

[0059] For purposes of explanation, the foregoing description uses specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that specific details are not required to practice the invention. The foregoing description of specific embodiments of the invention has been provided for illustrative and descriptive purposes. These are not intended to be exhaustive or to limit the invention to the specific forms disclosed; various modifications and alterations can be made to this disclosure in light of the foregoing teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application. Therefore, these embodiments enable others skilled in the art to best utilize the invention, as well as various embodiments with modifications suitable for the contemplated particular uses. The following claims and their equivalents are intended to define the scope of the invention.

[0060] The methods, devices, processes, and logic components described above can be implemented in a variety of different ways and in a variety of different combinations of hardware and software. Controllers and estimators may include electronic circuitry. For example, all or part of an implementation may be a circuit including an instruction processor, such as a central processing unit (CPU), microcontroller, or microprocessor; an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA); or a circuit including discrete logic components or other circuit components (including analog circuit components, digital circuit components, or both); or any combination thereof. As an example, the circuit may include discretely interconnected hardware components and / or may be combined on a single integrated circuit die, distributed among multiple integrated circuit dies, or implemented in a multi-chip module (MCM) of multiple integrated circuit dies in a co-package.

[0061] The circuit may also include or access instructions that are executed by the circuit. These instructions may be stored in a tangible storage medium other than transient signals, such as flash memory, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM); or on a magnetic disk or optical disk, such as an optical disc read-only memory (CDROM), a hard disk drive (HDD), or other magnetic disk or optical disk; or in or on another machine-readable medium. A product (such as a computer program product) may include a storage medium and instructions stored in or on that medium, and these instructions, when executed by circuitry in the device, may cause the device to perform any of the processes described above or shown in the accompanying drawings.

[0062] These implementations can be distributed as circuits among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems. Parameters, databases, and other data structures can be stored and managed separately, or combined into a single memory or database. They can be organized logically and physically in a variety of different ways and implemented in a variety of different ways, including as data structures such as linked lists, hash tables, arrays, records, objects, or implicit storage mechanisms. Programs can be parts of a single program (e.g., subroutines), stand-alone programs, distributed across multiple memories and processors, or implemented in a variety of different ways, such as in libraries, such as shared libraries (e.g., dynamic link libraries (DLLs)). For example, when executed by the circuit, the DLL can store instructions for performing any of the processes described above or shown in the figures.

[0063] Furthermore, the various controllers discussed herein may take the form of, for example, processing circuitry, microprocessors or processors, and computer-readable media storing computer-readable program code (e.g., firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Controllers may be configured with hardware and / or firmware to perform the various functions described below and shown in the flowcharts. Additionally, some components shown as being internal to the controller may also be stored externally, and other components may be used.

Claims

1. A mobile interface device for a surgical robot system, comprising: A camera configured to capture real-time images of a portion of the surgical robotic system; Processor, the processor being configured to: The captured real-time images are processed to identify the target components of the surgical robot system to be controlled using the mobile interface device; Determine the current pose of the target component; as well as Generate a list of control options; The display is configured to: Display the captured real-time images; Display the list of control options; as well as Display one or more target poses of the target component; as well as One or more control interfaces, wherein the one or more control interfaces are configured to: Allows the user to select one or more options from the control options; The user is allowed to select one of the target poses; as well as Based on the selected control options, an input command is generated to drive the target component from the current pose to the selected target pose.

2. The mobile interface device according to claim 1, wherein, The display is further configured to present information to visually indicate to the user the one or more target poses of the target component.

3. The mobile interface device according to claim 1, wherein, The processor is further configured to determine a plurality of command options for controlling the target component, and the display is further configured to display the plurality of command options for the user to control the target component.

4. The mobile interface device according to claim 1, wherein, The processor is further configured to process the captured real-time images to identify a second component of the surgical robot system, and the display is further configured to display information indicating the current pose of the second component as a target pose among the target poses of the target component.

5. The mobile interface device according to claim 1, wherein, The processor is further configured to determine a trajectory for driving the target component from the current pose to the selected target pose.

6. The mobile interface device of claim 1, further comprising one or more sensors configured to measure the three-dimensional position of the interface device relative to a plurality of surface points of the surgical robot system captured in the real-time image, and wherein the processor is further configured to process the plurality of measured three-dimensional positions of the surface points of the surgical robot system captured in the real-time image to identify the target component.

7. The mobile interface device according to claim 1, wherein, The camera includes a 3D camera configured to scan a patient on the operating table of the surgical robotic system during preoperative setup, wherein the processor is further configured to process the patient's scan information to determine patient-specific movements of the target component.

8. The mobile interface device according to claim 1, wherein, The one or more control interfaces are further configured to generate a second input command to freeze the movement of the target component when the target component moves away from the captured real-time image.

9. The mobile interface device according to claim 1, wherein, The display is further configured to show visual indications of the target component to allow the user to confirm the target component.

10. The mobile interface device according to claim 1, wherein, The display is a touchscreen display, and the touchscreen display is configured to allow the user to select the current pose of the second component of the surgical robot system in the captured real-time image as the desired target pose of the target component.

11. A surgical robotic system, comprising: Operating table; Multiple robotic arms are coupled to the operating table; Mobile interface device, the mobile interface device being configured to: Capture real-time images of the operating table or the multiple robotic arms; The real-time images display rendered images of the operating table or the multiple robotic arms. Generate and display a list of control options; Receive user input to select one or more of the control options; Receive user input to control one of the operating table or the robotic arm; as well as In response to the user input, an input command is generated based on the selected control option to control one of the robotic arms, either the operating table or the robotic arm. as well as A control tower, communicatively coupled to the mobile interface device, the operating table, and the plurality of robotic arms, wherein the control tower is configured to: Receive the input command from the mobile interface device; as well as In response to the input command, control commands are generated to manipulate either the operating table or one of the robotic arms.

12. The surgical robot system of claim 11, wherein, The mobile interface device is further configured to present multiple input command options to control one of the robotic arms, either the operating table or the robotic arm.

Images